一 dubbo超时时间的设置
定义:这里说的超时,是针对dubbo provider调用方来说的,比如提供方执行需要3秒,设置了超时时间1秒(默认),则consumer调用方会报超时错误,但实际上服务提供方依然在正常执行方法逻辑。
dubbo默认超时时间1000ms,但对于有些业务可能不满足需求,可根据业务场景定义超时时间。
超时时间设置可以选择在consumer端、provider端全局、接口、或方法设置。
优先级:
1) 方法级配置级别优于接口级别,即小Scope优先
2) Consumer端配置 优于 Provider配置 优于 全局配置
关于如何设置超时时间及优先级 可参考这位博主的文章 阐述Dubbo接口调用的超时机制_dubbo调用超时设置-CSDN博客
博主在自己公司的项目里是在服务提供者server端设置的超时时间是30秒。
<dubbo:service timeout="30000" interface="***ScheduleJobService" ref="scheduleJobService"/>
但实际方法执行时间超过了30秒导致dubbo调用超时;因为dubbo默认的超时重试次数为2,所以会看到一个定时任务执行了3次(第1次调用 + 2次超时重试)。幸好该任务是支持重复执行的,否则又是个大坑!
org.apache.dubbo.rpc.RpcException: Failed to invoke the method delBindedSignInfoDayBefore
in the service ****.ScheduleJobService.
Tried 3 times of the providers [192.168.11.15:5004, 192.168.11.5:5004, 192.168.11.30:5004] (3/6) from the registry nacos.server:8848 on the consumer 192.168.11.15 using the dubbo version 2.7.5. Last error is: Invoke remote method timeout. method:
delBindedSignInfoDayBefore,
provider: dubbo://192.168.11.5:5004/*****.ScheduleJobService?anyhost=true&application=api-schedule&category=providers&check=false&checkDataToSignService.async=true&checkDataToTimeOutService.async=true&checkIsExistOverdueData.async=true&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&init=false&interface=****ScheduleJobService....
如何让任务只执行一次呢?对于定时任务逻辑耗时通常都比较长,设置多大的超时时间都不合适,最好在调用端设置成异步调用。
<dubbo:reference id="scheduleJobService" interface="****.ScheduleJobService">
<dubbo:method name="delBindedSignInfoDayBefore" async="true"/>
</dubbo:reference>
二 dubbo重试次数设置
dubbo默认的超时重试次数为2,也可通过 retries=自定义重试次数
三 dubbo线程池设置
在项目中cat监控发现凌晨有rpc异常,具体异常信息如下
org.apache.dubbo.rpc.RpcException: Failed to invoke remote method: approveNotify, provider: dubbo://172.16.6.178:4003/***.DoctorNotifyService?anyhost=true&application=api-am&category=providers&check=false&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&init=false&interface=***.DoctorNotifyService&metadata-type=remote&methods=faceNotify,certDownNotify,changePhoneNotify,invoiceNotify,bindFirmNotify,enableNotify,approveNotify,deleteNotify,payNotify,cancleNotify,uploadUPic,relapseNotify,disableNotify&owner=lt&path=****DoctorNotifyService&payload=53886080&pid=22900&protocol=dubbo&qos.enable=false&reference.filter=Consumer®ister.ip=192.168.1.15&release=2.7.22&remote.application=ywq-service-notify&service.filter=Provider,-exception&service.name=ServiceBean:/***.DoctorNotifyService&side=consumer&sticky=false&timeout=20000×tamp=1722420199999, cause: org.apache.dubbo.remoting.RemotingException: Server side(172.16.6.178,4003) thread pool is exhausted, detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-172.16.6.178:4003, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 6654 (completed: 6454), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://172.16.6.178:4003!
at org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.invoke(AsyncToSyncInvoker.java:74)
at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78)
at *****cat.filter.ConsumerFilter.invoke(ConsumerFilter.java:55)
at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
at org.apache.dubbo.monitor.support.MonitorFilter.invoke$original$Gnq7aQ6X(MonitorFilter.java:91)
at org.apache.dubbo.monitor.support.MonitorFilter.invoke$original$Gnq7aQ6X$accessor$cd0tfALf(MonitorFilter.java)
at org.apache.dubbo.monitor.support.MonitorFilter$auxiliary$XZxwxCmN.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:93)
at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java)
at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52)
at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
at org.apache.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:69)
at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:79)
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:265)
at org.apache.dubbo.rpc.cluster.interceptor.ClusterInterceptor.intercept(ClusterInterceptor.java:47)
at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$InterceptorInvokerNode.invoke(AbstractCluster.java:92)
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:98)
at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:170)
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:96)
at org.apache.dubbo.common.bytecode.proxy6.approveNotify(proxy6.java)
at ****.TempUserInfoServiceImpl.sendAsynFirmNotify(TempUserInfoServiceImpl.java:908)
at ****.impl.TempUserInfoServiceImpl.approveTempDoctorInfo(TempUserInfoServiceImpl.java:514)
at ****.impl.TempUserInfoServiceImpl$$FastClassBySpringCGLIB$$149c208c.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
at akka.actor.UntypedAbstractActor$$anonfun$receive$1.applyOrElse(AbstractActor.scala:332)
at akka.actor.Actor.aroundReceive(Actor.scala:537)
at akka.actor.Actor.aroundReceive$(Actor.scala:535)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
at akka.actor.ActorCell.invoke(ActorCell.scala:547)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
at akka.dispatch.Mailbox.run(Mailbox.scala:231)
at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.util.concurrent.ExecutionException: org.apache.dubbo.remoting.RemotingException: Server side(172.16.6.178,4003) thread pool is exhausted, detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-172.16.6.178:4003, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 6654 (completed: 6454), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://172.16.6.178:4003!
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
线程池如何设置,设置多少合适了?或者是因为dubbo方法本身耗时比较长,一直未释放线程连接导致线程池耗尽?