In my temporal java worker (self-hosted temporal), I see this exception happening a lot and the worker stops executing tasks. The worker is not being shut down, so what can cause this, or what can I do to get to the bottom of this.
Failure in poller thread Activity Poller taskQueue=“MY_TASK_QUEUE”
io.grpc.StatusRuntimeException: UNAVAILABLE: Subchannel shutdown invoked
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268) ~[worker.jar:?]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249) ~[worker.jar:?]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167) ~[worker.jar:?]
at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.pollActivityTaskQueue(WorkflowServiceGrpc.java:3959) ~[worker.jar:?]
at io.temporal.internal.worker.ActivityPollTask.poll(ActivityPollTask.java:109) ~[worker.jar:?]
at io.temporal.internal.worker.ActivityPollTask.poll(ActivityPollTask.java:44) ~[worker.jar:?]
at io.temporal.internal.worker.Poller$PollExecutionTask.run(Poller.java:298) ~[worker.jar:?]
at io.temporal.internal.worker.Poller$PollLoopTask.run(Poller.java:258) ~[worker.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
This is happening for all the workers on each task queue started in my application. We do have linkerd in place, but the temporal cluster is not part of linkerd, and disabling linkerd did not resolve the issue.
I am seeing this exception on pretty much all of our applications using the temporal SDK to set up workers, and I can not find any information if this is a problem or not, and what would be the cause for the channels to be shut down.
Hi All. We are seeing similar issue with temporal with linkerd disabled
io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.startWorkflowExecution(WorkflowServiceGrpc.java:2766)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Subchannel shutdown invoked
at java.base/java.lang.Thread.run(Thread.java:833)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at io.helidon.common.context.ContextAwareExecutorImpl.lambda$wrap$7(ContextAwareExecutorImpl.java:154)
at io.helidon.common.context.Contexts.runInContext(Contexts.java:117)
at io.helidon.webserver.jersey.JerseySupport$JerseyHandler.lambda$doAccept$4(JerseySupport.java:335)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at oracle.spectra.batch.workflowexecution.workflow.WorkFlowResource$Proxy$_$$_WeldClientProxy.runWorkflow(Unknown Source)
at oracle.spectra.batch.workflowexecution.workflow.WorkFlowResource.runWorkflow(WorkFlowResource.java:99)
at io.temporal.internal.sync.WorkflowStubImpl.start(WorkflowStubImpl.java:135)
at io.temporal.internal.sync.WorkflowStubImpl.startWithOptions(WorkflowStubImpl.java:117)
at io.temporal.internal.sync.WorkflowStubImpl.wrapStartException(WorkflowStubImpl.java:181)
We are seeing similar issue with temporal with linkerd disabled
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Subchannel shutdown invoked
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.startWorkflowExecution(WorkflowServiceGrpc.java:2766)
at io.temporal.internal.client.external.GenericWorkflowClientExternalImpl.lambda$start$0(GenericWorkflowClientExternalImpl.java:88)
at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61)
at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51)
at io.temporal.internal.client.external.GenericWorkflowClientExternalImpl.start(GenericWorkflowClientExternalImpl.java:81)
at io.temporal.internal.client.RootWorkflowClientInvoker.start(RootWorkflowClientInvoker.java:55)
at io.temporal.internal.sync.WorkflowStubImpl.startWithOptions(WorkflowStubImpl.java:110)
temporal-sdk version: 1.11.0
