Hi,
We are receiving DEADLINE_EXCEEDED error when initiating workflows, however unlike other posts in the forum, we are not facing any issues with the workflows themselves, that is all workflows successfully reach the Completed state. The issue is that the WorkflowClient.start method blocks the main thread for 10 sec before actually getting initiated asynchronously. This is leading to an increase in the response times.
We right now believe this has to do with the k8s ingress controller and the worker unable to communicate with the cluster. Is there any additional insights you can provide us? This only happens intermittently. To give some stats, only around one in 50 workflows return this error. Also almost always, the worker succeeds in connecting to the cluster on the second attempt.
Here is the stacktrace
DEADLINE_EXCEEDED: deadline exceeded after 9.999904972s. [closed=, open=[[remote_addr=temporal-frontend-address/IP]]]
io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
…lowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskCompleted(WorkflowServiceGrpc.java:3840)
io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.lambda$sendReply$0(ActivityWorker.java:303)
io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:52)
io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67)
io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60)
io.temporal.internal.retryer.GrpcRetryer.retry(GrpcRetryer.java:50)
io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.sendReply(ActivityWorker.java:298)
io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:252)
io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206)
io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179)
io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base/java.lang.Thread.run(Thread.java:1589)