Busy infinite retrying: "Retrying after failure"

Community Support
1

We are using temporal java SDK, and seeing this infinite retry loop happens every millisecs.

Stack trace:

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.35.0.jar:1.35.0]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.35.0.jar:1.35.0]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.35.0.jar:1.35.0]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskCompleted(WorkflowServiceGrpc.java:2727) ~[temporal-serviceclient-1.0.5.jar:?]
	at io.temporal.internal.external.ManualActivityCompletionClientImpl.lambda$complete$0(ManualActivityCompletionClientImpl.java:110) ~[temporal-sdk-1.0.5.jar:?]
	at io.temporal.internal.common.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:109) ~[temporal-sdk-1.0.5.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) ~[temporal-sdk-1.0.5.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retry(GrpcRetryer.java:106) ~[temporal-sdk-1.0.5.jar:?]
	at io.temporal.internal.external.ManualActivityCompletionClientImpl.complete(ManualActivityCompletionClientImpl.java:104) ~[temporal-sdk-1.0.5.jar:?]
	at io.temporal.internal.sync.ActivityCompletionClientImpl.complete(ActivityCompletionClientImpl.java:44) ~[temporal-sdk-1.0.5.jar:?]

Is there a way to set max retry on this?

2

Hi @zzz we are looking into this internally to figure out if it’s a bug. As far as for now, could you try setting maximumAttempts on RpcRetryOptions?

Do you get these logs forever? The expiration set should have a max of 1 minute. Do you get it for longer?

3

Yes, we have a 1 hour workflowRunTimeout on the workflow level, and the infinite retry was running for the whole one hour. We will try the maximumAttempts. Thanks!

4

Also from logs it shows you are on the Java SDK 1.0.5. Could you update to 1.0.9 (Release v1.0.9 release · temporalio/sdk-java · GitHub) and see if you get the same results? At least then we can be on the latest release (will be easier to debug stuff). Thanks!

5

Sure, I will update to 1.0.9. Actually in 1.0.5, RpcRetryOptions is not available as well. I will update here if we see this issue again