Busy infinite retrying: "Retrying after failure"

We are using temporal java SDK, and seeing this infinite retry loop happens every millisecs.

Stack trace:

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.35.0.jar:1.35.0]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.35.0.jar:1.35.0]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.35.0.jar:1.35.0]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskCompleted(WorkflowServiceGrpc.java:2727) ~[temporal-serviceclient-1.0.5.jar:?]
	at io.temporal.internal.external.ManualActivityCompletionClientImpl.lambda$complete$0(ManualActivityCompletionClientImpl.java:110) ~[temporal-sdk-1.0.5.jar:?]
	at io.temporal.internal.common.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:109) ~[temporal-sdk-1.0.5.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) ~[temporal-sdk-1.0.5.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retry(GrpcRetryer.java:106) ~[temporal-sdk-1.0.5.jar:?]
	at io.temporal.internal.external.ManualActivityCompletionClientImpl.complete(ManualActivityCompletionClientImpl.java:104) ~[temporal-sdk-1.0.5.jar:?]
	at io.temporal.internal.sync.ActivityCompletionClientImpl.complete(ActivityCompletionClientImpl.java:44) ~[temporal-sdk-1.0.5.jar:?]

Is there a way to set max retry on this?

Hi @zzz we are looking into this internally to figure out if it’s a bug. As far as for now, could you try setting maximumAttempts on RpcRetryOptions?

Do you get these logs forever? The expiration set should have a max of 1 minute. Do you get it for longer?

Yes, we have a 1 hour workflowRunTimeout on the workflow level, and the infinite retry was running for the whole one hour. We will try the maximumAttempts. Thanks!

Also from logs it shows you are on the Java SDK 1.0.5. Could you update to 1.0.9 (Release v1.0.9 release · temporalio/sdk-java · GitHub) and see if you get the same results? At least then we can be on the latest release (will be easier to debug stuff). Thanks!

Sure, I will update to 1.0.9. Actually in 1.0.5, RpcRetryOptions is not available as well. I will update here if we see this issue again