RESOURCE_EXHAUSTED: workflow operation can not be applied because workflow is closing

Community Support
1

We are seeing the below error sometimes, wondering what could cause it?

  1. Client is java, we use “signalwithstart()” to start the workflow
  2. Worker is in python, that picks up the work and works on
  3. Since signalwithstart is supposed handle this gracefully (if workflow is not running, start and signal it, if its already running, just signal it etc), wondering if we are missing something here

Error:

RESOURCE_EXHAUSTED: workflow operation can not be applied because workflow is closing

Callstack:

	io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
	at 	io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
	at 	io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
	at 	io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.signalWithStartWorkflowExecution(WorkflowServiceGrpc.java:4442)
	at 	io.temporal.internal.client.external.GenericWorkflowClientImpl.lambda$signalWithStart$3(GenericWorkflowClientImpl.java:110)
	at 	io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:69)
	at 	io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60)
	at 	io.temporal.internal.client.external.GenericWorkflowClientImpl.signalWithStart(GenericWorkflowClientImpl.java:105)
	at 	io.temporal.internal.client.RootWorkflowClientInvoker.signalWithStart(RootWorkflowClientInvoker.java:193)
	at 	io.temporal.client.WorkflowStubImpl.signalWithStartWithOptions(WorkflowStubImpl.java:139)
	at 	io.temporal.client.WorkflowStubImpl.signalWithStart(WorkflowStubImpl.java:176)
	at 	io.temporal.client.SignalWithStartBatchRequest.signalWithStart(SignalWithStartBatchRequest.java:57)
	at 	io.temporal.client.SignalWithStartBatchRequest.invoke(SignalWithStartBatchRequest.java:50)
	at 	io.temporal.client.WorkflowClientInternalImpl.signalWithStart(WorkflowClientInternalImpl.java:256)
	at 	java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at 	java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at 	io.temporal.internal.WorkflowThreadMarker.lambda$protectFromWorkflowThread$1(WorkflowThreadMarker.java:83)
	at 	jdk.proxy4/jdk.proxy4.$Proxy315.signalWithStart(Unknown Source)
	at 	com.foo.bar.service.RequestSearchSyncService.syncDataIngestion(RequestSearchSyncService.java:143)
	at 	com.foo.bar.service.RequestSearchSyncService.processRequestEvent(RequestSearchSyncService.java:102)
	at 	java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at 	java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at 	org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
	at 	org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at 	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at 	org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
	at 	org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
	at 	org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:391)
	at 	org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
	at 	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at 	org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
	at 	org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703)
	at 	com.foo.bar.service.RequestSearchSyncService$SpringCGLIB$0.processRequestEvent(<generated>)
	at 	com.foo.bar.kafka.RequestSearchSyncListener.syncRequestToElasticsearch(RequestSearchSyncListener.java:41)
	at 	java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at 	java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at 	org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
	at 	org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at 	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at 	org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
	at 	org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
	at 	java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at 	java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at 	java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at 	java.base/java.lang.Thread.run(Thread.java:1583)
2

Hi @Suresh

This can happen if the workflow receives a request while it is closing temporal/service/history/consts/const.go at 0483e4b2156d3fd1f1753da1b352305371c3320b · temporalio/temporal · GitHub

Since signalwithstart is supposed handle this gracefully (if workflow is not running, start and signal it, if its already running, just signal it etc), wondering if we are missing something here

My understanding is the SDK retries on resource_exhausted, so you won’t have lost any of those requests. Could you confirm it?

Antonio

3

Thanks @antonio.perez

Sorry, missed this, on further inspection, it seems we didn’t lose any requests, I will keep an eye on this and come back

1 Like