Using Java SDK 1.19.1.
Seeing the following exception stack trace in temporal UI:
“message”: “Failure handling event 3 of type ‘EVENT_TYPE_WORKFLOW_TASK_STARTED’ during execution. {WorkflowTaskStartedEventId=3, CurrentStartedEventId=3}”,
“source”: “JavaSDK”,
“stackTrace”: “io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:257)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:236)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:208)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:224)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:208)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:156)\nio.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:131)\nio.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:96)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:407)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:317)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:259)\nio.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105)\njava.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\njava.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\njava.base/java.lang.Thread.run(Thread.java:829)\n”,
“encodedAttributes”: null,
“cause”: {
“message”: “WorkflowTask: failure executing SCHEDULED->WORKFLOW_TASK_STARTED, transition history is [CREATED->WORKFLOW_TASK_SCHEDULED]”,
“source”: “JavaSDK”,
“stackTrace”: “io.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:152)\nio.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:102)\nio.temporal.internal.statemachines.EntityStateMachineBase.handleEvent(EntityStateMachineBase.java:68)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleSingleEvent(WorkflowStateMachines.java:277)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:234)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:208)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:224)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:208)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:156)\nio.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:131)\nio.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:96)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:407)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:317)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:259)\nio.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105)\njava.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\njava.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\njava.base/java.lang.Thread.run(Thread.java:829)\n”,
“encodedAttributes”: null,
“cause”: {
“message”: “Potential deadlock detected. Workflow thread "workflow-method-xxx/Prodtest11178_1-291d46d7-9fba-451a-8320-8666c9149b81" didn’t yield control for over a second. {detectionTimestamp=1704729389407, threadDumpTimestamp=1704729389409} \n\nworkflow-method-xxx-demo/xxx\n\tat java.base@11.0.19/java.lang.Class.initClassName(Native Method)\n\tat java.base@11.0.19/java.lang.Class.getName(Class.java:793)\n\tat java.base@11.0.19/java.lang.Class.getPackageName(Class.java:1008)\n\tat java.base@11.0.19/java.lang.ClassLoader.postDefineClass(ClassLoader.java:923)\n\tat java.base@11.0.19/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)\n\tat java.base@11.0.19/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)\n\tat java.base@11.0.19/java.net.URLClassLoader.defineClass(URLClassLoader.java:555)\n\tat java.base@11.0.19/java.net.URLClassLoader$1.run(URLClassLoader.java:458)\n\tat java.base@11.0.19/java.net.URLClassLoader$1.run(URLClassLoader.java:452)\n\tat java.base@11.0.19/java.security.AccessController.doPrivileged(Native Method)\n\tat java.base@11.0.19/java.net.URLClassLoader.findClass(URLClassLoader.java:451)\n\tat java.base@11.0.19/java.lang.ClassLoader.loadClass(ClassLoader.java:589)\n\tat app//org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:151)\n\tat java.base@11.0.19/java.lang.ClassLoader.loadClass(ClassLoader.java:522)\n\tat io.temporal.api.workflowservice.v1.PollActivityTaskQueueResponse.toBuilder(PollActivityTaskQueueResponse.java:1250)\n\tat io.temporal.api.workflowservice.v1.PollActivityTaskQueueResponse.newBuilder(PollActivityTaskQueueResponse.java:1242)\n\tat io.temporal.internal.sync.SyncWorkflowContext.constructExecuteLocalActivityParameters(SyncWorkflowContext.java:537)\n\tat io.temporal.internal.sync.SyncWorkflowContext.executeLocalActivityLocally(SyncWorkflowContext.java:468)\n\tat io.temporal.internal.sync.SyncWorkflowContext.executeLocalActivityOverLocalRetryThreshold(SyncWorkflowContext.java:406)\n\tat io.temporal.internal.sync.SyncWorkflowContext.executeLocalActivity(SyncWorkflowContext.java:369)\n\tat io.temporal.common.interceptors.WorkflowOutboundCallsInterceptorBase.executeLocalActivity(WorkflowOutboundCallsInterceptorBase.java:49)\n\tat io.temporal.common.interceptors.WorkflowOutboundCallsInterceptorBase.executeLocalActivity(WorkflowOutboundCallsInterceptorBase.java:49)\n\tat xxx.XXXDiagnosticsWorkerInterceptor$XXXDiagnosticsWorkflowOutboundCallsInterceptor.executeLocalActivity(XXXDiagnosticsWorkerInterceptor.java:123)\n\tat io.temporal.internal.sync.LocalActivityStubImpl.executeAsync(LocalActivityStubImpl.java:51)\n\tat io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:39)\n\tat io.temporal.internal.sync.LocalActivityInvocationHandler.lambda$getActivityFunc$0(LocalActivityInvocationHandler.java:73)\n\tat io.temporal.internal.sync.LocalActivityInvocationHandler$$Lambda$1435/0x0000000100d22040.apply(Unknown Source)\n\tat io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:60)\n\tat com.sun.proxy.$Proxy228.xxMethod(Unknown Source)\n\tat xxx.xyzMethod(XXXWorkflowOutboundInterceptor.java:123)\n\tat xxx.XXXWorkflowOutboundInterceptor.executeActivity(XXXWorkflowOutboundInterceptor.java:12)\n\tat io.temporal.common.interceptors.WorkflowOutboundCallsInterceptorBase.executeActivity(WorkflowOutboundCallsInterceptorBase.java:44)\n\tat xxx.XXXDiagnosticsWorkerInterceptor$RequestDiagnosticsWorkflowOutboundCallsInterceptor.executeActivity(XXXDiagnosticsWorkerInterceptor.java:123)\n\tat io.temporal.internal.sync.ActivityStubImpl.executeAsync(ActivityStubImpl.java:50)\n\tat io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:39)\n\tat io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:78)\n\tat io.temporal.internal.sync.ActivityInvocationHandler$$Lambda$1432/0x0000000100d21440.apply(Unknown Source)\n\tat io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:60)\n\tat com.sun.proxy.$Proxy227.xxx(Unknown Source)\n\tat xxx.XXXWorkflowImpl.xxx(XXXWorkflowImpl.java:12)\n\tat java.base@11.0.19/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base@11.0.19/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base@11.0.19/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base@11.0.19/java.lang.reflect.Method.invoke(Method.java:566)\n\tat io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:341)\n\tat io.temporal.common.interceptors.WorkflowInboundCallsInterceptorBase.execute(WorkflowInboundCallsInterceptorBase.java:40)\n\tat xxx.XXXDiagnosticsWorkerInterceptor$XXXDiagnosticsContextWorkflowInboundCallsInterceptor.execute(XXXDiagnosticsWorkerInterceptor.java:12)\n\tat io.temporal.common.interceptors.WorkflowInboundCallsInterceptorBase.execute(WorkflowInboundCallsInterceptorBase.java:40)\n\tat xxx.XXXWorkflowInboundInterceptor.execute(XXXWorkflowInboundInterceptor.java:99)\n\tat io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:316)\n\tat io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70)\n\tat io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:128)\n\tat io.temporal.internal.sync.SyncWorkflow$$Lambda$1427/0x0000000100caac40.run(Unknown Source)\n\tat io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102)\n\tat io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:106)\n\tat io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53)\n\tat io.temporal.worker.ActiveThreadReportingExecutor$$Lambda$1424/0x0000000100cab040.run(Unknown Source)\n\tat java.base@11.0.19/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base@11.0.19/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base@11.0.19/java.lang.Thread.run(Thread.java:829)\n”,
It looks like class loading is taking over 1 second. Probably the process is constrained on CPU or IO. If this rarely happens, I wouldn’t be overly concerned as this error is retried.