Signal received after workflow is closed

Community Support
1

Hi team,

We encountered this issue in production.

image
image1920×880 133 KB

Take the screenshot one for example, the workflow execution timeline is like:

  1. workflow start
  2. start a timer (sleep for 3h)
  3. receive a signal 33mins later (this signal will restart the workflow via using “continueAsNew”)
  4. receive another signal 1h34mins later (this signal is also trying to restart the workflow)
  5. And then workflow started to report an exception right after second signal:
io.temporal.internal.replay.InternalWorkflowTaskException: Failure handling event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13, workflowTaskStartedEventId=17, Currently Processing StartedEventId=13} 
io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)
io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201)
io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:175)
io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:175)
io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:145)
io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:122)
io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:97)
io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:241)
io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:199)
io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:833)

Caused By: java.lang.IllegalStateException: Signal received after workflow is closed. 
io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.java:128)
io.temporal.internal.replay.ReplayWorkflowRunTaskHandler$StatesMachinesCallbackImpl.signal(ReplayWorkflowRunTaskHandler.java:323)
io.temporal.internal.statemachines.WorkflowStateMachines.handleNonStatefulEvent(WorkflowStateMachines.java:485)
io.temporal.internal.statemachines.WorkflowStateMachines.handleSingleEvent(WorkflowStateMachines.java:240)
io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:199)
io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:175)
io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:175)
io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:145)
io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:122)
io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:97)
io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:241)
io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:199)
io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:833)

Since this is uncaught exception, it will keep retry and workflow status will be stuck in “RUNNING” state. Any thought about what could cause this issue? Thanks!

2

Can you share your full history:

tctl wf show -w <wfid> -r <runid> --of myhistory.json

How many worker processes do you have for the voyager task queue? Can you make sure all of them have the same versions of workflow impls registered?

What Java SDK version are you using?

3

How many worker processes do you have for the voyager task queue?

we have 14 worker instances. And both the workflow poller and activity poller are 20. if you asking for this.

Can you make sure all of them have the same versions of workflow impls registered?

yes, this issue started to happen around 1 week ago. There is a small lombok Slf4j logging related change happened around that time. Could it be related?

What Java SDK version are you using?

we are using v1.18.0

Can you share your full history

requesting to get the permission to access temporal pods. Will send to you once I get that.

4

There is a small lombok Slf4j logging related change happened around that time. Could it be related?

Not sure but don’t think so, my guess is some non-deterministic code (check and share how you are calculating the timer duration please) or as mentioned difference if workflow code on one / some of your deployed worker processes. Yes looking at worker identities in history would help, thanks.

5 
   1  WorkflowExecutionStarted          {WorkflowType:{Name:VoyagerWorkflow}, ParentInitiatedEventId:0,                                                                       
                                        TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue, Kind:Normal},                                                   
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"name":"          
                                        minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4.147278471140              
                                        1E13,{}]}], WorkflowExecutionTimeout:0s, WorkflowRunTimeout:0s, WorkflowTaskTimeout:1m0s,                                             
                                        ContinuedExecutionRunId:fee50e59-b1ea-4bd0-8375-a700c0ba585d, Initiator:Workflow,                                                     
                                        OriginalExecutionRunId:5978ce77-4c6d-42a4-97b5-db9e4bc3e854, FirstExecutionRunId:c6616fa3-a34c-42f4-9fed-c61673214483, Attempt:1,     
                                        PrevAutoResetPoints:{Points:[]}, ParentInitiatedEventVersion:0}                                                                       
   2  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
   3  WorkflowTaskStarted               {ScheduledEventId:2,                                                                                                                  
                                        Identity:22@voyager-production-84b97ddc5f-bgz7h,                                                                                      
                                        RequestId:b70d3ddc-cc9d-4568-a11c-075e6fc444bb,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
   4  WorkflowTaskCompleted             {ScheduledEventId:2, StartedEventId:3,                                                                                                
                                        Identity:22@voyager-production-84b97ddc5f-bgz7h}                                                                                      
   5  MarkerRecorded                    {MarkerName:LocalActivity, Details:map{activityId:["b48fc72e-9002-3472-90da-4badb38d55d2"], input:["{}",                              
                                        {"temporalWorkflowId":"VOYAGER-40356734708102-USER-54252662","temporalRunId":"5978ce77-4c6d-42a4-97b5-db9e4bc3e854","wor              
                                        kflowClass":null,"subject":{"type":"USER","id":"54252662"},"eventMatcherId":null,"isExit":null,"isDebugMode":false,"task              
                                        Queue":"default","workflowDslId":40356734708102,"status":"LIFECYCLE_LIVE","runtime":"1.0","dslVersion":1}],                           
                                        time:[1666794915527], type:["InsertStartParamData"]}, WorkflowTaskCompletedEventId:4}                                                 
   6  MarkerRecorded                    {MarkerName:LocalActivity,                                                                                                            
                                        Details:map{activityId:["dd0d5cf7-5556-31ca-8425-febd8c06aea8"],                                                                      
                                        input:[40356734708102, "lifecycle", "5978ce77-4c6d-42a4-97b5-db9e4bc3e854",                                                           
                                        "START", {"type":"USER","id":"54252662"},                                                                                             
                                        "{\"start_node_id\":\"41472784711401\",\"start_node_type\":\"NODE_SLEEP\"}"],                                                         
                                        time:[1666794915527], type:["ProduceTrackingWorkflowEvent"]},                                                                         
                                        WorkflowTaskCompletedEventId:4}                                                                                                       
   7  MarkerRecorded                    {MarkerName:LocalActivity, Details:map{activityId:["dad72b48-4a2a-340e-bcff-b43bd29c6f9a"],                                           
                                        input:[{"workflowExecutionMetadataInput":{"status":"START","latestNodeId":41472784711401}},                                           
                                        {"temporalWorkflowId":"VOYAGER-40356734708102-USER-54252662","temporalRunId":"5978ce77-4c6d-42a4-97b5 ...                             
                                        },"eventMatcherId":null,"isExit":null,"isDebugMode":false,"taskQueue":"default","workflowDslId":40356734708102,"status":              
                                        "LIFECYCLE_LIVE","runtime":"1.0","dslVersion":1}], result:[{}], time:[1666794915537],                                                 
                                        type:["WORKFLOW_EXECUTION_METADATA_UPDATE"]}, WorkflowTaskCompletedEventId:4}                                                         
   8  MarkerRecorded                    {MarkerName:LocalActivity, Details:map{activityId:["c17283c1-079a-3a0e-817d-22e6d35aebe0"],                                           
                                        input:[40356734708102, "lifecycle", "5978ce77-4c6d-42a4-97b5-db9e4bc3e854", "START",                                                  
                                        {"type":"USER","id":"54252662"}, {"id":41472784711401,"viewData":{"x":593.0,"y":288.0,"title":""},"type":"NO ...                      
                                        :true,"delta":10800000},"transition":41023883535088,"paramInput":[]}},"endNode":false,"workflowId":40356734708102,"creat              
                                        ionTimeMs":1663875761516,"lastUpdatedTimeMs":1665626246858}, null, "{}"], time:[1666794915537],                                       
                                        type:["ProduceTrackingNodeEvent"]}, WorkflowTaskCompletedEventId:4}                                                                   
   9  MarkerRecorded                    {MarkerName:LocalActivity, Details:map{activityId:["a9873ead-f034-325f-b4d7-9faabb97a9a3"],                                           
                                        input:[{"isAfter":true,"delta":10800000}, {"value":[]},                                                                               
                                        {"temporalWorkflowId":"VOYAGER-40356734708102-USER-54252662","temporalRunId":"5978ce77-4c6d-42a4-97b5-db9e4bc3e854","wor              
                                        kflowClass":null, ...                                                                                                                 
                                        ,"id":"54252662"},"eventMatcherId":null,"isExit":null,"isDebugMode":false,"taskQueue":"default","workflowDslId":40356734              
                                        708102,"status":"LIFECYCLE_LIVE","runtime":"1.0","dslVersion":1}], result:[10800000], time:[1666794915537],                           
                                        type:["GetSleepTime"]}, WorkflowTaskCompletedEventId:4}                                                                               
  10  TimerStarted                      {TimerId:24632a77-2520-3843-9f2f-35254909f6e7,                                                                                        
                                        StartToFireTimeout:3h0m0s,                                                                                                            
                                        WorkflowTaskCompletedEventId:4}                                                                                                       
  11  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-84b97ddc5f-jgwdr}                                                                                      
  12  WorkflowTaskScheduled             {TaskQueue:{Name:22@voyager-production-84b97ddc5f-bgz7h:cc39343d-a790-43e0-a9b5-967ff165839f,                                         
                                        Kind:Sticky}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  13  WorkflowTaskStarted               {ScheduledEventId:12,                                                                                                                 
                                        Identity:cc39343d-a790-43e0-a9b5-967ff165839f,                                                                                        
                                        RequestId:2d76b2d9-dc36-433e-a3f7-0c4df8a2d5b9,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  14  WorkflowTaskCompleted             {ScheduledEventId:12, StartedEventId:13,                                                                                              
                                        Identity:22@voyager-production-84b97ddc5f-bgz7h}                                                                                      
  15  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-84b97ddc5f-xmzs5}                                                                                      
  16  WorkflowTaskScheduled             {TaskQueue:{Name:22@voyager-production-84b97ddc5f-bgz7h:cc39343d-a790-43e0-a9b5-967ff165839f,                                         
                                        Kind:Sticky}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  17  WorkflowTaskStarted               {ScheduledEventId:16,                                                                                                                 
                                        Identity:cc39343d-a790-43e0-a9b5-967ff165839f,                                                                                        
                                        RequestId:6633e950-414f-421e-9f58-1307d3d653bc,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  18  WorkflowTaskFailed                {ScheduledEventId:16, StartedEventId:17, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=17, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-84b97ddc5f-bgz7h, ForkEventVersion:0}                                                                  
  19  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  20  WorkflowTaskStarted               {ScheduledEventId:19,                                                                                                                 
                                        Identity:22@voyager-production-84b97ddc5f-sdh2d,                                                                                      
                                        RequestId:7e3e74ea-a0bd-4e62-a592-f2fbab4520bc,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  21  WorkflowTaskFailed                {ScheduledEventId:19, StartedEventId:20, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=20, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-84b97ddc5f-sdh2d, ForkEventVersion:0}                                                                  
  22  TimerFired                        {TimerId:24632a77-2520-3843-9f2f-35254909f6e7,                                                                                        
                                        StartedEventId:10}                                                                                                                    
  23  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  24  WorkflowTaskStarted               {ScheduledEventId:23,                                                                                                                 
                                        Identity:22@voyager-production-84b97ddc5f-bgz7h,                                                                                      
                                        RequestId:f653131f-0dc8-4427-b3b8-d26c2f8279b8,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  25  WorkflowTaskFailed                {ScheduledEventId:23, StartedEventId:24, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=24, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-84b97ddc5f-bgz7h, ForkEventVersion:0}                                                                  
  26  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-86f577bcd9-jz77b}                                                                                      
  27  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  28  WorkflowTaskStarted               {ScheduledEventId:27,                                                                                                                 
                                        Identity:22@voyager-production-86f577bcd9-t8c78,                                                                                      
                                        RequestId:aca246c0-79dd-4a65-a7c4-f2bcd7c601b6,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  29  WorkflowTaskFailed                {ScheduledEventId:27, StartedEventId:28, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=28, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-86f577bcd9-t8c78, ForkEventVersion:0}                                                                  
  30  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-86f577bcd9-jz77b}
6 
  31  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  32  WorkflowTaskStarted               {ScheduledEventId:31,                                                                                                                 
                                        Identity:22@voyager-production-86f577bcd9-mw9c2,                                                                                      
                                        RequestId:69baf87a-5038-402c-8511-d85ce34fee30,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  33  WorkflowTaskFailed                {ScheduledEventId:31, StartedEventId:32, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=32, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-86f577bcd9-mw9c2, ForkEventVersion:0}                                                                  
  34  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-86f577bcd9-lbbnq}                                                                                      
  35  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  36  WorkflowTaskStarted               {ScheduledEventId:35,                                                                                                                 
                                        Identity:22@voyager-production-86f577bcd9-jz77b,                                                                                      
                                        RequestId:fb320232-8073-4fb3-a8b6-2956acea1606,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  37  WorkflowTaskFailed                {ScheduledEventId:35, StartedEventId:36, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=36, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-86f577bcd9-jz77b, ForkEventVersion:0}                                                                  
  38  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-86f577bcd9-lbbnq}                                                                                      
  39  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  40  WorkflowTaskStarted               {ScheduledEventId:39,                                                                                                                 
                                        Identity:23@voyager-production-86f577bcd9-2tdml,                                                                                      
                                        RequestId:6a979b19-e800-4db2-8640-1e1bc61d0875,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  41  WorkflowTaskFailed                {ScheduledEventId:39, StartedEventId:40, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=40, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:23@voyager-production-86f577bcd9-2tdml, ForkEventVersion:0}                                                                  
  42  WorkflowExecutionCancelRequested  {ExternalInitiatedEventId:0,                                                                                                          
                                        Identity:23@voyager-production-86f577bcd9-2tdml}                                                                                      
  43  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  44  WorkflowTaskStarted               {ScheduledEventId:43,                                                                                                                 
                                        Identity:22@voyager-production-86f577bcd9-6qtjl,                                                                                      
                                        RequestId:9aa984bd-7cc8-43ba-8d62-bf45c649d9f3,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  45  WorkflowTaskFailed                {ScheduledEventId:43, StartedEventId:44, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=44, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-86f577bcd9-6qtjl, ForkEventVersion:0}                                                                  
  46  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-86f577bcd9-6qtjl}                                                                                      
  47  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  48  WorkflowTaskStarted               {ScheduledEventId:47,                                                                                                                 
                                        Identity:22@voyager-production-86f577bcd9-t8c78,                                                                                      
                                        RequestId:07711fa2-faf9-426d-a4bb-07bfd1b53eba,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  49  WorkflowTaskFailed                {ScheduledEventId:47, StartedEventId:48, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=48, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-86f577bcd9-t8c78, ForkEventVersion:0}                                                                  
  50  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-86f577bcd9-6qtjl}                                                                                      
  51  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  52  WorkflowTaskStarted               {ScheduledEventId:51,                                                                                                                 
                                        Identity:23@voyager-production-86f577bcd9-mnwxg,                                                                                      
                                        RequestId:31644840-9b23-46c0-9303-a356c9134224,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  53  WorkflowTaskFailed                {ScheduledEventId:51, StartedEventId:52, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=52, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:23@voyager-production-86f577bcd9-mnwxg, ForkEventVersion:0}                                                                  
  54  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-86f577bcd9-xwkrc}                                                                                      
  55  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}
7 
  56  WorkflowTaskStarted               {ScheduledEventId:55,                                                                                                                 
                                        Identity:22@voyager-production-86f577bcd9-mw9c2,                                                                                      
                                        RequestId:41527f48-460f-4cd2-b315-a6962a95296f,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  57  WorkflowTaskFailed                {ScheduledEventId:55, StartedEventId:56, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=56, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-86f577bcd9-mw9c2, ForkEventVersion:0}                                                                  
  58  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-6b685545fd-9qk6f}                                                                                      
  59  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  60  WorkflowTaskStarted               {ScheduledEventId:59,                                                                                                                 
                                        Identity:22@voyager-production-6b685545fd-tgwsj,                                                                                      
                                        RequestId:7420f51e-ba0d-4f4d-8559-d177929026b2,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  61  WorkflowTaskFailed                {ScheduledEventId:59, StartedEventId:60, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=60, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-6b685545fd-tgwsj, ForkEventVersion:0}                                                                  
  62  WorkflowExecutionSignaled         {SignalName:restart,                                                                                                                  
                                        Input:[{"payloads":null,"converter":null,"values":["{\"id\":40356734708102,\"dslVersion\":\"1\",\"runtime\":\"1.0\",\"version\        
                                        ":1,\"name\":\"test abandon - mtl\",\"ownership\":{\"ownerId\":229170694,\"editorIds\":[]},\"taxonomy\":{\"description\":\"N/A\",\    
                                        ... 3883535088":{"paramForInputs":[],"paramFromCriterion":[{"name":"isP2P3AbandonedCorridorOpen"},{"name":"p4placeId"},{"nam          
                                        e":"minutesSinceAccountCreation"},{"name":"placeId"},{"name":"userInDisasterResponseArea"}],"flowParams":[]}},4147278471 1401,{}]}],  
                                        Identity:22@voyager-production-6b685545fd-rfmjq}                                                                                      
  63  WorkflowTaskScheduled             {TaskQueue:{Name:voyager-prod-high-qps-promotional.voyager_test_queue,                                                                
                                        Kind:Normal}, StartToCloseTimeout:1m0s, Attempt:1}                                                                                    
  64  WorkflowTaskStarted               {ScheduledEventId:63,                                                                                                                 
                                        Identity:22@voyager-production-6b685545fd-z697q,                                                                                      
                                        RequestId:cd2d6534-5810-4a5e-966f-fd6c0cbf6dcf,                                                                                       
                                        SuggestContinueAsNew:false, HistorySizeBytes:0}                                                                                       
  65  WorkflowTaskFailed                {ScheduledEventId:63, StartedEventId:64, Cause:Unspecified, Failure:{Message:Failure handling                                         
                                        event 15 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED' during replay. {PreviousStartedEventId=13,                                  
                                        workflowTaskStartedEventId=64, Currently Processing StartedEventId=13}, Source:JavaSDK,                                               
                                        StackTrace:io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:222)    
                                         io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:201) io.temporal.i             
                                        nternal.st ... TaskExecutor.java:93) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)        
                                        java.bas e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.     
                                        java:833) , Cause:{Message:Signal received after workflow is closed., Source:JavaSDK,                                                 
                                        StackTrace:io.temporal.internal.replay.ReplayWorkflowExecutor.handleWorkflowExecutionSignaled(ReplayWorkflowExecutor.jav              
                                        a:128) io.t emporal.internal.replay.ReplayWorkflowRunTaskHandl ... .java:1136) java.bas                                               
                                        e/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.              
                                         java:833) , FailureInfo:{ApplicationFailureInfo:{Type:java.lang.IllegalStateException, NonRetryable:false}}},                        
                                        FailureInfo:{ApplicationFailureInfo:{Type:io.temporal.internal.replay.InternalWorkflowTaskException, NonRetryable:false}}},           
                                        Identity:22@voyager-production-6b685545fd-z697q, ForkEventVersion:0}
8

check and share how you are calculating the timer duration please

3 hour timer duration is fetching from workflow DSL, which is fixed before workflow starts to run.

9

Thanks for the info, from that tctl command, you shared the output of the command but it also created the json file myhistory.json, could you share it instead (its easier to read and contains more info).

10

Could you also show results of

tctl tq desc --tq voyager-prod-high-qps-promotional.voyager_test_queue

can you see if you see poller identity:
22@voyager-production-84b97ddc5f-bgz7h ?

11

Tried to upload the json file but it complained “Sorry, the file you are trying to upload is not authorized (authorized extensions: jpg, jpeg, png, gif, heic, heif, webp).”

12 
root@temporal-bastion-voyager-production-c6d7db597-dwk4g:/# tctl --ns voyager-prod-high-qps-promotional tq desc --tq voyager-prod-high-qps-promotional.voyager_test_queue
         WORKFLOW POLLER IDENTITY        |   LAST ACCESS TIME    
  22@voyager-production-5bc586555d-dthwd | 2022-11-02T00:08:32Z  
  22@voyager-production-5bc586555d-4fdx5 | 2022-11-02T00:08:32Z  
  23@voyager-production-5bc586555d-b2zbn | 2022-11-02T00:08:32Z  
  22@voyager-production-5bc586555d-gttdv | 2022-11-02T00:08:32Z  
  22@voyager-production-5bc586555d-vm48j | 2022-11-02T00:08:32Z  
  22@voyager-production-5bc586555d-q2tns | 2022-11-02T00:08:32Z  
  22@voyager-production-5bc586555d-pxqkv | 2022-11-02T00:08:32Z  
  22@voyager-production-5bc586555d-r2nvk | 2022-11-02T00:08:31Z  
  23@voyager-production-5bc586555d-6pjh2 | 2022-11-02T00:08:31Z  
  22@voyager-production-5bc586555d-bknvc | 2022-11-02T00:08:31Z  
  22@voyager-production-5bc586555d-8nsnr | 2022-11-02T00:08:31Z  
  22@voyager-production-5bc586555d-5gwz2 | 2022-11-02T00:08:31Z  
  23@voyager-production-5bc586555d-l6rxv | 2022-11-02T00:08:31Z  
  22@voyager-production-5bc586555d-xk8gn | 2022-11-02T00:08:31Z

Our infra will evict pods older than a certain age so suppose it is expected to not see 22@voyager-production-84b97ddc5f-bgz7h here I guess.

13

Just update - we looked at code and had a suggestion on not calling continueasnew inside signal handler but do

Workflow.await(duration, () -> condition);
if(condition) { continueasnew }

instead in workfow code(and have signal handler just update consition). will try to create a test that reproduces this and have eng look at it to see if this is a bug in sdk or not. Will provide info as soon as possible.

1 Like
14

Thank you Tihomir for all the help! This is helpful.

15

@tihomir Inspired from this post, do you think this may can help?

Currently, what we used is io.temporal.workflow.Workflow.continueAsNew(args);

Changed to:

MyWorkflow next = io.temporal.workflow.Workflow.newContinueAsNewStub(MyWorkflow.class);
WorkflowStub.fromTyped(next).start(args);
16

WorkflowStub.fromTyped(next).start(args);

Don’t think this will work, should throw:
java.lang.IllegalArgumentException: arguments must be created through WorkflowClient.newWorkflowStub

You should be able to do

MyWorkflow next = io.temporal.workflow.Workflow.newContinueAsNewStub(MyWorkflow.class);
next.myWorkflowMethod(args);
1 Like