Session retry on worker restart

AhmedMozaly · October 20, 2022, 2:24pm

We have a workflow that uses session
When the worker restart for any reason (new deployment for example) we need to retry the whole workflow again on any available worker than the one that died/restarted

Instead, we see the workflow is canceled.
I could reproduce the issue with the FileProcessing example, I’ve forked here with only one change (to run 1 iteration instead of 5 to make the issue simpler to find)

github.com

AhmedMozaly/samples-go/blob/main/fileprocessing/workflow.go#L28


      
          			BackoffCoefficient: 2.0,
          			MaximumInterval:    time.Minute,
          		},
          	}
          	ctx = workflow.WithActivityOptions(ctx, ao)
          
          
	// Retry the whole sequence from the first activity on any error
          	// to retry it on a different host. In a real application it might be reasonable to
          	// retry individual activities as well as the whole sequence discriminating between different types of errors.
          	// See the retryactivity sample for a more sophisticated retry implementation.
          	// for i := 1; i < 5; i++ {
          	err = processFile(ctx, fileName)
          	// 	if err == nil {
          	// 		break
          	// 	}
          	// }
          	if err != nil {
          		workflow.GetLogger(ctx).Error("Workflow failed.", "Error", err.Error())
          	} else {
          		workflow.GetLogger(ctx).Info("Workflow completed.")
          	}

I’ve ran the sample > restarted the worker while workflow was in-progress > got these errors when the worker started again

What should be the correct configuration to let the session be retried on any available worker when the original worker dies/restarts?

16:8:58 app         | 2022/10/20 16:08:58 INFO  Starting session worker Namespace default TaskQueue fileprocessing WorkerID 67789@Ahmeds-MacBook-Pro.local@

16:8:58 app         | 2022/10/20 16:08:58 INFO  Started Worker Namespace default TaskQueue fileprocessing WorkerID 67789@Ahmeds-MacBook-Pro.local@

16:9:13 app         | 2022/10/20 16:09:13 DEBUG Session failed Namespace default TaskQueue fileprocessing WorkerID 67789@Ahmeds-MacBook-Pro.local@ WorkflowType SampleFileProcessingWorkflow WorkflowID fileprocessing_cb880727-8112-492e-b998-b76eaaa504ee RunID 8d5159a0-1f1f-413a-bfc3-b362d3233891 Attempt 1 sessionID f70a2544-1e3b-42dc-89fe-447e9027a759 Error activity error (type: internalSessionCreationActivity, scheduledEventID: 6, startedEventID: 18, identity: ): activity Heartbeat timeout (type: Heartbeat)

2022/10/20 16:09:13 DEBUG RequestCancelActivity Namespace default TaskQueue fileprocessing WorkerID 67789@Ahmeds-MacBook-Pro.local@ WorkflowType SampleFileProcessingWorkflow WorkflowID fileprocessing_cb880727-8112-492e-b998-b76eaaa504ee RunID 8d5159a0-1f1f-413a-bfc3-b362d3233891 Attempt 1 ActivityID 17

2022/10/20 16:09:13 ERROR Workflow failed. Namespace default TaskQueue fileprocessing WorkerID 67789@Ahmeds-MacBook-Pro.local@ WorkflowType SampleFileProcessingWorkflow WorkflowID fileprocessing_cb880727-8112-492e-b998-b76eaaa504ee RunID 8d5159a0-1f1f-413a-bfc3-b362d3233891 Attempt 1 Error canceled```

tihomir · October 31, 2022, 3:30am

Believe there is related feature request open here to add ability to keep session open on worker restart.

Haven’t had the chance to run your sample yet but will do and report back.

Topic		Replies	Views
Example of workflow reschedule to different task queue if worker dies Community Support go-sdk	5	724	October 8, 2022
Session recovery after work restart Community Support go-sdk	0	329	November 11, 2023
Worker does not start activity after restart Community Support go-sdk , retries , worker	17	3103	May 24, 2021
How to make the worker to log the failed task just once Community Support go-sdk	0	12	August 15, 2024
Replay of Workflow.retry() calling an Activity Community Support retries	3	846	August 13, 2021

Session retry on worker restart

Related topics