We are considering a Temporal setup where we have a dedicated task queue per worker
Is there is a way to catch error of certain worker dies & reschedule the workflows that was executing to different task queue?
When rescheduling, we need the workflows to start from the first activity (even if some activities has been already executed on the dead worker)
Is there is an example, docs for such setup I can check?
Feel free to suggest a recommendation for achieving this setup
where we have a dedicated task queue per worker
Can you explain your use case for this type of need? With Temporal a workflow execution is not tied to a specific worker. Typically you want to have multiple worker processes polling on same task queue (that have your wf impl registered) in order to achieve HA.
If your workflow execution has started on worker 1 and worker 1 crashes, it can continue to worker 2 that is listening on the same task queue.
Does your use case require running activities on the same worker process? If so, please go to this link: https://github.com/temporalio/samples-go/tree/main/fileprocessing
Let me know if this helps.
Yes its about running workflow activities on the same worker, but we had issues with sessions when we deploy new versions to the worker
As graceful shutdown doesn’t wait for the the whole session to finish, Which resulted an unexpected behavior with some activities being canceled instead of being retried/continued on the the same worker (after deploying new version of the worker)
Does setting a custom identity for the worker deployed on the same machine helps? (we only deploy 1 worker per machine)
we had issues with sessions when we deploy new versions to the worker
What’s the Go SDK version you are using?
Can you give more info on the issue you ran into? Can you see if it might be related to this open issue? If not how hard would it be to create a reproducible test for it?
We don’t have a reproduce scenario yet, if we got it will let you know
We’re using sdk v1.15.0 and can upgrade for sure, Although I still see there are open issue for using sessions like the one you mentioned Unable to reproduce `workflow.ErrSessionFailed`, session gets recreated with same ID after process restarts · Issue #926 · temporalio/sdk-go · GitHub which will keep the question of this thread active (until we’re sure that session is working fine in our use case)
Is there is a way to catch a workflow error after the worker dies, to be able to route the workflow to other task queue?
I believe our fileprocessing sample does this already.