From what I got so far, when a worker that has been started with
SIGTERM it will do graceful shutdown by not polling for new activities and wait a while for the in-progress activities to finish
Once this waiting time is over, Worker will be terminated & in-progress activities that was not finished will timeout based on the configured settings for
If the goal was to quickly retry the activity on different worker when the original worker shut down.
HeatbeatTimeout can be used, But what if the activity is doing blocking IO logic that will make it hard to keep sending heartbeats. is it recommended practice to use some goroutine in such activities to keep sending heartbeat?
Is there is another way (using Interceptors for example) to mark these in-progress activities as failed during the graceful shutdown to be able to retry quickly on different worker?
Instead of implementing heartbeat logic for every single activity