Proper termination of a worker deployed on Kubernetes?


I am deploying my (workflow & activity) workers on Kubernetes. Today, I was wondering what the proper way is to upgrade these workers to a new version? Given Kubernetes wants to stop the old version, how can I align this with the worker process itself? My intent is this:

  1. the old worker can still finish the ongoing activity.
  2. the old worker may not pull for additional messages to process.
  3. the worker process ends gracefully and the pod is cleaned up

Aligned to it, if my workflow uses session support, can I keep my old worker alive until all the required activities for that session are executed?



I believe both Java and Go SDK support a clean shutdown that waits for activities to complete without polling for new ones. The ability to wait for the whole session completion is not currently supported. I filed a feature request to get this added.

Go SDK worker.Run waits for TERM signal which pod sends to the process on clean shutdown and then executes the clean shutdown.

In Java the main method of the application has to deal with signal handling and then use worker.shutdown and worker.awaitTermination to wait for activity completions.

1 Like