Proper termination of a worker in Python in Kubernetes

Hi,

I am testing a new worker in python that is going to be run in Kubernetes as a job, however it seems that after the temporal worker has been completed, the pod stays alive.

I am trying to make this run as an Argo Workflow workflow.

How should I handle this correctly?

The code I have is here:

        try:
            worker = Worker(
                self.client,
                task_queue=self.task_queue_name,
                workflows=[example_workflow],
                activities=[activities.example_activity],
            )
            logging.info("started worker")
            await worker.run()
            logging.info("finished worker")
            sys.exit(0)
        except Exception as e:
            logging.error(f"TemporalWorker >> Failed to start the worker: {e}")

if __name__ == "__main__":
    asyncio.run(main())

i am never reaching sys.exit(0)

There are not exact approaches, this is the same as stopping any asyncio system. Some of the examples, like the worker on the encryption one, show a way to support KeyboardInterrupt in Python to set an interrupt event which shuts down the worker. Note, this will initiate worker shutdown, but worker shutdown will wait for all outstanding activities to handle the cancellations they’ll be sent.

thank you for the clarification.

would you happen to know how to kill a pod after the worker.run() is done?

worker.run() runs forever until you call worker.shutdown() or cancel the asyncio task. You can choose to exit the process then. How that affects the pod is up to you.

hi chad,

so if i was say put:

await worker.run()
await worker.shutdown()

would this help? apologies i am not one for async programming w/ python.

No, await worker.run() blocks until the worker is shutdown, which can occur via a shutdown call or a fatal error.

so in this case, how would you know when the worker is done?

It runs forever until you decide you want it to be done and tell it to shutdown (or it has a rare fatal error). When await worker.run() returns/throws, it has completed shutting down.

Hi, our design is to start pod when the workflow is running and turn off the pod when the workflow run is finished. since based on documentation await worker.run() suppose to run worker and execute activity and workflow. Can you guide us where execute we need to put await worker.shutdown() in terms of workflow is done running. Based on my starting this code just start workflow and actucal execution happening in worker handle = await self.temporal_client.start_workflow(
Workflow.run,
args=[…],
id=temporal_workflow_id,
task_queue=task_queue,
)
logging.info(f"Started workflow. Workflow ID: {handle.id}, RunID {handle.result_run_id}")
return handle.id, handle.result_run_id

When you want to shutdown the worker. This is up to you. Maybe you want to shut it down on signal for example. What others do is wait on signal inside the with like this example shows.

You don’t usually run the worker for the life of the workflow, you run it for a long time as a server/system that handles all workflows.