# Proper termination of a worker in Python in Kubernetes **URL:** \ **Category:** Community Support\ **Tags:** python-sdk, general-impl, kubernetes\ **Created:** [May 13, 2024, 5:49pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087 "2024-05-13T17:49:14Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![jefftherobot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/jefftherobot/32/5335_2.png) [@jefftherobot](https://community.temporal.io/u/jefftherobot)\ **Post date:** [May 13, 2024, 5:49pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/1 "2024-05-13T17:49:14Z")
Hi, I am testing a new worker in python that is going to be run in Kubernetes as a job, however it seems that after the temporal worker has been completed, the pod stays alive. I am trying to make this run as an Argo Workflow workflow. How should I handle this correctly? The code I have is here: ```python try: worker = Worker( self.client, task_queue=self.task_queue_name, workflows=[example_workflow], activities=[activities.example_activity], ) logging.info("started worker") await worker.run() logging.info("finished worker") sys.exit(0) except Exception as e: logging.error(f"TemporalWorker >> Failed to start the worker: {e}") if __name__ == " __main__": asyncio.run(main()) ``` i am never reaching sys.exit(0) ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [May 13, 2024, 5:55pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/2 "2024-05-13T17:55:04Z")
There are not exact approaches, this is the same as stopping any asyncio system. Some of the examples, like the [worker on the encryption one](https://github.com/temporalio/samples-python/blob/main/encryption/worker.py), show a way to support `KeyboardInterrupt` in Python to set an interrupt event which shuts down the worker. Note, this will initiate worker shutdown, but worker shutdown will wait for all outstanding activities to handle the cancellations they’ll be sent. ---
**Author:** ![jefftherobot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/jefftherobot/32/5335_2.png) [@jefftherobot](https://community.temporal.io/u/jefftherobot)\ **Post date:** [May 13, 2024, 9:22pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/3 "2024-05-13T21:22:55Z")
thank you for the clarification. would you happen to know how to kill a pod after the worker.run() is done? ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [May 13, 2024, 10:07pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/4 "2024-05-13T22:07:53Z")
`worker.run()` runs forever until you call `worker.shutdown()` or cancel the asyncio task. You can choose to exit the process then. How that affects the pod is up to you. ---
**Author:** ![jefftherobot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/jefftherobot/32/5335_2.png) [@jefftherobot](https://community.temporal.io/u/jefftherobot)\ **Post date:** [May 14, 2024, 3:27pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/5 "2024-05-14T15:27:20Z")
hi chad, so if i was say put: await worker.run() await worker.shutdown() would this help? apologies i am not one for async programming w/ python. ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [May 14, 2024, 3:34pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/6 "2024-05-14T15:34:21Z")
No, `await worker.run()` blocks until the worker is shutdown, which can occur via a shutdown call or a fatal error. ---
**Author:** ![jefftherobot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/jefftherobot/32/5335_2.png) [@jefftherobot](https://community.temporal.io/u/jefftherobot)\ **Post date:** [May 14, 2024, 7:17pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/7 "2024-05-14T19:17:29Z")
so in this case, how would you know when the worker is done? ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [May 14, 2024, 7:20pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/8 "2024-05-14T19:20:53Z")
It runs forever until you decide you want it to be done and tell it to shutdown (or it has a rare fatal error). When `await worker.run()` returns/throws, it has completed shutting down. ---
**Author:** ![Sahel\_Mastoureshgh](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/sahel_mastoureshgh/32/5353_2.png) [@Sahel\_Mastoureshgh](https://community.temporal.io/u/Sahel_Mastoureshgh)\ **Post date:** [May 16, 2024, 4:24pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/9 "2024-05-16T16:24:51Z")
Hi, our design is to start pod when the workflow is running and turn off the pod when the workflow run is finished. since based on documentation await worker.run() suppose to run worker and execute activity and workflow. Can you guide us where execute we need to put await worker.shutdown() in terms of workflow is done running. Based on my starting this code just start workflow and actucal execution happening in worker handle = await self.temporal\_client.start\_workflow( Workflow.run, args=[…], id=temporal\_workflow\_id, task\_queue=task\_queue, ) logging.info(f"Started workflow. Workflow ID: {handle.id}, RunID {handle.result\_run\_id}") return handle.id, handle.result\_run\_id ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [May 16, 2024, 4:29pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/10 "2024-05-16T16:29:51Z")
> [@Sahel\_Mastoureshgh](#): > > Can you guide us where execute we need to put await worker.shutdown() in terms of workflow is done running When you want to shutdown the worker. This is up to you. Maybe you want to shut it down on signal for example. What others do is wait on signal inside the `with` like [this example](https://github.com/temporalio/samples-python/blob/main/encryption/worker.py) shows. You don’t usually run the worker for the life of the workflow, you run it for a long time as a server/system that handles all workflows. ---
**Author:** ![Suresh](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@Suresh](https://community.temporal.io/u/Suresh)\ **Post date:** [June 11, 2024, 7:07am UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/11 "2024-06-11T07:07:27Z")
The examples (tried encryption one) seem to be not working as expected. I tried something like below in my setup (k8s, workers running as deployment with autoscaling, sync activities with custom ThreadPoolExecutor etc), this seems to be working (had sync activity that was sleeping etc, so once I sent SIGTERM, it was waiting for it to finish before exiting etc). ```auto import signal async def handler(): print("shutting down worker") await worker.shutdown() print("finished shutting down worker") if __name__ == " __main__": loop = asyncio.get_event_loop() for signame in ('SIGINT', 'SIGTERM'): loop.add_signal_handler(getattr(signal, signame), lambda: asyncio.create_task(handler())) loop.run_until_complete(main()) ``` @Chad_Retz Not sure the interaction between temporal asyncio loop and what I have done above. Does the above one look okay? anything I should be aware of? ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [June 11, 2024, 12:12pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/12 "2024-06-11T12:12:58Z")
> [@Suresh](#): > > Does the above one look okay? This seems ok (I can’t see `main()` but I assume it’s just awaiting a `worker.run()`). I can’t think of anything you need to be aware of so long as you’re properly awaiting the worker run. ---
**Author:** ![Suresh](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@Suresh](https://community.temporal.io/u/Suresh)\ **Post date:** [June 12, 2024, 7:00am UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/13 "2024-06-12T07:00:36Z")
Thanks, I have rolled out this in prod, tested this multiple times (by doing rolling restart etc), seems to be working fine so far, will keep an eye on it. For reference, here is the full code, including the main ```auto worker = None async def handler(): print("signal received, shutting down worker") await worker.shutdown() print("finished shutting down worker") async def main(): logging.basicConfig(level=logging.INFO) with open(settings.TEMPORAL_CLOUD_TLS_CERT, "rb") as f: client_cert = f.read() with open(settings.TEMPORAL_CLOUD_TLS_KEY, "rb") as f: client_private_key = f.read() client = await Client.connect( settings.TEMPORAL_CLOUD_HOST_PORT, namespace=settings.TEMPORAL_CLOUD_NAMESPACE, tls=TLSConfig( client_cert=client_cert, client_private_key=client_private_key, ), data_converter=pydantic_data_converter, ) with concurrent.futures.ThreadPoolExecutor( max_workers=settings.TEMPORAL_WORKER_MAX_THREADS ) as activity_executor: global worker worker = Worker( client, task_queue=settings.TEMPORAL_CLOUD_TASK_QUEUE, workflows=[MyWorkflow], activities=[ my_activity, ], activity_executor=activity_executor, workflow_runner=new_sandbox_runner(), graceful_shutdown_timeout=timedelta(seconds=300), ) await asyncio.gather(worker.run()) if __name__ == " __main__": loop = asyncio.get_event_loop() for signame in ("SIGINT", "SIGTERM"): loop.add_signal_handler( getattr(signal, signame), lambda: asyncio.create_task(handler()) ) loop.run_until_complete(main()) ``` ---
**Author:** ![rishabhc32](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/rishabhc32/32/7267_2.png) [@rishabhc32](https://community.temporal.io/u/rishabhc32)\ **Post date:** [May 9, 2025, 2:23pm UTC](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/14 "2025-05-09T14:23:59Z")
Thanks @Suresh [this](https://community.temporal.io/t/proper-termination-of-a-worker-in-python-in-kubernetes/12087/13) worked well.