I have integrated Temporal jobs into my Django project, where workflows need to interact with the database for fetching and updating data. To run the Temporal worker, I am using manage.py as specified in the documentation: django-temporalio.
Since the worker is a long-running process that restarts weekly or monthly, it sometimes encounters issues. If an unexpected event occurs, such as a database restart or failure, the worker loses its database connection. As a result, my recurring jobs fail with errors stating that the connection is lost and cannot be re-established.
My main concern is why a new connection is not being established at the workflow level and why the workflow relies on the worker’s database connection. If a new database connection were established with each workflow execution, it would resolve this issue. Since long-running processes in Django can cause database connection problems, this seems to be affecting the Temporal worker in my case.
A Temporal activity worker is simply a running program that listens for activity tasks to do an then runs your code to perform those actions. How you write your own code, such as making database connections, is up to you. I don’t know what django-temporalio does, but you might want to use a database connection pool, or at least reopen the database connection if it closed. You can debug a problem with your activity code by writing a small standalone program which runs your activity code. For example:
open a database connection
make a request to the database
wait for you to press Enter
in a different terminal, restart the database
make another request to the database
See if the your code successfully makes a new database connection when the previous one was closed.
@awwx I am facing the same issue—on every database server restart, my worker connection is lost, causing workflow and activity task failures.
My main concern is why Temporal relies on a worker’s database connection instead of establishing a new connection during workflow execution. Would it be possible to use a connection from the pool or create a fresh database connection for each workflow execution? This could potentially resolve my issue.
Yes, I am using a database connection pool in both the worker and the entire application.