I have integrated Temporal jobs into my Django project, where workflows need to interact with the database for fetching and updating data. To run the Temporal worker, I am using manage.py as specified in the documentation: django-temporalio.
Since the worker is a long-running process that restarts weekly or monthly, it sometimes encounters issues. If an unexpected event occurs, such as a database restart or failure, the worker loses its database connection. As a result, my recurring jobs fail with errors stating that the connection is lost and cannot be re-established.
My main concern is why a new connection is not being established at the workflow level and why the workflow relies on the worker’s database connection. If a new database connection were established with each workflow execution, it would resolve this issue. Since long-running processes in Django can cause database connection problems, this seems to be affecting the Temporal worker in my case.
A Temporal activity worker is simply a running program that listens for activity tasks to do an then runs your code to perform those actions. How you write your own code, such as making database connections, is up to you. I don’t know what django-temporalio does, but you might want to use a database connection pool, or at least reopen the database connection if it closed. You can debug a problem with your activity code by writing a small standalone program which runs your activity code. For example:
open a database connection
make a request to the database
wait for you to press Enter
in a different terminal, restart the database
make another request to the database
See if the your code successfully makes a new database connection when the previous one was closed.
@awwx I am facing the same issue—on every database server restart, my worker connection is lost, causing workflow and activity task failures.
My main concern is why Temporal relies on a worker’s database connection instead of establishing a new connection during workflow execution. Would it be possible to use a connection from the pool or create a fresh database connection for each workflow execution? This could potentially resolve my issue.
Yes, I am using a database connection pool in both the worker and the entire application.
@nitin I’m sorry I didn’t see your question earlier. Are you still having this problem? When you say you’re restarting the database, are you referring to the database that the Termporal service itself is using for workflow and activity orchestration, or do you mean an application database that you’re connecting to from an activity worker?
why Temporal relies on a worker’s database connection instead of establishing a new connection during workflow execution
From original question it sounds like user creates a connection to custom db which then they pass as input to activities. Temporal worker itself does not rely on any db connections, it connects to your server frontend service exposed grpc endpoint. In client used when creating worker SDKs provide means to define grpc connect timeout to re-establish connection if there is a server failover.
Sounds like user need to be able to handle connection errors (continuously check maybe, depending on lib used) to re-establish connection.