Issue: Worker Saturation When Re-pushing Large Volume of DLQ Workflows in Self-Hosted Temporal

Community Support
, ,
1

Hi Temporal community :waving_hand:,

We are running a self-hosted Temporal setup and have implemented a custom DLQ mechanism as a safety net for workflow start failures.

Current Design

  • If a workflow fails to start / enter the Temporal server, we retry it up to 3 times.

  • If it still fails after retries, we push the workflow payload into a DLQ.

  • A scheduled process picks up all entries from the DLQ and re-publishes them to Temporal every 30 minutes (at :00 and :30).

Problem We Are Seeing

  • Sometimes, when there is a large backlog in the DLQ, a big burst of workflows gets re-pushed into Temporal at once.

  • After this happens:

    • Workers become completely unresponsive

    • No workflows are picked up or progressed

    • The system only recovers after:

      • Restarting worker pods, or

      • Scaling up the number of worker pods

This feels like some form of worker saturation, task queue backlog, or resource exhaustion, but we are not fully sure what we are missing in our design.

What We Are Looking For

We’d really appreciate guidance on:

  • What are the anti-patterns in the above DLQ + bulk re-push approach?

  • Is burst re-publishing a known problem with Temporal task queues or workers?

  • Should we be:

    • Rate-limiting workflow starts from DLQ?

    • Using separate task queues for DLQ retries?

    • Leveraging Temporal retry policies / backoff instead of external DLQ retries?

    • Tuning worker concurrency, pollers, or task queue settings?

  • Are there Temporal-native patterns for handling “workflow start failures” and delayed retries more safely?

Environment (if relevant)

  • Self-hosted Temporal

  • Kubernetes-based workers

  • Workers recover only after pod restart or horizontal scaling

Any insights, best practices, or references would be very helpful.
Thanks in advance for your support

2

Your design is not DLQ, it is rather retry queue. A well configured Temporal cluster has pretty high availability so such queue is not really needed.

I recommended “queue in front” approach when the traffic is very spiky and can the starts can overwhelm the cluster. Then queue absorbs the writes and the queue consumer calls starts using a rate limiter.