How does Temporal make sure it doesn't schedule the same task twice?

Community Support
,
1

if the workers poll the task queues, how does the server not give out the same task to two different workers at the “same” time? does it use mutex locks? maybe a better question is, how does temporal make sure it doesn’t schedule the same task twice?

2

Temporal never gives a task to more than one worker. It is not only “at the same time” but even after that task has already been completed.

It is done by deduping a task before delivery. The matching engine component of the Temporal service does implement task queues. These queues indeed can deliver duplicated tasks in various failure scenarios. But just before returning the long poll call to the worker a start task call is done to the history service component of the service. As a result of this call a WorkflowTaskStarted or an ActivityTaskStartet event *depending on the task type) is recorded into the workflow history. The history service is fully consistent. So if it returns already started failure to the matching engine the task is considered a duplicate and is safely dropped.

Such design is possible by task queues not dealing with task retries and timeouts. When a history service receives the start task call for the first time, besides recording the event in the workflow history, it starts a task timeout timer. For example, for an activity task, the timeout is defined through StartToCloseTimeout option. If an activity reports a failure or the timeout fires the history service either reports the task failure to the workflow or retries the activity by scheduling a new activity task.

In this design, any failure that leads to the task not being processed can be detected only through its timeout. So it is important to set the task timeout to the lowest value that exceeds the longest activity duration. If an activity execution time is long or unpredictable the heartbeating should be used with a short heartbeat timeout.

3

@maxim thanks for the reply.

That makes sense, i have a few follow up questions:

  1. How is the history service fully consistent? I looked at the code and saw that there is some use of waitgroups and mutex locks but how does that system work when you run multiple history workers? cant you get into a spot where you have two workers working on the same start task event and both write to the cassandra database (which is eventually consistent?)
  2. Does this mean that if a client takes a start workflow task from the queue which is then recorded into history but then dies on the client side (not gracefully, the server shuts down before it can execute anything else), temporal wont retry this workflow start execution task until the StartToCloseTimeout timer fires?

Thanks again!

4
  1. Temporal relies on Cassandra lightweight transactions which support fully consistent conditional updates. All DB updates are conditional to ensure that only one process can own any given shard at a time.
  2. Yes, you are correct. A workflow task will be retried after the WorkflowTaskTimeout (which is 10 seconds by default), and an activity task retried after StartToCloseTimeout (no default for this one).