Local Activity vs Activity

Community Support
,
3

Here is a sequence of steps to execute a single activity workflow.

  1. Workflow is started which adds a workflow task into a workflow task queue.
  2. A workflow task is received by a workflow worker listening to the workflow task queue.
  3. The workflow task is completed with ScheduleActivityTask command.
  4. An activity task is added to an activity task queue.
  5. The activity task is received by an activity worker.
  6. The activity task is completed by the activity worker.
  7. The workflow task is added to the workflow task queue.
  8. The workflow task with the result of the activity is received by a workflow worker.
  9. The workflow task is completed with CompleteWorkflowExecution command.

Here is a sequence of steps to execute a local activity.

  1. Workflow is started which adds a workflow task into a workflow task queue.
  2. A workflow task is received by a workflow worker listening to the workflow task queue.
  3. A workflow schedules a local activity task into the in-process local activity task queue.
  4. Local activity is executed by a local activity worker that listens to the in-process local activity task queue.
  5. The local activity task is completed back to the workflow.
  6. The workflow task is completed with RecordMarker and CompleteWorkflowExecution commands. The marker command contains the result of the local activity execution.

Note that steps 3, 4, and 5 are in memory of the worker and do not require any DB operations. So the number of updates in case of normal activity is 9 versus 3 in case of the local one. The difference becomes even more pronounced when multiple activities are executed in parallel.

The advantages of the local activity are clear. Here are the drawbacks:

  • It works only for short activities that do not exceed the workflow task timeout. It implies that heartbeating is not supported for local activities.
  • It is not efficient for long retries. When a retry interval exceeds the workflow task timeout a timer is used to schedule the next retry. This implies that multiple events are added to the history on each retry. Normal activities can be retried practically indefinitely.
  • Local activity has at least once semantic as a failure of a workflow task would lead to their re-execution. This includes reexecution of the whole sequences of local activities.
  • Local activity extends workflow task execution. While the task is running it cannot react to signals. So it increases the latency for signal handling.

We’ve seen multiple cases when use of local activities without understanding their limitations caused various production issues. So my advice is to stick to normal activities unless your use case requires very high throughput and large activity fanouts of very short activities.

In the long term, we plan to implement a mixed model. An activity invocation with a short timeout would start as a local activity, but in case of failure changed to a normal activity which is more efficient for long retries.

my question is what’s going to happen when a workflow has 2 local activity and 1 regular activity, and when the last regular activity fails, is entire workflow going to be retried, or only the 3rd one on a different machine?

It depends on how activities are chained together. Let’ say they are executed in a sequence. When 2 local activities complete the workflow task is completed. The completion request is going to contain three commands:

  1. RecordMarker with the result of local activity 1
  2. RecordMarker with the result of local activity 2
  3. ScheduleActivityTask for the activity 3

So when activity 3 fails the workflow knows that local activities are already executed by looking at the correspondent MarkerRecorded events and is not going to rerun them.

21 Likes