We are using Temporal as a microservice orchestrator. The majority of the microservices take up to 3 seconds and for them we are using local activities. We are using local activities for better throughput and latency. We also retry the activities 3 times with exponential backoff. So, in the worst case, a single local activity can take more than the workflow task timeout (10 seconds). Can this be a problem?
a single local activity can take more than the workflow task timeout (10 seconds)
Local activities can extend the mentioned workflow task timeout of 10 seconds.
This can be a problem with queries if the local activity execution (which can include retries) extends the max workflow task timeout that defaults to 60 seconds (when a local activity is retried the whole workflow task is retried).
in this case you should see “context deadline exceeded” errors in your client on query call, meaning local activity blocked it causing it to time out.
This can also cause issues with delays of signal delivery to your workflow (they wont be handled until local activity completes, fails or times out).
So local activities are not really efficient for long retries nor long executions (they do not support heartbeating, can block query calls, signals). On the other hand normal activities can be retried basically forever.
This forum post is also imo very helpful.