We have a Workflow that is made of several Local Activities and a few Normal Activities. For some executions, I can see that there is a loop of WorkflowTaskScheduled, WorkflowTaskStarted, WorkflowTaskCompleted events of 8 seconds. The loop repeats until the Workflow times out. I am unable to figure out what is going wrong here.
This is how the Workflow starts:
And this is where the loop starts:
It continues like this until it times out.
TLDR; your local activity is stuck
You are witnessing workflow task heartbeat in action. If a local activity takes longer than a workflow task timeout the SDK keep completing the workflow task and synchronously starting the new task with 80% of the workflow task timeout interval. In your case the default workflow task timeout is 10 seconds and the task completion happens every 8 seconds. In your case it looks like the local activity takes longer than the workflow task timeout.
Thanks, @maxim. Had a few queries:
If there are 10 local activities in a workflow task and the task timeouts after local activity#5, will the workflow task persist the 5 completed activities (as MarkerRecord) before quitting?
Are there any drawbacks if the local activity (or a series of local activities) takes longer to complete than the workflow task timeout? Is it recommended to increase the workflow task timeout in such cases?
I looked at my configuration and all the local activities have a timeout(StartToCloseTimeout) of 5 seconds. Is it possible that a local activity still gets stuck?
To be 100% sure, can I see the local activity configurations in the History tab of Web UI (I can see the configurations for workflow and normal activity, not for local activity).
- If there are 10 local activities in a workflow task and the task timeouts after local activity#5, will the workflow task persist the 5 completed activities (as MarkerRecord) before quitting?
The task doesn’t time out in this case. It is going to complete and open another one synchronously (aka workflow task heartbeat). The 5 completed are going to be recorded.
- Are there any drawbacks if the local activity (or a series of local activities) takes longer to complete than the workflow task timeout? Is it recommended to increase the workflow task timeout in such cases?
The task doesn’t timeout. It is extended through the heartbeat mechanism I described earlier. Note that it is intended for exceptional cases. Our recommendation is to use normal activities if you expect their execution to take longer than a few seconds.
I looked at my configuration and all the local activities have a timeout(StartToCloseTimeout) of 5 seconds. Is it possible that a local activity still gets stuck?
Yes, Java doesn’t support killing executing threads.
To be 100% sure, can I see the local activity configurations in the History tab of Web UI (I can see the configurations for workflow and normal activity, not for local activity).
No, I don’t think they are recorded.
