I am debugging some bug in our code.
In debugging the issue, I have found out that in some cases, the time between ActivityTaskScheduled and ActivityTaskStarted is giant (6 seconds). It seems to me that Temporal waits 6 seconds doing… something… before running the activity. (That also seems like the underlying issue of the bug.)
What could be the reason?
See the screenshot
I have looked at the logs of Temporal itself (all the sub-services) between that time and it seems like nothing out of the ordinary. It doesn’t seem like anything was wrong.
However it still took about 5-6 seconds for the activity to be run. The activity itself is very quick (a DB write) so it definitely isn’t an issue with the activity itself.
What can be wrong there? Where to even look?