What could be the reason for ActivityTaskScheduled and ActivityTaskStarted differing 6 seconds?

Community Support
, ,
1

Hello.

I am debugging some bug in our code.

In debugging the issue, I have found out that in some cases, the time between ActivityTaskScheduled and ActivityTaskStarted is giant (6 seconds). It seems to me that Temporal waits 6 seconds doing… something… before running the activity. (That also seems like the underlying issue of the bug.)

What could be the reason?

See the screenshot

Screenshot 2023-10-17 at 18.01.18
Screenshot 2023-10-17 at 18.01.181562×286 39.2 KB

I have looked at the logs of Temporal itself (all the sub-services) between that time and it seems like nothing out of the ordinary. It doesn’t seem like anything was wrong.

However it still took about 5-6 seconds for the activity to be run. The activity itself is very quick (a DB write) so it definitely isn’t an issue with the activity itself.

What can be wrong there? Where to even look?

2

Look into the ActivityTaskStarted event. What is the value of attempts field?

3

It’s

Attempt 1

:frowning: so that cannot be the reason either.

4

I realized that at the same time this 6 second delay is happening, we query the workflow each 250 ms in a loop. To tell what state it is in.

Can that be the issue?

I thought querying workflow is “cheap”; and as the workflow is “replayed” anyway when querying, it would not matter; but, could that be the reason why is the activity so delayed?