The key here, it’s happen periodically.
We have setup which runs many cronjobs (implemented close to example from samples) but runs it through child workflow (because we have registered workflow which adds some logic to every child workflow). Basically only difference is that all workflows routed through same function to add extra logic shared between all workflows
Basically it’s like that
worker ->
execute()
<some arbitrary logic common to all tasks>
<runs child workflow>
<runs activity specific to task>
All cron tasks are ran every minute. It’s all runs fine 99% of time aside from times when it fails. Basically, problematic activity ran by child ends up either fine or fail, but it’s happen quickly (milliseconds to seconds) and then it’s timeouts (after a minute) instead of notifying workflow about error which happened at the very start of a minute. Per logs it’s apparent that we have an end in activity, but for some reason child activity never communicates to child workflow, or child workflow never communicates to parent workflow, I have no idea.
One thing I noticed digging through logs is that error “Cached state staled, new task has unexpected events” always precedes timeout (but not vice versa, so it’s not direct correlation) even if it occur for different cronjob