I am trying to understand why this happens.
It doesn’t affect the workflow completion.
But I am running the simple MoneyTransfer example on localhost, and I don’t understand what could be the possible reasons for a timeout. I am not even running many workflows in parallel. I just InitiateMoneyTransfer 2-3 times, and usually get a WorkflowTaskTimedOut between the withdraw and deposit boundary.
Would really appreciate a better understanding of what causes this under the hood.
Hey @animesh, WorkflowTaskTimedOut might happen in various failure situation. The most common is when a worker restarts which has workflow executions cached. Any new event for those workflow will result in a WorkflowTask being scheduled on host specific tasklist, but since there is no poller on the tasklist it results in a WorkflowTaskTimedOut and new workflow task getting scheduled on global tasklist so another worker can pick up the WorkflowTask.
Some other useful forum posts which goes into more details about WorkflowTask:
Thanks @samar, those are helpful.
What has me wondering is that I am just running this on localhost, with one workflow at a time. And I can see that the worker process did not crash / restart. So is it possible that the cached workflow executions get garbage collected ? That seems like a teeny tiny cache though 
- I can reproduce it on trying to run the workflow for the 2nd or 3rd time. Sometimes it happens on the 1st run itself.
I am passing no special JVM settings, and am running temporal via the default docker compose up.
WorkflowTaskTimeout is not expected in happy path. Do you see any error/warn logs on either client or server when this happens?
Well it is rather strange. On that particular day, it seemed to be happening quite often, but today to see logs in more details, I just can’t get this to reproduce. I’ll keep an eye out if this happens again and try to get some better logs
Please reach out if you see them again.
