Workflow Unhandled Error

Yesterday we were doing some performance tests with cadence. Most of the initial cases worked flawlessly. The only problem we got was when we tried 300 request per second and we got the following error in one of our workflows:

Event: DecisionTaskFailed

Cause: WORKFLOW_WORKER_UNHANDLED_FAILURE

Details:

"runtime error: invalid memory address or nil pointer dereference"

After that, the workflow would restart every minute from the beginning without any success.

What would the reason be for this happening?

It looks like nil pointer panic in the workflow code. Could you see the whole panic backtrace in the worker log?

If you cannot locate the backtrace then try replaying the broken workflow history locally. It would allow executing it under a debugger. To do that download the history using UI by pressing the Export button on the top right in the History view of a workflow.

Then call WorkflowReplayer.ReplayWorkflowHistoryFromJSONFile to replay the workflow logic.