Temporal internal corrupted state

Community Support
1

Hi, we use the tutorial Temporal docker setup. It worked for some time, but we ran into a strange issue where there is no flow open anymore (in fact after rebooting the docker instance, the history is completely empty), but the worker is picking up a workflow and erroring out internally.

Here’s the list of errors we get (unfortunately no stracktraces):

image
image1730×157 18.1 KB

image
image1713×153 16.5 KB

image
image1759×152 18.7 KB

image
image1687×156 18.4 KB

So no active/closed workflows in the dashboard or tctl, but somehow the worker is doing a replay of a very old flow (that value “newFlow”, is a very old tryout, long gone).

To find out which workflowId/runId this is about, I had to run the application in debug mode and break on the exception inside Temporal (Cadence?) code. Using this ID and running tctl wf showid yielded the following internal stacktrace:

image
image1661×521 28.9 KB

This is all gibberish to me, so I have no idea what’s going on or how to fix it. Cleaning up the workflow would solve the issue I guess. Currently the worker keeps trying to pick it up and is spamming the logs.

Any help appreciated.

2

I’m not sure how you’ve got into this situation. I would recommend resetting your docker based setup. Usually docker-compose down and then up is enough. Sometimes pruning docker though docker system prune -a is necessary.

For individual workflows terminating them through tctl workflow terminate usually helps.

Please upgrade to the latest version of Temporal. There were a few changes that addressed the broken list operation.