ERROR Workflow panic - Error lookup failed for scheduledEventID to activityID

but this happens without any change or even restart of worker

Before the panic you have a timer (sleep) for 3 days it seems:
Timer Started: “eventTime”: “2022-07-30T10:53:10.979829652Z”,
Timer Fired: “eventTime”: “2022-08-02T10:53:11.027287627Z”,

Workers have in-memory workflow executions cache and when your workflow execution is blocked (on a sleep for example) workers can evict it from it’s cache to allow other ones to be processed (this is optimization technique workers in all sdks have).

I think this is what may be happening here, during this long sleep worker evicted it from cache and when timer fired, worker picked up the workflow task to continue, but since this workflow exec no longer in its cache it has to replay the whole history from the start.

During this replay it seems that you might have some non-deterministic code, or this is SDK bug, check if maybe you are using system time or a lib that uses system time when you define or calculate
your sleep durations.
Try debugging using WorkflowReplayer and see if you are able to find the exact place.

If you could share your workflow code or could provide a reproducer that would help as well.

1 Like