Panic occurs for multiple changeID

Panic occurs if we have multiple changeID inside the same workflow. An abstract example is below:

	versionStepA := workflow.GetVersion(ctx, "stepA", workflow.DefaultVersion, 1)
	var state someState
	if versionWorkflowA == workflow.DefaultVersion {
		state = getState1()
	} else {
		state = getState3()
	}

	versionStepB := workflow.GetVersion(ctx, "stepB", workflow.DefaultVersion, 1)
	var (
		secondState  someState 
		thirdState someState 
	)
	if versionStepB  == workflow.DefaultVersion {
		secondState  = getState2()
		thirdState = getState2()
	} else {
		secondState  = getState4()
		thirdState = getState4()
	}

How to reproduce:

  1. Start workflow
  2. Restart of Go process
  3. Try to finish workflow

After that, you get following error:

2021/06/09 14:38:30 ERROR Workflow panic Namespace clients TaskQueue ClientServiceTaskList WorkerID 867130@archlinux@ WorkflowType ClassicOnboardingCFD WorkflowID OUR_WORKFLOW_ID RunID a0a07560-fca3-4744-bbb0-68f8b7146de8 Attempt 1 Error lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9 StackTrace process event for ClientServiceTaskList [panic]:

Everything works well if delete the second changeID and call the latest version without conditional statement or completely delete all versions and directly call the latest version of steates.

Can you also share workflow execution history?

[{“eventId”:“1”,“eventTime”:{“seconds”:“1623253200”,“nanos”:68319442},“eventType”:“EVENT_TYPE_WORKFLOW_EXECUTION_STARTED”,“version”:“0”,“taskId”:“1048576”,“workflowExecutionStartedEventAttributes”:{“workflowType”:{“name”:“ClassicOnboardingCFD”},“parentWorkflowNamespace”:“clients”,“parentWorkflowExecution”:{“workflowId”:“1623253183015/SignUpWeb”,“runId”:“586f4e63-3455-43f8-b33b-e8516fffa758”},“parentInitiatedEventId”:“44”,“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjMyNTMxODMwMTUi”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlJFVEFJTCI=”}]},“workflowExecutionTimeout”:{“seconds”:“0”,“nanos”:0},“workflowRunTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskTimeout”:{“seconds”:“10”,“nanos”:0},“continuedExecutionRunId”:"",“initiator”:“CONTINUE_AS_NEW_INITIATOR_UNSPECIFIED”,“originalExecutionRunId”:“e49e4fb5-fc98-4a8f-8759-b9b8a335d7b9”,“identity”:"",“firstExecutionRunId”:“e49e4fb5-fc98-4a8f-8759-b9b8a335d7b9”,“attempt”:1,“cronSchedule”:"",“firstWorkflowTaskBackoff”:{“seconds”:“0”,“nanos”:0},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFBQXkycGRLbkFLUyt2dE9kMWh5bXp3QVRadXF0ejBtQ3A1QWdBPSJ9”}}}},“attributes”:“workflowExecutionStartedEventAttributes”},{“eventId”:“2”,“eventTime”:{“seconds”:“1623253200”,“nanos”:74663475},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048579”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“3”,“eventTime”:{“seconds”:“1623253200”,“nanos”:78026976},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048582”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“2”,“identity”:“980282@archlinux@”,“requestId”:“01c425df-1e2a-467a-b752-c6fee1ae0e08”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“4”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82759474},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048585”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“2”,“startedEventId”:“3”,“identity”:“980282@archlinux@”,“binaryChecksum”:“f9add7201b68e897cd0239007217d4a1”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“5”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82784374},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048586”,“markerRecordedEventAttributes”:{“details”:{“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjIwMjEwNS1icm9rZXJBZ3JlZW1lbnQi”}]},“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“6”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82809294},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048587”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“WyIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“7”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82813214},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048588”,“markerRecordedEventAttributes”:{“details”:{“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]},“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“InZlcmlmeVByb2ZpbGUi”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“8”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82823644},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048589”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“WyJ2ZXJpZnlQcm9maWxlLTEiLCIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“9”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82831274},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048590”,“activityTaskScheduledEventAttributes”:{“activityId”:“9”,“activityType”:{“name”:“GetStatusFlagActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFBQXkycGRLbkFLUyt2dE9kMWh5bXp3QVRadXF0ejBtQ3A1QWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjMyNTMxODMwMTUi”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlBlcnNvbmFsRGV0YWlscyI=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“60”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“4”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:5}},“attributes”:“activityTaskScheduledEventAttributes”},{“eventId”:“10”,“eventTime”:{“seconds”:“1623253200”,“nanos”:86185975},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_STARTED”,“version”:“0”,“taskId”:“1048597”,“activityTaskStartedEventAttributes”:{“scheduledEventId”:“9”,“identity”:“980282@archlinux@”,“requestId”:“53104e20-0c03-4966-8e4b-0a9b3895a9f7”,“attempt”:1},“attributes”:“activityTaskStartedEventAttributes”},{“eventId”:“11”,“eventTime”:{“seconds”:“1623253200”,“nanos”:89233977},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048598”,“activityTaskCompletedEventAttributes”:{“result”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“ZmFsc2U=”}]},“scheduledEventId”:“9”,“startedEventId”:“10”,“identity”:“980282@archlinux@”},“attributes”:“activityTaskCompletedEventAttributes”},{“eventId”:“12”,“eventTime”:{“seconds”:“1623253200”,“nanos”:89240407},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048601”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“archlinux:4d64965f-b931-457a-806b-619b66a3cfca”,“kind”:“TASK_QUEUE_KIND_STICKY”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“13”,“eventTime”:{“seconds”:“1623253200”,“nanos”:93012957},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048605”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“12”,“identity”:“980282@archlinux@”,“requestId”:“04c41b3b-b4d8-4a43-af02-8700aec63bdf”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“14”,“eventTime”:{“seconds”:“1623253200”,“nanos”:97254986},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048608”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“12”,“startedEventId”:“13”,“identity”:“980282@archlinux@”,“binaryChecksum”:“f9add7201b68e897cd0239007217d4a1”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“15”,“eventTime”:{“seconds”:“1623253200”,“nanos”:97275436},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048609”,“activityTaskScheduledEventAttributes”:{“activityId”:“ProvidePersonalDetailsActivity”,“activityType”:{“name”:“ProvidePersonalDetailsActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFBQXkycGRLbkFLUyt2dE9kMWh5bXp3QVRadXF0ejBtQ3A1QWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJjYXRlZ29yeSI6ImNsaWVudC1wcm9jZXNzLXN0YXRlIiwiVVVJRCI6ImMzMGUxazRubXZsZmk3azcwanVnIiwic291cmNlIjoiMTYyMzI1MzE4MzAxNS9DbGFzc2ljT25ib2FyZGluZ0NGRCIsIm1lc3NhZ2UiOiJ7XCJQSURcIjpcIkNsYXNzaWNPbmJvYXJkaW5nQ0ZEXCIsXCJDbGllbnRJRFwiOlwiMTYyMzI1MzE4MzAxNVwiLFwiU3RlcFJ1bklEXCI6XCJcIixcIlN0ZXBFcnJvclwiOlwiXCIsXCJTdGVwc0xpc3RcIjpbe1wiTmFtZVwiOlwiUGVyc29uYWxEZXRhaWxzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfSx7XCJOYW1lXCI6XCJSZXNpZGVudGlhbEFkZHJlc3NcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVGF4ZXNDb3VudHJ5XCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRheElEXCIsXCJDYW5Ta2lwXCI6dHJ1ZSxcIkNhblNhdmVcIjpmYWxzZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlNvdXJjZU9mRnVuZHNcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiY2ZkQU1MXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRyYWRpbmdBY2NvdW50UHJlZmVyZW5jaWVzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIkJyb2tlckFncmVlbWVudFwiLFwiQ2FuU2tpcFwiOmZhbHNlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVmVyaWZ5SWRlbnRpdHlcIixcIkNhblNraXBcIjp0cnVlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOmZhbHNlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlZlcmlmeVBPUlwiLFwiQ2FuU2tpcFwiOnRydWUsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfV0sXCJQb3NpdGlvblwiOjAsXCJEb25lXCI6ZmFsc2UsXCJJc1Byb2Nlc3NpbmdcIjpmYWxzZSxcIlN0YXJ0ZWRcIjpmYWxzZSxcIlJlZGlyZWN0XCI6bnVsbCxcIlJlc3VsdFwiOm51bGx9In0=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“30672000”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“14”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:0}},“attributes”:“activityTaskScheduledEventAttributes”},{“eventId”:“16”,“eventTime”:{“seconds”:“1623253200”,“nanos”:100716157},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_STARTED”,“version”:“0”,“taskId”:“1048614”,“activityTaskStartedEventAttributes”:{“scheduledEventId”:“15”,“identity”:“980282@archlinux@”,“requestId”:“ebeba944-167d-48d3-b4a0-ddecaeff23bd”,“attempt”:1},“attributes”:“activityTaskStartedEventAttributes”},{“eventId”:“17”,“eventTime”:{“seconds”:“1623253226”,“nanos”:637457577},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048615”,“activityTaskCompletedEventAttributes”:{“result”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJDb21tYW5kIjoiU0FWRSIsIkRhdGEiOnsiRmlyc3ROYW1lIjoiYXNkZmEiLCJMYXN0TmFtZSI6InNkZmFzZCIsIkRhdGVPZkJpcnRoIjoiMTExMS0xMS0xMSIsIkdlbmRlciI6Im1hbGUiLCJOYXRpb25hbGl0eUNvZGUiOm51bGwsIk5vblVTIjpmYWxzZSwiTm9uUHJvU3Vic2NyaWJlciI6bnVsbH19”}]},“scheduledEventId”:“15”,“startedEventId”:“16”,“identity”:“845434@archlinux@”},“attributes”:“activityTaskCompletedEventAttributes”},{“eventId”:“18”,“eventTime”:{“seconds”:“1623253226”,“nanos”:637471197},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048618”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“archlinux:4d64965f-b931-457a-806b-619b66a3cfca”,“kind”:“TASK_QUEUE_KIND_STICKY”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“19”,“eventTime”:{“seconds”:“1623253231”,“nanos”:638920473},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_TIMED_OUT”,“version”:“0”,“taskId”:“1048622”,“workflowTaskTimedOutEventAttributes”:{“scheduledEventId”:“18”,“startedEventId”:“0”,“timeoutType”:“TIMEOUT_TYPE_SCHEDULE_TO_START”},“attributes”:“workflowTaskTimedOutEventAttributes”},{“eventId”:“20”,“eventTime”:{“seconds”:“1623253231”,“nanos”:638933873},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048623”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“21”,“eventTime”:{“seconds”:“1623253231”,“nanos”:643120822},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048626”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“20”,“identity”:“980857@archlinux@”,“requestId”:“9f9268e6-f106-4eda-91d1-a1dc4231db2f”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“22”,“eventTime”:{“seconds”:“1623253231”,“nanos”:648673099},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_FAILED”,“version”:“0”,“taskId”:“1048629”,“workflowTaskFailedEventAttributes”:{“scheduledEventId”:“20”,“startedEventId”:“21”,“cause”:“WORKFLOW_TASK_FAILED_CAUSE_WORKFLOW_WORKER_UNHANDLED_FAILURE”,“failure”:{“message”:“lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9”,“source”:“GoSDK”,“stackTrace”:“process event for ClientServiceTaskList [panic]:\ngo.temporal.io/sdk/internal.panicIllegalState(0xc00004e9c0, 0x53)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_decision_state_machine.go:393 +0x66\ngo.temporal.io/sdk/internal.(*commandsHelper).handleActivityTaskScheduled(0xc00069a480, 0x28ac108, 0x1, 0x9)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_decision_state_machine.go:878 +0x265\ngo.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc000c104e0, 0xc0000aac00, 0x1, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_event_handlers.go:800 +0x6fa\ngo.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc00011a240, 0xc00092f110, 0x0, 0x0, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_handlers.go:876 +0xc94\ngo.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc0009b24d0, 0xc00092f110, 0xc000507e90, 0x0, 0x0, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_handlers.go:727 +0x6af\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc00064a4e0, 0xc00092f110, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_pollers.go:288 +0x35a\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc00064a4e0, 0x1b52d40, 0xc00092f110, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_pollers.go:259 +0x107\ngo.temporal.io/sdk/internal.(*baseWorker).processTask(0xc000536ff0, 0x1b52d40, 0xc00092f110)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_worker_base.go:343 +0x19a\ncreated by go.temporal.io/sdk/internal.(*baseWorker).runTaskDispatcher\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_worker_base.go:270 +0x365”,“applicationFailureInfo”:{“type”:“PanicError”,“nonRetryable”:true},“failureInfo”:“applicationFailureInfo”},“identity”:“980857@archlinux@”,“baseRunId”:"",“newRunId”:"",“forkEventVersion”:“0”,“binaryChecksum”:“f9add7201b68e897cd0239007217d4a1”},“attributes”:“workflowTaskFailedEventAttributes”}]

Looks like you are using Go SDK v1.5.0. Can you try to reproduce it with v1.6.0? We have some fixes for similar problems in that release.

There was temporalio/auto-setup:1.6.5 .
Here logs for temporalio/auto-setup:1.9.2

[{"eventId":"1","eventTime":{"seconds":"1623309980","nanos":9848200},"eventType":"EVENT_TYPE_WORKFLOW_EXECUTION_STARTED","version":"0","taskId":"1048693","workflowExecutionStartedEventAttributes":{"workflowType":{"name":"ClassicOnboardingCFD"},"parentWorkflowNamespace":"clients","parentWorkflowExecution":{"workflowId":"1623309979654/SignUpWeb","runId":"f98e40b0-e7b3-4fb7-9ab9-901d99b853fa"},"parentInitiatedEventId":"33","taskQueue":{"name":"ClientServiceTaskList","kind":"TASK_QUEUE_KIND_NORMAL"},"input":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"IjE2MjMzMDk5Nzk2NTQi"},{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"IlJFVEFJTCI="}]},"workflowExecutionTimeout":{"seconds":"0","nanos":0},"workflowRunTimeout":{"seconds":"0","nanos":0},"workflowTaskTimeout":{"seconds":"10","nanos":0},"continuedExecutionRunId":"","initiator":"CONTINUE_AS_NEW_INITIATOR_UNSPECIFIED","originalExecutionRunId":"a6c5d1c6-e3d5-417c-9911-70c461b99af8","identity":"","firstExecutionRunId":"a6c5d1c6-e3d5-417c-9911-70c461b99af8","attempt":1,"cronSchedule":"","firstWorkflowTaskBackoff":{"seconds":"0","nanos":0},"header":{"fields":{"opencensusPayload":{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"eyJEYXRhIjoiQUFCOUJhR1AwSlZzM0lKdWQ2RDFGdUhPQWFOU2RBbm5zMEpxQWdBPSJ9"}}}},"attributes":"workflowExecutionStartedEventAttributes"},{"eventId":"2","eventTime":{"seconds":"1623309980","nanos":15032459},"eventType":"EVENT_TYPE_WORKFLOW_TASK_SCHEDULED","version":"0","taskId":"1048696","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"ClientServiceTaskList","kind":"TASK_QUEUE_KIND_NORMAL"},"startToCloseTimeout":{"seconds":"10","nanos":0},"attempt":1},"attributes":"workflowTaskScheduledEventAttributes"},{"eventId":"3","eventTime":{"seconds":"1623309980","nanos":17678533},"eventType":"EVENT_TYPE_WORKFLOW_TASK_STARTED","version":"0","taskId":"1048699","workflowTaskStartedEventAttributes":{"scheduledEventId":"2","identity":"1967065@archlinux@","requestId":"5845a024-2aed-4a4f-acb4-816279b9fab9"},"attributes":"workflowTaskStartedEventAttributes"},{"eventId":"4","eventTime":{"seconds":"1623309982","nanos":254052994},"eventType":"EVENT_TYPE_WORKFLOW_TASK_COMPLETED","version":"0","taskId":"1048702","workflowTaskCompletedEventAttributes":{"scheduledEventId":"2","startedEventId":"3","identity":"1967065@archlinux@","binaryChecksum":"ad8af47b2d94e32093fb384f93ca4720"},"attributes":"workflowTaskCompletedEventAttributes"},{"eventId":"5","eventTime":{"seconds":"1623309982","nanos":254087154},"eventType":"EVENT_TYPE_MARKER_RECORDED","version":"0","taskId":"1048703","markerRecordedEventAttributes":{"details":{"version":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"MQ=="}]},"change-id":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"IjIwMjEwNS1icm9rZXJBZ3JlZW1lbnQi"}]}},"markerName":"Version","workflowTaskCompletedEventId":"4"},"attributes":"markerRecordedEventAttributes"},{"eventId":"6","eventTime":{"seconds":"1623309982","nanos":254135604},"eventType":"EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES","version":"0","taskId":"1048704","upsertWorkflowSearchAttributesEventAttributes":{"workflowTaskCompletedEventId":"4","searchAttributes":{"indexedFields":{"TemporalChangeVersion":{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"WyIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ=="}}}},"attributes":"upsertWorkflowSearchAttributesEventAttributes"},{"eventId":"7","eventTime":{"seconds":"1623309982","nanos":254151585},"eventType":"EVENT_TYPE_MARKER_RECORDED","version":"0","taskId":"1048705","markerRecordedEventAttributes":{"details":{"version":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"MQ=="}]},"change-id":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"InZlcmlmeVByb2ZpbGUi"}]}},"markerName":"Version","workflowTaskCompletedEventId":"4"},"attributes":"markerRecordedEventAttributes"},{"eventId":"8","eventTime":{"seconds":"1623309982","nanos":254195215},"eventType":"EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES","version":"0","taskId":"1048706","upsertWorkflowSearchAttributesEventAttributes":{"workflowTaskCompletedEventId":"4","searchAttributes":{"indexedFields":{"TemporalChangeVersion":{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"WyJ2ZXJpZnlQcm9maWxlLTEiLCIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ=="}}}},"attributes":"upsertWorkflowSearchAttributesEventAttributes"},{"eventId":"9","eventTime":{"seconds":"1623309982","nanos":254212645},"eventType":"EVENT_TYPE_ACTIVITY_TASK_SCHEDULED","version":"0","taskId":"1048707","activityTaskScheduledEventAttributes":{"activityId":"9","activityType":{"name":"GetStatusFlagActivity"},"namespace":"","taskQueue":{"name":"ClientServiceTaskList","kind":"TASK_QUEUE_KIND_NORMAL"},"header":{"fields":{"opencensusPayload":{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"eyJEYXRhIjoiQUFCOUJhR1AwSlZzM0lKdWQ2RDFGdUhPQWFOU2RBbm5zMEpxQWdBPSJ9"}}},"input":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"IjE2MjMzMDk5Nzk2NTQi"},{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"IlBlcnNvbmFsRGV0YWlscyI="}]},"scheduleToCloseTimeout":{"seconds":"0","nanos":0},"scheduleToStartTimeout":{"seconds":"10","nanos":0},"startToCloseTimeout":{"seconds":"60","nanos":0},"heartbeatTimeout":{"seconds":"0","nanos":0},"workflowTaskCompletedEventId":"4","retryPolicy":{"nonRetryableErrorTypes":[],"initialInterval":{"seconds":"1","nanos":0},"backoffCoefficient":2,"maximumInterval":{"seconds":"100","nanos":0},"maximumAttempts":5}},"attributes":"activityTaskScheduledEventAttributes"},{"eventId":"10","eventTime":{"seconds":"1623309982","nanos":259954539},"eventType":"EVENT_TYPE_ACTIVITY_TASK_STARTED","version":"0","taskId":"1048714","activityTaskStartedEventAttributes":{"scheduledEventId":"9","identity":"1967065@archlinux@","requestId":"511299c4-3357-4202-a950-f4b174b783ea","attempt":1},"attributes":"activityTaskStartedEventAttributes"},{"eventId":"11","eventTime":{"seconds":"1623309982","nanos":264279500},"eventType":"EVENT_TYPE_ACTIVITY_TASK_COMPLETED","version":"0","taskId":"1048715","activityTaskCompletedEventAttributes":{"result":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"ZmFsc2U="}]},"scheduledEventId":"9","startedEventId":"10","identity":"1967065@archlinux@"},"attributes":"activityTaskCompletedEventAttributes"},{"eventId":"12","eventTime":{"seconds":"1623309982","nanos":264287770},"eventType":"EVENT_TYPE_WORKFLOW_TASK_SCHEDULED","version":"0","taskId":"1048716","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"archlinux:225d7cec-d6ef-4402-8ddc-e319f634ec57","kind":"TASK_QUEUE_KIND_STICKY"},"startToCloseTimeout":{"seconds":"10","nanos":0},"attempt":1},"attributes":"workflowTaskScheduledEventAttributes"},{"eventId":"13","eventTime":{"seconds":"1623309982","nanos":268420339},"eventType":"EVENT_TYPE_WORKFLOW_TASK_STARTED","version":"0","taskId":"1048720","workflowTaskStartedEventAttributes":{"scheduledEventId":"12","identity":"1967065@archlinux@","requestId":"63026fcf-38ea-46a2-b815-7eb758cafd09"},"attributes":"workflowTaskStartedEventAttributes"},{"eventId":"14","eventTime":{"seconds":"1623309982","nanos":274027491},"eventType":"EVENT_TYPE_WORKFLOW_TASK_COMPLETED","version":"0","taskId":"1048723","workflowTaskCompletedEventAttributes":{"scheduledEventId":"12","startedEventId":"13","identity":"1967065@archlinux@","binaryChecksum":"ad8af47b2d94e32093fb384f93ca4720"},"attributes":"workflowTaskCompletedEventAttributes"},{"eventId":"15","eventTime":{"seconds":"1623309982","nanos":274050832},"eventType":"EVENT_TYPE_ACTIVITY_TASK_SCHEDULED","version":"0","taskId":"1048724","activityTaskScheduledEventAttributes":{"activityId":"ProvidePersonalDetailsActivity","activityType":{"name":"ProvidePersonalDetailsActivity"},"namespace":"","taskQueue":{"name":"ClientServiceTaskList","kind":"TASK_QUEUE_KIND_NORMAL"},"header":{"fields":{"opencensusPayload":{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"eyJEYXRhIjoiQUFCOUJhR1AwSlZzM0lKdWQ2RDFGdUhPQWFOU2RBbm5zMEpxQWdBPSJ9"}}},"input":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"eyJjYXRlZ29yeSI6ImNsaWVudC1wcm9jZXNzLXN0YXRlIiwiVVVJRCI6ImMzMHJ0N2tubXZsMHZ2Y3RqcWVnIiwic291cmNlIjoiMTYyMzMwOTk3OTY1NC9DbGFzc2ljT25ib2FyZGluZ0NGRCIsIm1lc3NhZ2UiOiJ7XCJQSURcIjpcIkNsYXNzaWNPbmJvYXJkaW5nQ0ZEXCIsXCJDbGllbnRJRFwiOlwiMTYyMzMwOTk3OTY1NFwiLFwiU3RlcFJ1bklEXCI6XCJcIixcIlN0ZXBFcnJvclwiOlwiXCIsXCJTdGVwc0xpc3RcIjpbe1wiTmFtZVwiOlwiUGVyc29uYWxEZXRhaWxzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfSx7XCJOYW1lXCI6XCJSZXNpZGVudGlhbEFkZHJlc3NcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVGF4ZXNDb3VudHJ5XCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRheElEXCIsXCJDYW5Ta2lwXCI6dHJ1ZSxcIkNhblNhdmVcIjpmYWxzZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlNvdXJjZU9mRnVuZHNcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiY2ZkQU1MXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRyYWRpbmdBY2NvdW50UHJlZmVyZW5jaWVzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIkJyb2tlckFncmVlbWVudFwiLFwiQ2FuU2tpcFwiOmZhbHNlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVmVyaWZ5SWRlbnRpdHlcIixcIkNhblNraXBcIjp0cnVlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOmZhbHNlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlZlcmlmeVBPUlwiLFwiQ2FuU2tpcFwiOnRydWUsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfV0sXCJQb3NpdGlvblwiOjAsXCJEb25lXCI6ZmFsc2UsXCJJc1Byb2Nlc3NpbmdcIjpmYWxzZSxcIlN0YXJ0ZWRcIjpmYWxzZSxcIlJlZGlyZWN0XCI6bnVsbCxcIlJlc3VsdFwiOm51bGx9In0="}]},"scheduleToCloseTimeout":{"seconds":"0","nanos":0},"scheduleToStartTimeout":{"seconds":"10","nanos":0},"startToCloseTimeout":{"seconds":"30672000","nanos":0},"heartbeatTimeout":{"seconds":"0","nanos":0},"workflowTaskCompletedEventId":"14","retryPolicy":{"nonRetryableErrorTypes":[],"initialInterval":{"seconds":"1","nanos":0},"backoffCoefficient":2,"maximumInterval":{"seconds":"100","nanos":0},"maximumAttempts":0}},"attributes":"activityTaskScheduledEventAttributes"},{"eventId":"16","eventTime":{"seconds":"1623309982","nanos":277984619},"eventType":"EVENT_TYPE_ACTIVITY_TASK_STARTED","version":"0","taskId":"1048739","activityTaskStartedEventAttributes":{"scheduledEventId":"15","identity":"1967065@archlinux@","requestId":"fa1ca986-9c44-47ab-a0b3-557382a5e623","attempt":1},"attributes":"activityTaskStartedEventAttributes"},{"eventId":"17","eventTime":{"seconds":"1623310012","nanos":437858560},"eventType":"EVENT_TYPE_ACTIVITY_TASK_COMPLETED","version":"0","taskId":"1048740","activityTaskCompletedEventAttributes":{"result":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"eyJDb21tYW5kIjoiU0FWRSIsIkRhdGEiOnsiRmlyc3ROYW1lIjoiYXNkZmFzZGYiLCJMYXN0TmFtZSI6ImFzZGYiLCJEYXRlT2ZCaXJ0aCI6IjExMTEtMTEtMTEiLCJHZW5kZXIiOiJtYWxlIiwiTmF0aW9uYWxpdHlDb2RlIjpudWxsLCJOb25VUyI6ZmFsc2UsIk5vblByb1N1YnNjcmliZXIiOm51bGx9fQ=="}]},"scheduledEventId":"15","startedEventId":"16","identity":"1965479@archlinux@"},"attributes":"activityTaskCompletedEventAttributes"},{"eventId":"18","eventTime":{"seconds":"1623310012","nanos":437872420},"eventType":"EVENT_TYPE_WORKFLOW_TASK_SCHEDULED","version":"0","taskId":"1048741","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"archlinux:225d7cec-d6ef-4402-8ddc-e319f634ec57","kind":"TASK_QUEUE_KIND_STICKY"},"startToCloseTimeout":{"seconds":"10","nanos":0},"attempt":1},"attributes":"workflowTaskScheduledEventAttributes"},{"eventId":"19","eventTime":{"seconds":"1623310017","nanos":442949325},"eventType":"EVENT_TYPE_WORKFLOW_TASK_TIMED_OUT","version":"0","taskId":"1048745","workflowTaskTimedOutEventAttributes":{"scheduledEventId":"18","startedEventId":"0","timeoutType":"TIMEOUT_TYPE_SCHEDULE_TO_START"},"attributes":"workflowTaskTimedOutEventAttributes"},{"eventId":"20","eventTime":{"seconds":"1623310017","nanos":442957265},"eventType":"EVENT_TYPE_WORKFLOW_TASK_SCHEDULED","version":"0","taskId":"1048746","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"ClientServiceTaskList","kind":"TASK_QUEUE_KIND_NORMAL"},"startToCloseTimeout":{"seconds":"10","nanos":0},"attempt":1},"attributes":"workflowTaskScheduledEventAttributes"},{"eventId":"21","eventTime":{"seconds":"1623310017","nanos":446818171},"eventType":"EVENT_TYPE_WORKFLOW_TASK_STARTED","version":"0","taskId":"1048749","workflowTaskStartedEventAttributes":{"scheduledEventId":"20","identity":"1967508@archlinux@","requestId":"acdc2a48-b358-41fd-8bca-e95a4589866b"},"attributes":"workflowTaskStartedEventAttributes"},{"eventId":"22","eventTime":{"seconds":"1623310017","nanos":452377643},"eventType":"EVENT_TYPE_WORKFLOW_TASK_FAILED","version":"0","taskId":"1048752","workflowTaskFailedEventAttributes":{"scheduledEventId":"20","startedEventId":"21","cause":"WORKFLOW_TASK_FAILED_CAUSE_WORKFLOW_WORKER_UNHANDLED_FAILURE","failure":{"message":"lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9","source":"GoSDK","stackTrace":"process event for ClientServiceTaskList [panic]:\ngo.temporal.io/sdk/internal.panicIllegalState(0xc000020c60, 0x53)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_decision_state_machine.go:393 +0x66\ngo.temporal.io/sdk/internal.(*commandsHelper).handleActivityTaskScheduled(0xc000656cc0, 0x28a8108, 0x1, 0x9)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_decision_state_machine.go:878 +0x265\ngo.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc000a95560, 0xc000a8f4c0, 0x1, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_event_handlers.go:800 +0x6fa\ngo.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc0001c7a70, 0xc000b71e60, 0x0, 0x0, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_handlers.go:876 +0xc94\ngo.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc0001793f0, 0xc000b71e60, 0xc00017c1e0, 0x0, 0x0, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_handlers.go:727 +0x6af\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc0004ed6c0, 0xc000b71e60, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_pollers.go:288 +0x35a\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc0004ed6c0, 0x1b50d00, 0xc000b71e60, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_pollers.go:259 +0x107\ngo.temporal.io/sdk/internal.(*baseWorker).processTask(0xc0006601e0, 0x1b50d00, 0xc000b71e60)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_worker_base.go:343 +0x19a\ncreated by go.temporal.io/sdk/internal.(*baseWorker).runTaskDispatcher\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_worker_base.go:270 +0x365","applicationFailureInfo":{"type":"PanicError","nonRetryable":true},"failureInfo":"applicationFailureInfo"},"identity":"1967508@archlinux@","baseRunId":"","newRunId":"","forkEventVersion":"0","binaryChecksum":"ad8af47b2d94e32093fb384f93ca4720"},"attributes":"workflowTaskFailedEventAttributes"}]

Hey guys, can you provide any suggestion or workaround for this case?

@korvinko temporal auto-setup sets up a server for you, what Samar was talking about above was an SDK version. Please run go get github.com/temporalio/sdk-go@v1.7.0 on your client code to update to the latest version of go SDK.

Sorry, I’ve updated everything, but still face with workflow panics:

2021/06/15 02:02:42 ERROR Workflow panic Namespace clients TaskQueue ClientServiceTaskList WorkerID 421400@archlinux@ WorkflowType ClassicOnboardingCFD WorkflowID 1623711709629/ClassicOnboardingCFD RunID 7265bbea-3db5-427a-b57c-f926291e0c1b Attempt 1 Error lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9 StackTrace process event for ClientServiceTaskList [panic]:

go.mod

go.temporal.io/api v1.4.1-0.20210429213054-a9a257b5cf16
go.temporal.io/sdk v1.7.0
go.temporal.io/server v1.10.2

docker-compose:

temporal-server:
image: temporalio/auto-setup:1.10.2
temporal-web:
image: temporalio/web:1.9.2

History:

[{“eventId”:“1”,“eventTime”:{“seconds”:“1623711719”,“nanos”:543989124},“eventType”:“EVENT_TYPE_WORKFLOW_EXECUTION_STARTED”,“version”:“0”,“taskId”:“1048766”,“workflowExecutionStartedEventAttributes”:{“workflowType”:{“name”:“ClassicOnboardingCFD”},“parentWorkflowNamespace”:“clients”,“parentWorkflowExecution”:{“workflowId”:“1623711709629/SignUpWeb”,“runId”:“71ebf963-35c7-4133-b07d-a8297aa004a1”},“parentInitiatedEventId”:“44”,“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjM3MTE3MDk2Mjki”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlJFVEFJTCI=”}]},“workflowExecutionTimeout”:{“seconds”:“0”,“nanos”:0},“workflowRunTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskTimeout”:{“seconds”:“10”,“nanos”:0},“continuedExecutionRunId”:"",“initiator”:“CONTINUE_AS_NEW_INITIATOR_UNSPECIFIED”,“originalExecutionRunId”:“7265bbea-3db5-427a-b57c-f926291e0c1b”,“identity”:"",“firstExecutionRunId”:“7265bbea-3db5-427a-b57c-f926291e0c1b”,“attempt”:1,“cronSchedule”:"",“firstWorkflowTaskBackoff”:{“seconds”:“0”,“nanos”:0},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFCV01GMlREelBMQmtTWU9PM0RHUnFKQVpQcm9tTWN3NjhTQWdBPSJ9”}}}},“attributes”:“workflowExecutionStartedEventAttributes”},{“eventId”:“2”,“eventTime”:{“seconds”:“1623711719”,“nanos”:548321764},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048772”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“3”,“eventTime”:{“seconds”:“1623711719”,“nanos”:550739633},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048777”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“2”,“identity”:“420853@archlinux@”,“requestId”:“66e947e3-3559-4a58-ad01-517bea6ceeaf”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“4”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555082993},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048783”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“2”,“startedEventId”:“3”,“identity”:“420853@archlinux@”,“binaryChecksum”:“20548eb1e1ddc71114739b64fcff6fa7”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“5”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555106093},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048784”,“markerRecordedEventAttributes”:{“details”:{“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]},“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjIwMjEwNS1icm9rZXJBZ3JlZW1lbnQi”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“6”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555133653},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048785”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“type”:“S2V5d29yZA==”,“encoding”:“anNvbi9wbGFpbg==”},“data”:“WyIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“7”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555140203},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048786”,“markerRecordedEventAttributes”:{“details”:{“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“InZlcmlmeVByb2ZpbGUi”}]},“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“8”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555157753},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048787”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”,“type”:“S2V5d29yZA==”},“data”:“WyJ2ZXJpZnlQcm9maWxlLTEiLCIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“9”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555172433},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048788”,“activityTaskScheduledEventAttributes”:{“activityId”:“9”,“activityType”:{“name”:“GetStatusFlagActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFCV01GMlREelBMQmtTWU9PM0RHUnFKQVpQcm9tTWN3NjhTQWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjM3MTE3MDk2Mjki”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlBlcnNvbmFsRGV0YWlscyI=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“60”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“4”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:5}},“attributes”:“activityTaskScheduledEventAttributes”},{“eventId”:“10”,“eventTime”:{“seconds”:“1623711719”,“nanos”:558921226},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_STARTED”,“version”:“0”,“taskId”:“1048801”,“activityTaskStartedEventAttributes”:{“scheduledEventId”:“9”,“identity”:“420853@archlinux@”,“requestId”:“b3661bff-fe3d-4ce5-afd6-9e02cabf5a14”,“attempt”:1},“attributes”:“activityTaskStartedEventAttributes”},{“eventId”:“11”,“eventTime”:{“seconds”:“1623711719”,“nanos”:563030727},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048802”,“activityTaskCompletedEventAttributes”:{“result”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“ZmFsc2U=”}]},“scheduledEventId”:“9”,“startedEventId”:“10”,“identity”:“420853@archlinux@”},“attributes”:“activityTaskCompletedEventAttributes”},{“eventId”:“12”,“eventTime”:{“seconds”:“1623711719”,“nanos”:563035517},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048803”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“archlinux:6900d039-2406-41e1-87f6-0284e4d97d32”,“kind”:“TASK_QUEUE_KIND_STICKY”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“13”,“eventTime”:{“seconds”:“1623711719”,“nanos”:567720495},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048818”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“12”,“identity”:“420853@archlinux@”,“requestId”:“4e51d674-aa60-4b2c-b595-5597fadafdd5”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“14”,“eventTime”:{“seconds”:“1623711719”,“nanos”:572633663},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048822”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“12”,“startedEventId”:“13”,“identity”:“420853@archlinux@”,“binaryChecksum”:“20548eb1e1ddc71114739b64fcff6fa7”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“15”,“eventTime”:{“seconds”:“1623711719”,“nanos”:572657003},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048823”,“activityTaskScheduledEventAttributes”:{“activityId”:“ProvidePersonalDetailsActivity”,“activityType”:{“name”:“ProvidePersonalDetailsActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFCV01GMlREelBMQmtTWU9PM0RHUnFKQVpQcm9tTWN3NjhTQWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJjYXRlZ29yeSI6ImNsaWVudC1wcm9jZXNzLXN0YXRlIiwiVVVJRCI6ImMzM3R2cHNubXZsNmZrOWNxNjYwIiwic291cmNlIjoiMTYyMzcxMTcwOTYyOS9DbGFzc2ljT25ib2FyZGluZ0NGRCIsIm1lc3NhZ2UiOiJ7XCJQSURcIjpcIkNsYXNzaWNPbmJvYXJkaW5nQ0ZEXCIsXCJDbGllbnRJRFwiOlwiMTYyMzcxMTcwOTYyOVwiLFwiU3RlcFJ1bklEXCI6XCJcIixcIlN0ZXBFcnJvclwiOlwiXCIsXCJTdGVwc0xpc3RcIjpbe1wiTmFtZVwiOlwiUGVyc29uYWxEZXRhaWxzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfSx7XCJOYW1lXCI6XCJSZXNpZGVudGlhbEFkZHJlc3NcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVGF4ZXNDb3VudHJ5XCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRheElEXCIsXCJDYW5Ta2lwXCI6dHJ1ZSxcIkNhblNhdmVcIjpmYWxzZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlNvdXJjZU9mRnVuZHNcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiY2ZkQU1MXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRyYWRpbmdBY2NvdW50UHJlZmVyZW5jaWVzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIkJyb2tlckFncmVlbWVudFwiLFwiQ2FuU2tpcFwiOmZhbHNlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVmVyaWZ5SWRlbnRpdHlcIixcIkNhblNraXBcIjp0cnVlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOmZhbHNlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlZlcmlmeVBPUlwiLFwiQ2FuU2tpcFwiOnRydWUsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfV0sXCJQb3NpdGlvblwiOjAsXCJEb25lXCI6ZmFsc2UsXCJJc1Byb2Nlc3NpbmdcIjpmYWxzZSxcIlN0YXJ0ZWRcIjpmYWxzZSxcIlJlZGlyZWN0XCI6bnVsbCxcIlJlc3VsdFwiOm51bGx9In0=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“30672000”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“14”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:0}},“attributes”:“activityTaskScheduledEventAttributes”}]