Panic occurs for multiple changeID

korvinko · June 9, 2021, 3:15pm

Panic occurs if we have multiple changeID inside the same workflow. An abstract example is below:

	versionStepA := workflow.GetVersion(ctx, "stepA", workflow.DefaultVersion, 1)
	var state someState
	if versionWorkflowA == workflow.DefaultVersion {
		state = getState1()
	} else {
		state = getState3()
	}

	versionStepB := workflow.GetVersion(ctx, "stepB", workflow.DefaultVersion, 1)
	var (
		secondState  someState 
		thirdState someState 
	)
	if versionStepB  == workflow.DefaultVersion {
		secondState  = getState2()
		thirdState = getState2()
	} else {
		secondState  = getState4()
		thirdState = getState4()
	}

How to reproduce:

Start workflow
Restart of Go process
Try to finish workflow

After that, you get following error:

2021/06/09 14:38:30 ERROR Workflow panic Namespace clients TaskQueue ClientServiceTaskList WorkerID 867130@archlinux@ WorkflowType ClassicOnboardingCFD WorkflowID OUR_WORKFLOW_ID RunID a0a07560-fca3-4744-bbb0-68f8b7146de8 Attempt 1 Error lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9 StackTrace process event for ClientServiceTaskList [panic]:

Everything works well if delete the second changeID and call the latest version without conditional statement or completely delete all versions and directly call the latest version of steates.

samar · June 9, 2021, 3:21pm

Can you also share workflow execution history?

korvinko · June 9, 2021, 3:42pm

[{“eventId”:“1”,“eventTime”:{“seconds”:“1623253200”,“nanos”:68319442},“eventType”:“EVENT_TYPE_WORKFLOW_EXECUTION_STARTED”,“version”:“0”,“taskId”:“1048576”,“workflowExecutionStartedEventAttributes”:{“workflowType”:{“name”:“ClassicOnboardingCFD”},“parentWorkflowNamespace”:“clients”,“parentWorkflowExecution”:{“workflowId”:“1623253183015/SignUpWeb”,“runId”:“586f4e63-3455-43f8-b33b-e8516fffa758”},“parentInitiatedEventId”:“44”,“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjMyNTMxODMwMTUi”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlJFVEFJTCI=”}]},“workflowExecutionTimeout”:{“seconds”:“0”,“nanos”:0},“workflowRunTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskTimeout”:{“seconds”:“10”,“nanos”:0},“continuedExecutionRunId”:"",“initiator”:“CONTINUE_AS_NEW_INITIATOR_UNSPECIFIED”,“originalExecutionRunId”:“e49e4fb5-fc98-4a8f-8759-b9b8a335d7b9”,“identity”:"",“firstExecutionRunId”:“e49e4fb5-fc98-4a8f-8759-b9b8a335d7b9”,“attempt”:1,“cronSchedule”:"",“firstWorkflowTaskBackoff”:{“seconds”:“0”,“nanos”:0},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFBQXkycGRLbkFLUyt2dE9kMWh5bXp3QVRadXF0ejBtQ3A1QWdBPSJ9”}}}},“attributes”:“workflowExecutionStartedEventAttributes”},{“eventId”:“2”,“eventTime”:{“seconds”:“1623253200”,“nanos”:74663475},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048579”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“3”,“eventTime”:{“seconds”:“1623253200”,“nanos”:78026976},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048582”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“2”,“identity”:“980282@archlinux@”,“requestId”:“01c425df-1e2a-467a-b752-c6fee1ae0e08”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“4”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82759474},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048585”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“2”,“startedEventId”:“3”,“identity”:“980282@archlinux@”,“binaryChecksum”:“f9add7201b68e897cd0239007217d4a1”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“5”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82784374},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048586”,“markerRecordedEventAttributes”:{“details”:{“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjIwMjEwNS1icm9rZXJBZ3JlZW1lbnQi”}]},“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“6”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82809294},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048587”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“WyIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“7”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82813214},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048588”,“markerRecordedEventAttributes”:{“details”:{“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]},“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“InZlcmlmeVByb2ZpbGUi”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“8”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82823644},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048589”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“WyJ2ZXJpZnlQcm9maWxlLTEiLCIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“9”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82831274},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048590”,“activityTaskScheduledEventAttributes”:{“activityId”:“9”,“activityType”:{“name”:“GetStatusFlagActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFBQXkycGRLbkFLUyt2dE9kMWh5bXp3QVRadXF0ejBtQ3A1QWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjMyNTMxODMwMTUi”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlBlcnNvbmFsRGV0YWlscyI=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“60”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“4”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:5}},“attributes”:“activityTaskScheduledEventAttributes”},{“eventId”:“10”,“eventTime”:{“seconds”:“1623253200”,“nanos”:86185975},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_STARTED”,“version”:“0”,“taskId”:“1048597”,“activityTaskStartedEventAttributes”:{“scheduledEventId”:“9”,“identity”:“980282@archlinux@”,“requestId”:“53104e20-0c03-4966-8e4b-0a9b3895a9f7”,“attempt”:1},“attributes”:“activityTaskStartedEventAttributes”},{“eventId”:“11”,“eventTime”:{“seconds”:“1623253200”,“nanos”:89233977},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048598”,“activityTaskCompletedEventAttributes”:{“result”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“ZmFsc2U=”}]},“scheduledEventId”:“9”,“startedEventId”:“10”,“identity”:“980282@archlinux@”},“attributes”:“activityTaskCompletedEventAttributes”},{“eventId”:“12”,“eventTime”:{“seconds”:“1623253200”,“nanos”:89240407},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048601”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“archlinux:4d64965f-b931-457a-806b-619b66a3cfca”,“kind”:“TASK_QUEUE_KIND_STICKY”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“13”,“eventTime”:{“seconds”:“1623253200”,“nanos”:93012957},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048605”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“12”,“identity”:“980282@archlinux@”,“requestId”:“04c41b3b-b4d8-4a43-af02-8700aec63bdf”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“14”,“eventTime”:{“seconds”:“1623253200”,“nanos”:97254986},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048608”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“12”,“startedEventId”:“13”,“identity”:“980282@archlinux@”,“binaryChecksum”:“f9add7201b68e897cd0239007217d4a1”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“15”,“eventTime”:{“seconds”:“1623253200”,“nanos”:97275436},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048609”,“activityTaskScheduledEventAttributes”:{“activityId”:“ProvidePersonalDetailsActivity”,“activityType”:{“name”:“ProvidePersonalDetailsActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFBQXkycGRLbkFLUyt2dE9kMWh5bXp3QVRadXF0ejBtQ3A1QWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJjYXRlZ29yeSI6ImNsaWVudC1wcm9jZXNzLXN0YXRlIiwiVVVJRCI6ImMzMGUxazRubXZsZmk3azcwanVnIiwic291cmNlIjoiMTYyMzI1MzE4MzAxNS9DbGFzc2ljT25ib2FyZGluZ0NGRCIsIm1lc3NhZ2UiOiJ7XCJQSURcIjpcIkNsYXNzaWNPbmJvYXJkaW5nQ0ZEXCIsXCJDbGllbnRJRFwiOlwiMTYyMzI1MzE4MzAxNVwiLFwiU3RlcFJ1bklEXCI6XCJcIixcIlN0ZXBFcnJvclwiOlwiXCIsXCJTdGVwc0xpc3RcIjpbe1wiTmFtZVwiOlwiUGVyc29uYWxEZXRhaWxzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfSx7XCJOYW1lXCI6XCJSZXNpZGVudGlhbEFkZHJlc3NcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVGF4ZXNDb3VudHJ5XCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRheElEXCIsXCJDYW5Ta2lwXCI6dHJ1ZSxcIkNhblNhdmVcIjpmYWxzZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlNvdXJjZU9mRnVuZHNcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiY2ZkQU1MXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRyYWRpbmdBY2NvdW50UHJlZmVyZW5jaWVzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIkJyb2tlckFncmVlbWVudFwiLFwiQ2FuU2tpcFwiOmZhbHNlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVmVyaWZ5SWRlbnRpdHlcIixcIkNhblNraXBcIjp0cnVlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOmZhbHNlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlZlcmlmeVBPUlwiLFwiQ2FuU2tpcFwiOnRydWUsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfV0sXCJQb3NpdGlvblwiOjAsXCJEb25lXCI6ZmFsc2UsXCJJc1Byb2Nlc3NpbmdcIjpmYWxzZSxcIlN0YXJ0ZWRcIjpmYWxzZSxcIlJlZGlyZWN0XCI6bnVsbCxcIlJlc3VsdFwiOm51bGx9In0=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“30672000”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“14”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:0}},“attributes”:“activityTaskScheduledEventAttributes”},{“eventId”:“16”,“eventTime”:{“seconds”:“1623253200”,“nanos”:100716157},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_STARTED”,“version”:“0”,“taskId”:“1048614”,“activityTaskStartedEventAttributes”:{“scheduledEventId”:“15”,“identity”:“980282@archlinux@”,“requestId”:“ebeba944-167d-48d3-b4a0-ddecaeff23bd”,“attempt”:1},“attributes”:“activityTaskStartedEventAttributes”},{“eventId”:“17”,“eventTime”:{“seconds”:“1623253226”,“nanos”:637457577},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048615”,“activityTaskCompletedEventAttributes”:{“result”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJDb21tYW5kIjoiU0FWRSIsIkRhdGEiOnsiRmlyc3ROYW1lIjoiYXNkZmEiLCJMYXN0TmFtZSI6InNkZmFzZCIsIkRhdGVPZkJpcnRoIjoiMTExMS0xMS0xMSIsIkdlbmRlciI6Im1hbGUiLCJOYXRpb25hbGl0eUNvZGUiOm51bGwsIk5vblVTIjpmYWxzZSwiTm9uUHJvU3Vic2NyaWJlciI6bnVsbH19”}]},“scheduledEventId”:“15”,“startedEventId”:“16”,“identity”:“845434@archlinux@”},“attributes”:“activityTaskCompletedEventAttributes”},{“eventId”:“18”,“eventTime”:{“seconds”:“1623253226”,“nanos”:637471197},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048618”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“archlinux:4d64965f-b931-457a-806b-619b66a3cfca”,“kind”:“TASK_QUEUE_KIND_STICKY”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“19”,“eventTime”:{“seconds”:“1623253231”,“nanos”:638920473},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_TIMED_OUT”,“version”:“0”,“taskId”:“1048622”,“workflowTaskTimedOutEventAttributes”:{“scheduledEventId”:“18”,“startedEventId”:“0”,“timeoutType”:“TIMEOUT_TYPE_SCHEDULE_TO_START”},“attributes”:“workflowTaskTimedOutEventAttributes”},{“eventId”:“20”,“eventTime”:{“seconds”:“1623253231”,“nanos”:638933873},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048623”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“21”,“eventTime”:{“seconds”:“1623253231”,“nanos”:643120822},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048626”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“20”,“identity”:“980857@archlinux@”,“requestId”:“9f9268e6-f106-4eda-91d1-a1dc4231db2f”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“22”,“eventTime”:{“seconds”:“1623253231”,“nanos”:648673099},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_FAILED”,“version”:“0”,“taskId”:“1048629”,“workflowTaskFailedEventAttributes”:{“scheduledEventId”:“20”,“startedEventId”:“21”,“cause”:“WORKFLOW_TASK_FAILED_CAUSE_WORKFLOW_WORKER_UNHANDLED_FAILURE”,“failure”:{“message”:“lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9”,“source”:“GoSDK”,“stackTrace”:“process event for ClientServiceTaskList [panic]:\ngo.temporal.io/sdk/internal.panicIllegalState(0xc00004e9c0, 0x53)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_decision_state_machine.go:393 +0x66\ngo.temporal.io/sdk/internal.(*commandsHelper).handleActivityTaskScheduled(0xc00069a480, 0x28ac108, 0x1, 0x9)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_decision_state_machine.go:878 +0x265\ngo.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc000c104e0, 0xc0000aac00, 0x1, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_event_handlers.go:800 +0x6fa\ngo.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc00011a240, 0xc00092f110, 0x0, 0x0, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_handlers.go:876 +0xc94\ngo.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc0009b24d0, 0xc00092f110, 0xc000507e90, 0x0, 0x0, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_handlers.go:727 +0x6af\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc00064a4e0, 0xc00092f110, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_pollers.go:288 +0x35a\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc00064a4e0, 0x1b52d40, 0xc00092f110, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_pollers.go:259 +0x107\ngo.temporal.io/sdk/internal.(*baseWorker).processTask(0xc000536ff0, 0x1b52d40, 0xc00092f110)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_worker_base.go:343 +0x19a\ncreated by go.temporal.io/sdk/internal.(*baseWorker).runTaskDispatcher\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_worker_base.go:270 +0x365”,“applicationFailureInfo”:{“type”:“PanicError”,“nonRetryable”:true},“failureInfo”:“applicationFailureInfo”},“identity”:“980857@archlinux@”,“baseRunId”:"",“newRunId”:"",“forkEventVersion”:“0”,“binaryChecksum”:“f9add7201b68e897cd0239007217d4a1”},“attributes”:“workflowTaskFailedEventAttributes”}]

samar · June 9, 2021, 3:59pm

Looks like you are using Go SDK v1.5.0. Can you try to reproduce it with v1.6.0? We have some fixes for similar problems in that release.

korvinko · June 10, 2021, 7:31am

There was temporalio/auto-setup:1.6.5 .
Here logs for temporalio/auto-setup:1.9.2

korvinko · June 14, 2021, 8:33am

Hey guys, can you provide any suggestion or workaround for this case?

Vitaly · June 14, 2021, 7:45pm

@korvinko temporal auto-setup sets up a server for you, what Samar was talking about above was an SDK version. Please run go get github.com/temporalio/sdk-go@v1.7.0 on your client code to update to the latest version of go SDK.

korvinko · June 14, 2021, 11:10pm

Sorry, I’ve updated everything, but still face with workflow panics:

2021/06/15 02:02:42 ERROR Workflow panic Namespace clients TaskQueue ClientServiceTaskList WorkerID 421400@archlinux@ WorkflowType ClassicOnboardingCFD WorkflowID 1623711709629/ClassicOnboardingCFD RunID 7265bbea-3db5-427a-b57c-f926291e0c1b Attempt 1 Error lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9 StackTrace process event for ClientServiceTaskList [panic]:

go.mod

go.temporal.io/api v1.4.1-0.20210429213054-a9a257b5cf16
go.temporal.io/sdk v1.7.0
go.temporal.io/server v1.10.2

docker-compose:

temporal-server:
image: temporalio/auto-setup:1.10.2
temporal-web:
image: temporalio/web:1.9.2

History:

[{“eventId”:“1”,“eventTime”:{“seconds”:“1623711719”,“nanos”:543989124},“eventType”:“EVENT_TYPE_WORKFLOW_EXECUTION_STARTED”,“version”:“0”,“taskId”:“1048766”,“workflowExecutionStartedEventAttributes”:{“workflowType”:{“name”:“ClassicOnboardingCFD”},“parentWorkflowNamespace”:“clients”,“parentWorkflowExecution”:{“workflowId”:“1623711709629/SignUpWeb”,“runId”:“71ebf963-35c7-4133-b07d-a8297aa004a1”},“parentInitiatedEventId”:“44”,“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjM3MTE3MDk2Mjki”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlJFVEFJTCI=”}]},“workflowExecutionTimeout”:{“seconds”:“0”,“nanos”:0},“workflowRunTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskTimeout”:{“seconds”:“10”,“nanos”:0},“continuedExecutionRunId”:"",“initiator”:“CONTINUE_AS_NEW_INITIATOR_UNSPECIFIED”,“originalExecutionRunId”:“7265bbea-3db5-427a-b57c-f926291e0c1b”,“identity”:"",“firstExecutionRunId”:“7265bbea-3db5-427a-b57c-f926291e0c1b”,“attempt”:1,“cronSchedule”:"",“firstWorkflowTaskBackoff”:{“seconds”:“0”,“nanos”:0},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFCV01GMlREelBMQmtTWU9PM0RHUnFKQVpQcm9tTWN3NjhTQWdBPSJ9”}}}},“attributes”:“workflowExecutionStartedEventAttributes”},{“eventId”:“2”,“eventTime”:{“seconds”:“1623711719”,“nanos”:548321764},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048772”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“3”,“eventTime”:{“seconds”:“1623711719”,“nanos”:550739633},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048777”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“2”,“identity”:“420853@archlinux@”,“requestId”:“66e947e3-3559-4a58-ad01-517bea6ceeaf”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“4”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555082993},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048783”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“2”,“startedEventId”:“3”,“identity”:“420853@archlinux@”,“binaryChecksum”:“20548eb1e1ddc71114739b64fcff6fa7”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“5”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555106093},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048784”,“markerRecordedEventAttributes”:{“details”:{“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]},“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjIwMjEwNS1icm9rZXJBZ3JlZW1lbnQi”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“6”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555133653},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048785”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“type”:“S2V5d29yZA==”,“encoding”:“anNvbi9wbGFpbg==”},“data”:“WyIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“7”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555140203},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048786”,“markerRecordedEventAttributes”:{“details”:{“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“InZlcmlmeVByb2ZpbGUi”}]},“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“8”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555157753},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048787”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”,“type”:“S2V5d29yZA==”},“data”:“WyJ2ZXJpZnlQcm9maWxlLTEiLCIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“9”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555172433},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048788”,“activityTaskScheduledEventAttributes”:{“activityId”:“9”,“activityType”:{“name”:“GetStatusFlagActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFCV01GMlREelBMQmtTWU9PM0RHUnFKQVpQcm9tTWN3NjhTQWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjM3MTE3MDk2Mjki”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlBlcnNvbmFsRGV0YWlscyI=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“60”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“4”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:5}},“attributes”:“activityTaskScheduledEventAttributes”},{“eventId”:“10”,“eventTime”:{“seconds”:“1623711719”,“nanos”:558921226},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_STARTED”,“version”:“0”,“taskId”:“1048801”,“activityTaskStartedEventAttributes”:{“scheduledEventId”:“9”,“identity”:“420853@archlinux@”,“requestId”:“b3661bff-fe3d-4ce5-afd6-9e02cabf5a14”,“attempt”:1},“attributes”:“activityTaskStartedEventAttributes”},{“eventId”:“11”,“eventTime”:{“seconds”:“1623711719”,“nanos”:563030727},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048802”,“activityTaskCompletedEventAttributes”:{“result”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“ZmFsc2U=”}]},“scheduledEventId”:“9”,“startedEventId”:“10”,“identity”:“420853@archlinux@”},“attributes”:“activityTaskCompletedEventAttributes”},{“eventId”:“12”,“eventTime”:{“seconds”:“1623711719”,“nanos”:563035517},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048803”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“archlinux:6900d039-2406-41e1-87f6-0284e4d97d32”,“kind”:“TASK_QUEUE_KIND_STICKY”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“13”,“eventTime”:{“seconds”:“1623711719”,“nanos”:567720495},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048818”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“12”,“identity”:“420853@archlinux@”,“requestId”:“4e51d674-aa60-4b2c-b595-5597fadafdd5”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“14”,“eventTime”:{“seconds”:“1623711719”,“nanos”:572633663},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048822”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“12”,“startedEventId”:“13”,“identity”:“420853@archlinux@”,“binaryChecksum”:“20548eb1e1ddc71114739b64fcff6fa7”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“15”,“eventTime”:{“seconds”:“1623711719”,“nanos”:572657003},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048823”,“activityTaskScheduledEventAttributes”:{“activityId”:“ProvidePersonalDetailsActivity”,“activityType”:{“name”:“ProvidePersonalDetailsActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFCV01GMlREelBMQmtTWU9PM0RHUnFKQVpQcm9tTWN3NjhTQWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJjYXRlZ29yeSI6ImNsaWVudC1wcm9jZXNzLXN0YXRlIiwiVVVJRCI6ImMzM3R2cHNubXZsNmZrOWNxNjYwIiwic291cmNlIjoiMTYyMzcxMTcwOTYyOS9DbGFzc2ljT25ib2FyZGluZ0NGRCIsIm1lc3NhZ2UiOiJ7XCJQSURcIjpcIkNsYXNzaWNPbmJvYXJkaW5nQ0ZEXCIsXCJDbGllbnRJRFwiOlwiMTYyMzcxMTcwOTYyOVwiLFwiU3RlcFJ1bklEXCI6XCJcIixcIlN0ZXBFcnJvclwiOlwiXCIsXCJTdGVwc0xpc3RcIjpbe1wiTmFtZVwiOlwiUGVyc29uYWxEZXRhaWxzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfSx7XCJOYW1lXCI6XCJSZXNpZGVudGlhbEFkZHJlc3NcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVGF4ZXNDb3VudHJ5XCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRheElEXCIsXCJDYW5Ta2lwXCI6dHJ1ZSxcIkNhblNhdmVcIjpmYWxzZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlNvdXJjZU9mRnVuZHNcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiY2ZkQU1MXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRyYWRpbmdBY2NvdW50UHJlZmVyZW5jaWVzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIkJyb2tlckFncmVlbWVudFwiLFwiQ2FuU2tpcFwiOmZhbHNlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVmVyaWZ5SWRlbnRpdHlcIixcIkNhblNraXBcIjp0cnVlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOmZhbHNlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlZlcmlmeVBPUlwiLFwiQ2FuU2tpcFwiOnRydWUsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfV0sXCJQb3NpdGlvblwiOjAsXCJEb25lXCI6ZmFsc2UsXCJJc1Byb2Nlc3NpbmdcIjpmYWxzZSxcIlN0YXJ0ZWRcIjpmYWxzZSxcIlJlZGlyZWN0XCI6bnVsbCxcIlJlc3VsdFwiOm51bGx9In0=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“30672000”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“14”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:0}},“attributes”:“activityTaskScheduledEventAttributes”}]