Panic occurs for multiple changeID

korvinko · June 9, 2021, 3:04pm

Panic occurs if we have multiple changeID inside the same workflow. An abstract example is below:

	versionStepA := workflow.GetVersion(ctx, "stepA", workflow.DefaultVersion, 1)
	var state someState
	if versionWorkflowA == workflow.DefaultVersion {
		state = getState1()
	} else {
		state = getState3()
	}

	versionStepB := workflow.GetVersion(ctx, "stepB", workflow.DefaultVersion, 1)
	var (
		secondState  someState 
		thirdState someState 
	)
	if versionStepB  == workflow.DefaultVersion {
		secondState  = getState2()
		thirdState = getState2()
	} else {
		secondState  = getState4()
		thirdState = getState4()
	}

How to reproduce:

Start workflow
Restart of Go process
Try to finish workflow

After that, you get following error:

2021/06/09 14:38:30 ERROR Workflow panic Namespace clients TaskQueue ClientServiceTaskList WorkerID 867130@archlinux@ WorkflowType ClassicOnboardingCFD WorkflowID OUR_WORKFLOW_ID RunID a0a07560-fca3-4744-bbb0-68f8b7146de8 Attempt 1 Error lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9 StackTrace process event for ClientServiceTaskList [panic]:

Everything works well if delete the second changeID and call the latest version without conditional statement or completely delete all versions and directly call the latest version of steates.

samar · June 9, 2021, 3:21pm

Can you also share workflow execution history?

korvinko · June 9, 2021, 3:42pm

[{“eventId”:“1”,“eventTime”:{“seconds”:“1623253200”,“nanos”:68319442},“eventType”:“EVENT_TYPE_WORKFLOW_EXECUTION_STARTED”,“version”:“0”,“taskId”:“1048576”,“workflowExecutionStartedEventAttributes”:{“workflowType”:{“name”:“ClassicOnboardingCFD”},“parentWorkflowNamespace”:“clients”,“parentWorkflowExecution”:{“workflowId”:“1623253183015/SignUpWeb”,“runId”:“586f4e63-3455-43f8-b33b-e8516fffa758”},“parentInitiatedEventId”:“44”,“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjMyNTMxODMwMTUi”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlJFVEFJTCI=”}]},“workflowExecutionTimeout”:{“seconds”:“0”,“nanos”:0},“workflowRunTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskTimeout”:{“seconds”:“10”,“nanos”:0},“continuedExecutionRunId”:"",“initiator”:“CONTINUE_AS_NEW_INITIATOR_UNSPECIFIED”,“originalExecutionRunId”:“e49e4fb5-fc98-4a8f-8759-b9b8a335d7b9”,“identity”:"",“firstExecutionRunId”:“e49e4fb5-fc98-4a8f-8759-b9b8a335d7b9”,“attempt”:1,“cronSchedule”:"",“firstWorkflowTaskBackoff”:{“seconds”:“0”,“nanos”:0},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFBQXkycGRLbkFLUyt2dE9kMWh5bXp3QVRadXF0ejBtQ3A1QWdBPSJ9”}}}},“attributes”:“workflowExecutionStartedEventAttributes”},{“eventId”:“2”,“eventTime”:{“seconds”:“1623253200”,“nanos”:74663475},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048579”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“3”,“eventTime”:{“seconds”:“1623253200”,“nanos”:78026976},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048582”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“2”,“identity”:“980282@archlinux@”,“requestId”:“01c425df-1e2a-467a-b752-c6fee1ae0e08”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“4”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82759474},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048585”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“2”,“startedEventId”:“3”,“identity”:“980282@archlinux@”,“binaryChecksum”:“f9add7201b68e897cd0239007217d4a1”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“5”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82784374},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048586”,“markerRecordedEventAttributes”:{“details”:{“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjIwMjEwNS1icm9rZXJBZ3JlZW1lbnQi”}]},“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“6”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82809294},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048587”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“WyIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“7”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82813214},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048588”,“markerRecordedEventAttributes”:{“details”:{“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]},“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“InZlcmlmeVByb2ZpbGUi”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“8”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82823644},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048589”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“WyJ2ZXJpZnlQcm9maWxlLTEiLCIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“9”,“eventTime”:{“seconds”:“1623253200”,“nanos”:82831274},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048590”,“activityTaskScheduledEventAttributes”:{“activityId”:“9”,“activityType”:{“name”:“GetStatusFlagActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFBQXkycGRLbkFLUyt2dE9kMWh5bXp3QVRadXF0ejBtQ3A1QWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjMyNTMxODMwMTUi”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlBlcnNvbmFsRGV0YWlscyI=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“60”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“4”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:5}},“attributes”:“activityTaskScheduledEventAttributes”},{“eventId”:“10”,“eventTime”:{“seconds”:“1623253200”,“nanos”:86185975},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_STARTED”,“version”:“0”,“taskId”:“1048597”,“activityTaskStartedEventAttributes”:{“scheduledEventId”:“9”,“identity”:“980282@archlinux@”,“requestId”:“53104e20-0c03-4966-8e4b-0a9b3895a9f7”,“attempt”:1},“attributes”:“activityTaskStartedEventAttributes”},{“eventId”:“11”,“eventTime”:{“seconds”:“1623253200”,“nanos”:89233977},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048598”,“activityTaskCompletedEventAttributes”:{“result”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“ZmFsc2U=”}]},“scheduledEventId”:“9”,“startedEventId”:“10”,“identity”:“980282@archlinux@”},“attributes”:“activityTaskCompletedEventAttributes”},{“eventId”:“12”,“eventTime”:{“seconds”:“1623253200”,“nanos”:89240407},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048601”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“archlinux:4d64965f-b931-457a-806b-619b66a3cfca”,“kind”:“TASK_QUEUE_KIND_STICKY”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“13”,“eventTime”:{“seconds”:“1623253200”,“nanos”:93012957},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048605”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“12”,“identity”:“980282@archlinux@”,“requestId”:“04c41b3b-b4d8-4a43-af02-8700aec63bdf”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“14”,“eventTime”:{“seconds”:“1623253200”,“nanos”:97254986},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048608”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“12”,“startedEventId”:“13”,“identity”:“980282@archlinux@”,“binaryChecksum”:“f9add7201b68e897cd0239007217d4a1”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“15”,“eventTime”:{“seconds”:“1623253200”,“nanos”:97275436},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048609”,“activityTaskScheduledEventAttributes”:{“activityId”:“ProvidePersonalDetailsActivity”,“activityType”:{“name”:“ProvidePersonalDetailsActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFBQXkycGRLbkFLUyt2dE9kMWh5bXp3QVRadXF0ejBtQ3A1QWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJjYXRlZ29yeSI6ImNsaWVudC1wcm9jZXNzLXN0YXRlIiwiVVVJRCI6ImMzMGUxazRubXZsZmk3azcwanVnIiwic291cmNlIjoiMTYyMzI1MzE4MzAxNS9DbGFzc2ljT25ib2FyZGluZ0NGRCIsIm1lc3NhZ2UiOiJ7XCJQSURcIjpcIkNsYXNzaWNPbmJvYXJkaW5nQ0ZEXCIsXCJDbGllbnRJRFwiOlwiMTYyMzI1MzE4MzAxNVwiLFwiU3RlcFJ1bklEXCI6XCJcIixcIlN0ZXBFcnJvclwiOlwiXCIsXCJTdGVwc0xpc3RcIjpbe1wiTmFtZVwiOlwiUGVyc29uYWxEZXRhaWxzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfSx7XCJOYW1lXCI6XCJSZXNpZGVudGlhbEFkZHJlc3NcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVGF4ZXNDb3VudHJ5XCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRheElEXCIsXCJDYW5Ta2lwXCI6dHJ1ZSxcIkNhblNhdmVcIjpmYWxzZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlNvdXJjZU9mRnVuZHNcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiY2ZkQU1MXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRyYWRpbmdBY2NvdW50UHJlZmVyZW5jaWVzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIkJyb2tlckFncmVlbWVudFwiLFwiQ2FuU2tpcFwiOmZhbHNlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVmVyaWZ5SWRlbnRpdHlcIixcIkNhblNraXBcIjp0cnVlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOmZhbHNlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlZlcmlmeVBPUlwiLFwiQ2FuU2tpcFwiOnRydWUsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfV0sXCJQb3NpdGlvblwiOjAsXCJEb25lXCI6ZmFsc2UsXCJJc1Byb2Nlc3NpbmdcIjpmYWxzZSxcIlN0YXJ0ZWRcIjpmYWxzZSxcIlJlZGlyZWN0XCI6bnVsbCxcIlJlc3VsdFwiOm51bGx9In0=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“30672000”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“14”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:0}},“attributes”:“activityTaskScheduledEventAttributes”},{“eventId”:“16”,“eventTime”:{“seconds”:“1623253200”,“nanos”:100716157},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_STARTED”,“version”:“0”,“taskId”:“1048614”,“activityTaskStartedEventAttributes”:{“scheduledEventId”:“15”,“identity”:“980282@archlinux@”,“requestId”:“ebeba944-167d-48d3-b4a0-ddecaeff23bd”,“attempt”:1},“attributes”:“activityTaskStartedEventAttributes”},{“eventId”:“17”,“eventTime”:{“seconds”:“1623253226”,“nanos”:637457577},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048615”,“activityTaskCompletedEventAttributes”:{“result”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJDb21tYW5kIjoiU0FWRSIsIkRhdGEiOnsiRmlyc3ROYW1lIjoiYXNkZmEiLCJMYXN0TmFtZSI6InNkZmFzZCIsIkRhdGVPZkJpcnRoIjoiMTExMS0xMS0xMSIsIkdlbmRlciI6Im1hbGUiLCJOYXRpb25hbGl0eUNvZGUiOm51bGwsIk5vblVTIjpmYWxzZSwiTm9uUHJvU3Vic2NyaWJlciI6bnVsbH19”}]},“scheduledEventId”:“15”,“startedEventId”:“16”,“identity”:“845434@archlinux@”},“attributes”:“activityTaskCompletedEventAttributes”},{“eventId”:“18”,“eventTime”:{“seconds”:“1623253226”,“nanos”:637471197},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048618”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“archlinux:4d64965f-b931-457a-806b-619b66a3cfca”,“kind”:“TASK_QUEUE_KIND_STICKY”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“19”,“eventTime”:{“seconds”:“1623253231”,“nanos”:638920473},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_TIMED_OUT”,“version”:“0”,“taskId”:“1048622”,“workflowTaskTimedOutEventAttributes”:{“scheduledEventId”:“18”,“startedEventId”:“0”,“timeoutType”:“TIMEOUT_TYPE_SCHEDULE_TO_START”},“attributes”:“workflowTaskTimedOutEventAttributes”},{“eventId”:“20”,“eventTime”:{“seconds”:“1623253231”,“nanos”:638933873},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048623”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“21”,“eventTime”:{“seconds”:“1623253231”,“nanos”:643120822},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048626”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“20”,“identity”:“980857@archlinux@”,“requestId”:“9f9268e6-f106-4eda-91d1-a1dc4231db2f”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“22”,“eventTime”:{“seconds”:“1623253231”,“nanos”:648673099},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_FAILED”,“version”:“0”,“taskId”:“1048629”,“workflowTaskFailedEventAttributes”:{“scheduledEventId”:“20”,“startedEventId”:“21”,“cause”:“WORKFLOW_TASK_FAILED_CAUSE_WORKFLOW_WORKER_UNHANDLED_FAILURE”,“failure”:{“message”:“lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9”,“source”:“GoSDK”,“stackTrace”:“process event for ClientServiceTaskList [panic]:\ngo.temporal.io/sdk/internal.panicIllegalState(0xc00004e9c0, 0x53)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_decision_state_machine.go:393 +0x66\ngo.temporal.io/sdk/internal.(*commandsHelper).handleActivityTaskScheduled(0xc00069a480, 0x28ac108, 0x1, 0x9)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_decision_state_machine.go:878 +0x265\ngo.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc000c104e0, 0xc0000aac00, 0x1, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_event_handlers.go:800 +0x6fa\ngo.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc00011a240, 0xc00092f110, 0x0, 0x0, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_handlers.go:876 +0xc94\ngo.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc0009b24d0, 0xc00092f110, 0xc000507e90, 0x0, 0x0, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_handlers.go:727 +0x6af\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc00064a4e0, 0xc00092f110, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_pollers.go:288 +0x35a\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc00064a4e0, 0x1b52d40, 0xc00092f110, 0x0, 0x0)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_task_pollers.go:259 +0x107\ngo.temporal.io/sdk/internal.(*baseWorker).processTask(0xc000536ff0, 0x1b52d40, 0xc00092f110)\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_worker_base.go:343 +0x19a\ncreated by go.temporal.io/sdk/internal.(*baseWorker).runTaskDispatcher\n\t/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.5.0/internal/internal_worker_base.go:270 +0x365”,“applicationFailureInfo”:{“type”:“PanicError”,“nonRetryable”:true},“failureInfo”:“applicationFailureInfo”},“identity”:“980857@archlinux@”,“baseRunId”:"",“newRunId”:"",“forkEventVersion”:“0”,“binaryChecksum”:“f9add7201b68e897cd0239007217d4a1”},“attributes”:“workflowTaskFailedEventAttributes”}]

samar · June 9, 2021, 3:59pm

Looks like you are using Go SDK v1.5.0. Can you try to reproduce it with v1.6.0? We have some fixes for similar problems in that release.

korvinko · June 10, 2021, 7:30am

There was temporalio/auto-setup:1.6.5 .
Here logs for temporalio/auto-setup:1.9.2

korvinko · June 14, 2021, 8:33am

Hey guys, can you provide any suggestion or workaround for this case?

Vitaly · June 14, 2021, 7:45pm

@korvinko temporal auto-setup sets up a server for you, what Samar was talking about above was an SDK version. Please run go get github.com/temporalio/sdk-go@v1.7.0 on your client code to update to the latest version of go SDK.

korvinko · June 14, 2021, 11:07pm

Sorry, I’ve updated everything, but still face with workflow panics:

2021/06/15 02:02:42 ERROR Workflow panic Namespace clients TaskQueue ClientServiceTaskList WorkerID 421400@archlinux@ WorkflowType ClassicOnboardingCFD WorkflowID 1623711709629/ClassicOnboardingCFD RunID 7265bbea-3db5-427a-b57c-f926291e0c1b Attempt 1 Error lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9 StackTrace process event for ClientServiceTaskList [panic]:

go.mod

go.temporal.io/api v1.4.1-0.20210429213054-a9a257b5cf16
go.temporal.io/sdk v1.7.0
go.temporal.io/server v1.10.2

docker-compose:

temporal-server:
image: temporalio/auto-setup:1.10.2
temporal-web:
image: temporalio/web:1.9.2

History:

[{“eventId”:“1”,“eventTime”:{“seconds”:“1623711719”,“nanos”:543989124},“eventType”:“EVENT_TYPE_WORKFLOW_EXECUTION_STARTED”,“version”:“0”,“taskId”:“1048766”,“workflowExecutionStartedEventAttributes”:{“workflowType”:{“name”:“ClassicOnboardingCFD”},“parentWorkflowNamespace”:“clients”,“parentWorkflowExecution”:{“workflowId”:“1623711709629/SignUpWeb”,“runId”:“71ebf963-35c7-4133-b07d-a8297aa004a1”},“parentInitiatedEventId”:“44”,“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjM3MTE3MDk2Mjki”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlJFVEFJTCI=”}]},“workflowExecutionTimeout”:{“seconds”:“0”,“nanos”:0},“workflowRunTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskTimeout”:{“seconds”:“10”,“nanos”:0},“continuedExecutionRunId”:"",“initiator”:“CONTINUE_AS_NEW_INITIATOR_UNSPECIFIED”,“originalExecutionRunId”:“7265bbea-3db5-427a-b57c-f926291e0c1b”,“identity”:"",“firstExecutionRunId”:“7265bbea-3db5-427a-b57c-f926291e0c1b”,“attempt”:1,“cronSchedule”:"",“firstWorkflowTaskBackoff”:{“seconds”:“0”,“nanos”:0},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFCV01GMlREelBMQmtTWU9PM0RHUnFKQVpQcm9tTWN3NjhTQWdBPSJ9”}}}},“attributes”:“workflowExecutionStartedEventAttributes”},{“eventId”:“2”,“eventTime”:{“seconds”:“1623711719”,“nanos”:548321764},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048772”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“3”,“eventTime”:{“seconds”:“1623711719”,“nanos”:550739633},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048777”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“2”,“identity”:“420853@archlinux@”,“requestId”:“66e947e3-3559-4a58-ad01-517bea6ceeaf”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“4”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555082993},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048783”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“2”,“startedEventId”:“3”,“identity”:“420853@archlinux@”,“binaryChecksum”:“20548eb1e1ddc71114739b64fcff6fa7”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“5”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555106093},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048784”,“markerRecordedEventAttributes”:{“details”:{“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]},“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjIwMjEwNS1icm9rZXJBZ3JlZW1lbnQi”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“6”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555133653},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048785”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“type”:“S2V5d29yZA==”,“encoding”:“anNvbi9wbGFpbg==”},“data”:“WyIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“7”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555140203},“eventType”:“EVENT_TYPE_MARKER_RECORDED”,“version”:“0”,“taskId”:“1048786”,“markerRecordedEventAttributes”:{“details”:{“change-id”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“InZlcmlmeVByb2ZpbGUi”}]},“version”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“MQ==”}]}},“markerName”:“Version”,“workflowTaskCompletedEventId”:“4”},“attributes”:“markerRecordedEventAttributes”},{“eventId”:“8”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555157753},“eventType”:“EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES”,“version”:“0”,“taskId”:“1048787”,“upsertWorkflowSearchAttributesEventAttributes”:{“workflowTaskCompletedEventId”:“4”,“searchAttributes”:{“indexedFields”:{“TemporalChangeVersion”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”,“type”:“S2V5d29yZA==”},“data”:“WyJ2ZXJpZnlQcm9maWxlLTEiLCIyMDIxMDUtYnJva2VyQWdyZWVtZW50LTEiXQ==”}}}},“attributes”:“upsertWorkflowSearchAttributesEventAttributes”},{“eventId”:“9”,“eventTime”:{“seconds”:“1623711719”,“nanos”:555172433},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048788”,“activityTaskScheduledEventAttributes”:{“activityId”:“9”,“activityType”:{“name”:“GetStatusFlagActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFCV01GMlREelBMQmtTWU9PM0RHUnFKQVpQcm9tTWN3NjhTQWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IjE2MjM3MTE3MDk2Mjki”},{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“IlBlcnNvbmFsRGV0YWlscyI=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“60”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“4”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:5}},“attributes”:“activityTaskScheduledEventAttributes”},{“eventId”:“10”,“eventTime”:{“seconds”:“1623711719”,“nanos”:558921226},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_STARTED”,“version”:“0”,“taskId”:“1048801”,“activityTaskStartedEventAttributes”:{“scheduledEventId”:“9”,“identity”:“420853@archlinux@”,“requestId”:“b3661bff-fe3d-4ce5-afd6-9e02cabf5a14”,“attempt”:1},“attributes”:“activityTaskStartedEventAttributes”},{“eventId”:“11”,“eventTime”:{“seconds”:“1623711719”,“nanos”:563030727},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048802”,“activityTaskCompletedEventAttributes”:{“result”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“ZmFsc2U=”}]},“scheduledEventId”:“9”,“startedEventId”:“10”,“identity”:“420853@archlinux@”},“attributes”:“activityTaskCompletedEventAttributes”},{“eventId”:“12”,“eventTime”:{“seconds”:“1623711719”,“nanos”:563035517},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048803”,“workflowTaskScheduledEventAttributes”:{“taskQueue”:{“name”:“archlinux:6900d039-2406-41e1-87f6-0284e4d97d32”,“kind”:“TASK_QUEUE_KIND_STICKY”},“startToCloseTimeout”:{“seconds”:“10”,“nanos”:0},“attempt”:1},“attributes”:“workflowTaskScheduledEventAttributes”},{“eventId”:“13”,“eventTime”:{“seconds”:“1623711719”,“nanos”:567720495},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_STARTED”,“version”:“0”,“taskId”:“1048818”,“workflowTaskStartedEventAttributes”:{“scheduledEventId”:“12”,“identity”:“420853@archlinux@”,“requestId”:“4e51d674-aa60-4b2c-b595-5597fadafdd5”},“attributes”:“workflowTaskStartedEventAttributes”},{“eventId”:“14”,“eventTime”:{“seconds”:“1623711719”,“nanos”:572633663},“eventType”:“EVENT_TYPE_WORKFLOW_TASK_COMPLETED”,“version”:“0”,“taskId”:“1048822”,“workflowTaskCompletedEventAttributes”:{“scheduledEventId”:“12”,“startedEventId”:“13”,“identity”:“420853@archlinux@”,“binaryChecksum”:“20548eb1e1ddc71114739b64fcff6fa7”},“attributes”:“workflowTaskCompletedEventAttributes”},{“eventId”:“15”,“eventTime”:{“seconds”:“1623711719”,“nanos”:572657003},“eventType”:“EVENT_TYPE_ACTIVITY_TASK_SCHEDULED”,“version”:“0”,“taskId”:“1048823”,“activityTaskScheduledEventAttributes”:{“activityId”:“ProvidePersonalDetailsActivity”,“activityType”:{“name”:“ProvidePersonalDetailsActivity”},“namespace”:"",“taskQueue”:{“name”:“ClientServiceTaskList”,“kind”:“TASK_QUEUE_KIND_NORMAL”},“header”:{“fields”:{“opencensusPayload”:{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJEYXRhIjoiQUFCV01GMlREelBMQmtTWU9PM0RHUnFKQVpQcm9tTWN3NjhTQWdBPSJ9”}}},“input”:{“payloads”:[{“metadata”:{“encoding”:“anNvbi9wbGFpbg==”},“data”:“eyJjYXRlZ29yeSI6ImNsaWVudC1wcm9jZXNzLXN0YXRlIiwiVVVJRCI6ImMzM3R2cHNubXZsNmZrOWNxNjYwIiwic291cmNlIjoiMTYyMzcxMTcwOTYyOS9DbGFzc2ljT25ib2FyZGluZ0NGRCIsIm1lc3NhZ2UiOiJ7XCJQSURcIjpcIkNsYXNzaWNPbmJvYXJkaW5nQ0ZEXCIsXCJDbGllbnRJRFwiOlwiMTYyMzcxMTcwOTYyOVwiLFwiU3RlcFJ1bklEXCI6XCJcIixcIlN0ZXBFcnJvclwiOlwiXCIsXCJTdGVwc0xpc3RcIjpbe1wiTmFtZVwiOlwiUGVyc29uYWxEZXRhaWxzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfSx7XCJOYW1lXCI6XCJSZXNpZGVudGlhbEFkZHJlc3NcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVGF4ZXNDb3VudHJ5XCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRheElEXCIsXCJDYW5Ta2lwXCI6dHJ1ZSxcIkNhblNhdmVcIjpmYWxzZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlNvdXJjZU9mRnVuZHNcIixcIkNhblNraXBcIjpmYWxzZSxcIkNhblNhdmVcIjp0cnVlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiY2ZkQU1MXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlRyYWRpbmdBY2NvdW50UHJlZmVyZW5jaWVzXCIsXCJDYW5Ta2lwXCI6ZmFsc2UsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjp0cnVlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIkJyb2tlckFncmVlbWVudFwiLFwiQ2FuU2tpcFwiOmZhbHNlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOnRydWUsXCJDYW5DYW5jZWxcIjpmYWxzZX0se1wiTmFtZVwiOlwiVmVyaWZ5SWRlbnRpdHlcIixcIkNhblNraXBcIjp0cnVlLFwiQ2FuU2F2ZVwiOmZhbHNlLFwiQ2FuQmFja1wiOmZhbHNlLFwiQ2FuQ2FuY2VsXCI6ZmFsc2V9LHtcIk5hbWVcIjpcIlZlcmlmeVBPUlwiLFwiQ2FuU2tpcFwiOnRydWUsXCJDYW5TYXZlXCI6dHJ1ZSxcIkNhbkJhY2tcIjpmYWxzZSxcIkNhbkNhbmNlbFwiOmZhbHNlfV0sXCJQb3NpdGlvblwiOjAsXCJEb25lXCI6ZmFsc2UsXCJJc1Byb2Nlc3NpbmdcIjpmYWxzZSxcIlN0YXJ0ZWRcIjpmYWxzZSxcIlJlZGlyZWN0XCI6bnVsbCxcIlJlc3VsdFwiOm51bGx9In0=”}]},“scheduleToCloseTimeout”:{“seconds”:“0”,“nanos”:0},“scheduleToStartTimeout”:{“seconds”:“10”,“nanos”:0},“startToCloseTimeout”:{“seconds”:“30672000”,“nanos”:0},“heartbeatTimeout”:{“seconds”:“0”,“nanos”:0},“workflowTaskCompletedEventId”:“14”,“retryPolicy”:{“nonRetryableErrorTypes”:,“initialInterval”:{“seconds”:“1”,“nanos”:0},“backoffCoefficient”:2,“maximumInterval”:{“seconds”:“100”,“nanos”:0},“maximumAttempts”:0}},“attributes”:“activityTaskScheduledEventAttributes”}]

korvinko · June 15, 2021, 6:56am

And there is another issue that is could be related to this topic.

We tried to restart broken workflows by adding follow options for worker:

workerOptions := worker.Options{
WorkflowPanicPolicy: worker.FailWorkflow,
}

And for execution of workflow:

cwo := workflow.ChildWorkflowOptions{
WorkflowID: api.GetClientWorkflowID(workflowType, clientID),
WorkflowIDReusePolicy: enums.WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE,
ParentClosePolicy: enums.PARENT_CLOSE_POLICY_ABANDON,
RetryPolicy: &temporal.RetryPolicy{
MaximumAttempts: 5,
},
}
cwCtx := workflow.WithChildOptions(ctx, cwo)

cwFuture := workflow.ExecuteChildWorkflow(cwCtx, workflowType, clientID, clientCategory)
if cwFuture == nil {
return errors.New(“ExecuteChildWorkflow failed”)
}

After a restart of go process we’ve seen just two panics :

2021/06/15 09:48:30 ERROR Workflow panic Namespace clients TaskQueue ClientServiceTaskList WorkerID 437054@archlinux@ WorkflowType ClassicOnboardingCFD WorkflowID 1623739666529/ClassicOnboardingCFD RunID 1429d782-ad3e-4ce1-94e7-0acf0f53981b Attempt 1 Error lookup failed for scheduledEventID to activityID: scheduleEventID: 9, activityID: 9 StackTrace process event for ClientServiceTaskList [panic]:
go.temporal.io/sdk/internal.panicIllegalState(0xc0001c1380, 0x53)
/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_decision_state_machine.go:395 +0x66
go.temporal.io/sdk/internal.(*commandsHelper).handleActivityTaskScheduled(0xc00082abe0, 0x2bc1cc8, 0x1, 0x9)
/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_decision_state_machine.go:891 +0x265
go.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc0008168e8, 0xc000d12880, 0x1, 0x0, 0x0)
/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_event_handlers.go:813 +0x6fa
go.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc000738cb0, 0xc000d732c0, 0x0, 0x0, 0x0, 0x0)
/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_task_handlers.go:875 +0xc94
go.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc000520f20, 0xc000d732c0, 0xc000283b00, 0x0, 0x0, 0x0, 0x0)
/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_task_handlers.go:726 +0x6af
go.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc0006fe410, 0xc000d732c0, 0x0, 0x0)
/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_task_pollers.go:286 +0x35a
go.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc0006fe410, 0x1d22560, 0xc000d732c0, 0x0, 0x0)
/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_task_pollers.go:257 +0x107
go.temporal.io/sdk/internal.(*baseWorker).processTask(0xc0007604b0, 0x1d22560, 0xc000d732c0)
/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_worker_base.go:343 +0x19a
created by go.temporal.io/sdk/internal.(*baseWorker).runTaskDispatcher
/home/korvinko/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_worker_base.go:270 +0x365

And then the same 5 workflow panics, but nothing changed in the statuses of these workflows. Could you please clarify what’s going on? Why did nothing change?

korvinko · June 21, 2021, 1:09pm

Hey guys, I’ve added more debug information for you:

We can see, that instead of ID of activity in history we have an ID of the marker. Probably, it helps you to find the issue.

Vitaly · June 22, 2021, 1:20am

I was able to reproduce the issue with fairly simple code:

// SampleVersionedWorkflow workflow definition
func SampleVersionedWorkflow(ctx workflow.Context, name string) error {
	ao := workflow.ActivityOptions{
		ScheduleToStartTimeout: time.Minute,
		StartToCloseTimeout:    time.Minute,
		HeartbeatTimeout:       time.Second * 20,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	logger := workflow.GetLogger(ctx)
	logger.Info("helloworld workflow started")
	var result string

	workflow.GetVersion(ctx, "test-change", workflow.DefaultVersion, 1)

	workflow.GetVersion(ctx, "test-change-2", workflow.DefaultVersion, 1)

	workflow.GetVersion(ctx, "test-change-3", workflow.DefaultVersion, 1)

	err := workflow.ExecuteActivity(ctx, HelloWorldActivity, name).Get(ctx, &result)
	if err != nil {
		logger.Error("Activity failed.", "Error", err)
		return err
	}
	timerFuture := workflow.NewTimer(ctx, 30*time.Second)
	err = timerFuture.Get(ctx, nil)
	if err != nil {
		return err
	}

	logger.Info("Workflow completed.", "Result", result)

	return nil
}

func HelloWorldActivity(ctx context.Context, name string) (string, error) {
	logger := activity.GetLogger(ctx)
	logger.Info("hello world activity started")
	return "Hello " + name + "!", nil
}

If I stop the worker while timer is in progress and restart it back I get an error you’ve mentioned above:

2021/06/21 18:18:53 ERROR Workflow panic Namespace default TaskQueue timer WorkerID 56567@arch@ WorkflowType SampleVersionedWorkflow WorkflowID versioning_0d959985-ffcd-4d94-92c8-ab0b7d125380 RunID 4d34d8dc-6ca1-4c56-83ca-1b565fc2004a Attempt 14 Error lookup failed for scheduledEventID to activityID: scheduleEventID: 11, activityID: 11 StackTrace process event for timer [panic]:
go.temporal.io/sdk/internal.panicIllegalState(...)
        /home/vit/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_decision_state_machine.go:395
go.temporal.io/sdk/internal.(*commandsHelper).handleActivityTaskScheduled(0xc000b101e0, 0xc000614258, 0x2, 0xb)
        /home/vit/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_decision_state_machine.go:891 +0x165
go.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc0008aa198, 0xc00066c340, 0x1, 0x0, 0x0)
        /home/vit/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_event_handlers.go:813 +0x42e
go.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc00031c0e0, 0xc0008aefc0, 0x145d9e0, 0xc0003cc770, 0xc00031c0e0, 0x0)
        /home/vit/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_task_handlers.go:875 +0x72c
go.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc00032fef0, 0xc0008aefc0, 0xc000b06210, 0x0, 0x0, 0x0, 0x0)
        /home/vit/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_task_handlers.go:726 +0x6d9
go.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc00035f5f0, 0xc0008aefc0, 0x0, 0x0)
        /home/vit/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_task_pollers.go:286 +0x4ae
go.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc00035f5f0, 0xcf7300, 0xc0008aefc0, 0x0, 0x0)
        /home/vit/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_task_pollers.go:257 +0x85
go.temporal.io/sdk/internal.(*baseWorker).processTask(0xc0003c00f0, 0xcf6ec0, 0xc00060a0c0)
        /home/vit/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_worker_base.go:343 +0xb8
created by go.temporal.io/sdk/internal.(*baseWorker).runTaskDispatcher
        /home/vit/go/pkg/mod/go.temporal.io/sdk@v1.7.0/internal/internal_worker_base.go:270 +0xff

Going to debug it, will let you know once once there is some progress.

Vitaly · June 22, 2021, 1:38am

So the problem was in the incrementNextCommandEventIDIfVersionMarker function that used to only increment next ID once if there were version marker(s) present in the history, this PR addresses it. We should be releasing new version of the SDK soon and you’ll be able to take advantage of this fix. Please let us know if this is a pressing issue and we can expedite release process.

korvinko · June 22, 2021, 9:21am

This PR solved our problem. Thanks a lot.