A workflow containing a timer encounters an error after the worker restarting

Community Support
1

Hi, I encountered an error while using Temporal in my Go language project.

  • The Temporal server version is 2.28.0.
  • The Temporal SDK version is v1.26.1.

Execution Process

  1. Start a worker.
  2. Execute the workflow, which creates a timer and waits.
  3. Stop the worker and restart it.
  4. When the timer fired, an error message indicating workflowTaskFailed event will appear.

Error log

{
  "message": "[TMPRL1100] unknown command CommandType: Timer, ID: 17, possible causes are nondeterministic workflow definition code or incompatible change in the workflow definition",
  "source": "GoSDK",
  "stackTrace": "process event for order_matching_task_queue [panic]:\ngo.temporal.io/sdk/internal.panicIllegalState(...)\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_command_state_machine.go:440\ngo.temporal.io/sdk/internal.(*commandsHelper).getCommand(0xc00658e080, {0x4, {0xc006655728, 0x2}})\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_command_state_machine.go:929 +0x3a5\ngo.temporal.io/sdk/internal.(*commandsHelper).handleTimerStarted(0x1f08f841348?, {0xc006655728?, 0xc000601008?})\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_command_state_machine.go:1385 +0x25\ngo.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc006588108, 0xc0066aae70, 0xa0?, 0x0)\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_event_handlers.go:1175 +0x3f1\ngo.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc006598000, 0xc0066aced0)\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_task_handlers.go:1100 +0x1608\ngo.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc0042f0000, 0xc0066aced0, 0xc006598000, 0xc00658a5a0)\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_task_handlers.go:889 +0x3c5\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc00391c240, 0xc0066aced0)\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_task_pollers.go:357 +0x3c3\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc00391c240, {0x1adcba0, 0xc0066aced0})\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_task_pollers.go:321 +0x78\ngo.temporal.io/sdk/internal.(*baseWorker).processTask(0xc0043ba160, {0x1add6e0, 0xc0066a8780})\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_worker_base.go:518 +0x153\ngo.temporal.io/sdk/internal.(*baseWorker).processTaskAsync.func1()\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_worker_base.go:369 +0x45\ncreated by go.temporal.io/sdk/internal.(*baseWorker).processTaskAsync in goroutine 71\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_worker_base.go:365 +0xa5",
  "applicationFailureInfo": {
    "type": "PanicError",
    "nonRetryable": true
  }
}

Self-diagnosis Information

When locating the (*commandsHelper).getCommand method in internal/internal_command_state_machine, I found that in h.commands, there exists a CommandType: Activity with ID: 17, but the commandID that needs to be retrieved is CommandType: Timer with the same ID 17. The IDs are the same, but the types are different.

My question is how to avoid this error after a restart?

Looking forward to your answer. Thanks!

2

Make sure that the workflow code is deterministic. See Core application - Go SDK | Temporal Platform Documentation

3

I will provide a workflow history, hoping it will be helpful in identifying the cause of the issue.

{
  "events": [
    {
      "eventId": "1",
      "eventTime": "2024-10-23T08:26:48.775296900Z",
      "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_STARTED",
      "taskId": "20971758",
      "workflowExecutionStartedEventAttributes": {
        "workflowType": {
          "name": "OrderWorkflowWaitingMatching"
        },
        "parentWorkflowNamespace": "default",
        "parentWorkflowNamespaceId": "c70ccd65-aea9-45b7-bf93-2697cab7dac1",
        "parentWorkflowExecution": {
          "workflowId": "order-workflow-test-apply-id-cscb6ehre9f60a0cvcfg",
          "runId": "294b9c7f-b308-47ac-9b39-56aee9873aec"
        },
        "parentInitiatedEventId": "23",
        "taskQueue": {
          "name": "order_matching_task_queue",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "InRlc3QtYXBwbHktaWQi"
            },
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "ImNzY2I2ZWhyZTlmNjBhMGN2Y2ZnIg=="
            }
          ]
        },
        "workflowExecutionTimeout": "0s",
        "workflowRunTimeout": "0s",
        "workflowTaskTimeout": "10s",
        "originalExecutionRunId": "b2b870a3-33bb-487e-abf2-26204069618e",
        "firstExecutionRunId": "b2b870a3-33bb-487e-abf2-26204069618e",
        "attempt": 1,
        "firstWorkflowTaskBackoff": "0s",
        "header": {},
        "workflowId": "match-waiting-workflow-test-apply-id-cscb6ehre9f60a0cvcfg",
        "rootWorkflowExecution": {
          "workflowId": "order-workflow-test-apply-id-cscb6ehre9f60a0cvcfg",
          "runId": "294b9c7f-b308-47ac-9b39-56aee9873aec"
        }
      }
    },
    {
      "eventId": "2",
      "eventTime": "2024-10-23T08:26:48.786515400Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
      "taskId": "20971768",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "order_matching_task_queue",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "3",
      "eventTime": "2024-10-23T08:26:48.801773800Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
      "taskId": "20971774",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "2",
        "identity": "25240@JYM-HpZhanX@",
        "requestId": "456dc1b3-8f9a-4e47-969a-225a254f0b2f",
        "historySizeBytes": "638",
        "workerVersion": {
          "buildId": "5e4efe030bb177ed5afa5533417de3d6"
        }
      }
    },
    {
      "eventId": "4",
      "eventTime": "2024-10-23T08:26:48.812868600Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
      "taskId": "20971781",
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "2",
        "startedEventId": "3",
        "identity": "25240@JYM-HpZhanX@",
        "workerVersion": {
          "buildId": "5e4efe030bb177ed5afa5533417de3d6"
        },
        "sdkMetadata": {
          "langUsedFlags": [
            3
          ],
          "sdkName": "temporal-go",
          "sdkVersion": "1.26.1"
        },
        "meteringMetadata": {}
      }
    },
    {
      "eventId": "5",
      "eventTime": "2024-10-23T08:26:48.812868600Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
      "taskId": "20971782",
      "activityTaskScheduledEventAttributes": {
        "activityId": "5",
        "activityType": {
          "name": "OrderInfoActivity"
        },
        "taskQueue": {
          "name": "order_matching_task_queue",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "header": {},
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "InRlc3QtYXBwbHktaWQi"
            },
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "ImNzY2I2ZWhyZTlmNjBhMGN2Y2ZnIg=="
            }
          ]
        },
        "scheduleToCloseTimeout": "60s",
        "scheduleToStartTimeout": "60s",
        "startToCloseTimeout": "60s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "4",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s"
        },
        "useWorkflowBuildId": true
      }
    },
    {
      "eventId": "6",
      "eventTime": "2024-10-23T08:26:48.824698100Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
      "taskId": "20971789",
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "5",
        "identity": "25240@JYM-HpZhanX@",
        "requestId": "4fbc4cf6-9d21-4a37-94b4-62e4612a11ad",
        "attempt": 1,
        "workerVersion": {
          "buildId": "5e4efe030bb177ed5afa5533417de3d6"
        }
      }
    },
    {
      "eventId": "7",
      "eventTime": "2024-10-23T08:26:49.062583600Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
      "taskId": "20971790",
      "activityTaskCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "eyJBcHBseSI6eyJJZCI6InRlc3QtYXBwbHktaWQiLCJDcmVhdGVkQXQiOiIyMDI0LTEwLTIzVDE2OjI2OjM0LjgzODk4OSswODowMCIsIlVwZGF0ZWRBdCI6IjIwMjQtMTAtMjNUMTY6MjY6MzQuODM4OTg5KzA4OjAwIiwiRGVsZXRlZEF0IjpudWxsLCJWZXJzaW9uIjoxLCJTZW5kVXNlcklkIjoiY3E4OHVnbTBlZzVqYnFnNWVmdjAiLCJTZW5kVXNlckdwcyI6WzEwOS41MTkyOCwxOC4yNzYxNjNdLCJUeXBlIjoicmVhbF90aW1lIiwiU3RhdHVzIjoib3JkZXJfbWF0Y2hpbmciLCJUb3RhbFByaWNlIjoiMjAuMiIsIlByaWNlIjoiMjAiLCJTZXJ2aWNlUHJpY2UiOiIwLjIiLCJUcmFkZU5vIjoiIiwiRW1wbG95ZXJJZCI6IiIsIlRyYW5zYWN0aW9uSWQiOiIiLCJDcmVhdGVUaW1lIjpudWxsLCJQYXlUaW1lIjpudWxsLCJDbG9zZVRpbWUiOm51bGwsIlNlbmRVc2VyTW9kZSI6ImVtcGxveWVyX3BlcnNvbiIsIlBheVR5cGUiOiIiLCJTZW5kVXNlckFkZHIiOiLmtbfljZfnnIHkuInkuprluILlkInpmLPljLrmnIjlhYnot68iLCJTZW5kVXNlckFkZHJOYW1lIjoi5rWL6K+V5Zyw5Z2A5Yy56YWN5ZCOIiwiU2VuZFVzZXJIb3VzZU51bWJlciI6IjI0MTAyMzE2MjYzNCIsIkN0bFN0YXR1cyI6IiIsIlNlbmRVc2VyIjpudWxsLCJPcmRlclNlbmRzIjpudWxsLCJPcmRlclJlY2VpdmVzIjpudWxsfSwiU2VuZCI6eyJJZCI6ImNzY2I2ZWhyZTlmNjBhMGN2Y2ZnIiwiQ3JlYXRlZEF0IjoiMjAyNC0xMC0yM1QxNjoyNjozNC45ODg5MTMrMDg6MDAiLCJVcGRhdGVkQXQiOiIyMDI0LTEwLTIzVDE2OjI2OjQ2LjY2MzY5MiswODowMCIsIkRlbGV0ZWRBdCI6bnVsbCwiVmVyc2lvbiI6Mywib3JkZXJBcHBseUlkIjoidGVzdC1hcHBseS1pZCIsInByaWNlIjoiMjAiLCJvcmlnaW5hbFByaWNlIjoiMCIsInN0YXR1cyI6Im1hdGNoaW5nX3dhaXRpbmciLCJqb2JJZCI6ImNzY2I2ZWhyZTlmNjBhMGN2Y2YwIiwiaXNQZW5hbGl6ZWQiOjAsIkVtcGxveWVlSWQiOiJjcWtvbWNwcmU5ZjByMjVyMjdhMCIsIlN0YXJ0VGltZSI6bnVsbCwiRW5kVGltZSI6bnVsbCwiQ2FuY2VsZWRBdCI6bnVsbCwiQ2FuY2VsUmVhc29uIjoiIiwiSXNFdmFsdWF0ZWQiOjAsIlNlcnZpY2VQcmljZSI6IjAuMiIsIldvcmtmbG93SWQiOiIiLCJSdW5JZCI6IjI5NGI5YzdmLWIzMDgtNDdhYy05YjM5LTU2YWVlOTg3M2FlYyIsIkNoaWxkV29ya2Zsb3dJZCI6Im1hdGNoaW5nLXdvcmtmbG93LXRlc3QtYXBwbHktaWQtY3NjYjZlaHJlOWY2MGEwY3ZjZmciLCJDaGlsZFJ1bklkIjoiOTM2MTVhYTUtODEyZC00OTI2LWE3M2YtMzE4OGM3ZTRmNDc5IiwiRXN0aW1hdGVkRGVwYXJ0VGltZSI6IjIwMjQtMTAtMjNUMTY6MzM6NDYuNjYyNzA3KzA4OjAwIiwiRXN0aW1hdGVkQXJyaXZlVGltZSI6bnVsbCwiTWF0Y2hMaW1pdFRpbWUiOiIyMDI0LTEwLTIzVDE2OjM2OjM1LjY0Nzk1OSswODowMCIsIk9yZGVyQXBwbHkiOm51bGwsIk9yZGVyUmVjZWl2ZSI6eyJJZCI6IiIsIkNyZWF0ZWRBdCI6IjAwMDEtMDEtMDFUMDA6MDA6MDBaIiwiVXBkYXRlZEF0IjoiMDAwMS0wMS0wMVQwMDowMDowMFoiLCJEZWxldGVkQXQiOm51bGwsIlZlcnNpb24iOm51bGwsIk9yZGVyU2VuZElkIjoiIiwiT3JkZXJBcHBseUlkIjoiIiwiUmVjZWl2ZVVzZXJJZCI6IiIsIlByaWNlIjoiMCIsIlN0YXR1cyI6IiIsIkVtcGxveWVlSWQiOiIiLCJBcnJpdmVkVGltZSI6bnVsbCwiSXNQZW5hbGl6ZWQiOjAsIlBlbmFsaXplZFJlYXNvbiI6IiIsIlBlbmFsaXplZFJlc3VsdCI6IiIsIlBlbmFsaXplZFRpbWUiOm51bGwsIklzRmFjZVZhbGlkIjowLCJDYW5DYW5jZWxUaW1lIjoiMDAwMS0wMS0wMVQwMDowMDowMFoiLCJDYW5jZWxSZWFzb24iOiIiLCJDYW5jZWxlZEF0IjpudWxsLCJPcmRlclR5cGUiOiIiLCJSZWNlaXZlVXNlckdwcyI6bnVsbCwiUmVjZWl2ZVVzZXJBZGRyIjoiIiwiSXNHcmFiYmluZyI6MCwiRmluaXNoVGltZSI6bnVsbCwiSXNFdmFsdWF0ZWQiOjAsIkN0bFN0YXR1cyI6IiIsIlJlY2VpdmVVc2VyIjpudWxsLCJFbXBsb3llZSI6bnVsbCwiT3JkZXJTZW5kIjpudWxsLCJPcmRlckFwcGx5IjpudWxsfSwiT3JkZXJSZXdhcmQiOnsiSWQiOiIiLCJDcmVhdGVkQXQiOiIwMDAxLTAxLTAxVDAwOjAwOjAwWiIsIlVwZGF0ZWRBdCI6IjAwMDEtMDEtMDFUMDA6MDA6MDBaIiwiRGVsZXRlZEF0IjpudWxsLCJWZXJzaW9uIjpudWxsLCJPcmRlclJlY2VpdmVJZCI6IiIsIk9yZGVyU2VuZElkIjoiIiwiT3JkZXJBcHBseUlkIjoiIiwiUHJpY2UiOiIwIiwiUGF5VGltZSI6bnVsbCwiQ2xvc2VUaW1lIjpudWxsLCJTdGF0dXMiOiIiLCJUcmFuc2FjdGlvbklkIjoiIiwiUGF5VHlwZSI6IiIsIlRyYWRlTm8iOiIiLCJPcmRlclJlY2VpdmUiOm51bGwsIk9yZGVyU2VuZCI6bnVsbCwiT3JkZXJBcHBseSI6bnVsbH0sIkpvYiI6bnVsbH0sIlJlY2VpdmUiOnsiSWQiOiJjc2NiNmhocmU5ZjY1NjJtZnFnZyIsIkNyZWF0ZWRBdCI6IjIwMjQtMTAtMjNUMTY6MjY6NDYuNzA5ODY3KzA4OjAwIiwiVXBkYXRlZEF0IjoiMjAyNC0xMC0yM1QxNjoyNjo0Ni43MDk4NjcrMDg6MDAiLCJEZWxldGVkQXQiOm51bGwsIlZlcnNpb24iOjEsIk9yZGVyU2VuZElkIjoiY3NjYjZlaHJlOWY2MGEwY3ZjZmciLCJPcmRlckFwcGx5SWQiOiJ0ZXN0LWFwcGx5LWlkIiwiUmVjZWl2ZVVzZXJJZCI6ImNxa29lamhyZTlmNmQxMGdmbnEwIiwiUHJpY2UiOiIxOS44IiwiU3RhdHVzIjoibWF0Y2hpbmdfd2FpdGluZyIsIkVtcGxveWVlSWQiOiJjcWtvbWNwcmU5ZjByMjVyMjdhMCIsIkFycml2ZWRUaW1lIjpudWxsLCJJc1BlbmFsaXplZCI6MCwiUGVuYWxpemVkUmVhc29uIjoiIiwiUGVuYWxpemVkUmVzdWx0IjoiIiwiUGVuYWxpemVkVGltZSI6bnVsbCwiSXNGYWNlVmFsaWQiOjAsIkNhbkNhbmNlbFRpbWUiOiIyMDI0LTEwLTIzVDE2OjI4OjQ2LjcwOTIyMiswODowMCIsIkNhbmNlbFJlYXNvbiI6IiIsIkNhbmNlbGVkQXQiOm51bGwsIk9yZGVyVHlwZSI6InJlYWxfdGltZSIsIlJlY2VpdmVVc2VyR3BzIjpbMTA5LjUxOTI4MDU5ODk1ODMzLDE4LjI3NjE2MjkyMzE3NzA4M10sIlJlY2VpdmVVc2VyQWRkciI6Iua1t+WNl+ecgeS4ieS6muW4guWQiemYs+WMuuays+S4nOWMuuihl+mBk+aksOWbreW3t+WNk+i+vuS4nOaWueW3tOWTiOmprOeDreW4pumbqOael+aksOmjjuiLkSIsIklzR3JhYmJpbmciOjAsIkZpbmlzaFRpbWUiOm51bGwsIklzRXZhbHVhdGVkIjowLCJDdGxTdGF0dXMiOiIiLCJSZWNlaXZlVXNlciI6eyJJZCI6ImNxa29lamhyZTlmNmQxMGdmbnEwIiwiQ3JlYXRlZEF0IjoiMjAyNC0wNy0zMVQwODozNzozNC40MTkxODYrMDg6MDAiLCJVcGRhdGVkQXQiOiIyMDI0LTEwLTE0VDE1OjUxOjE4Ljk0Nzc5NCswODowMCIsIkRlbGV0ZWRBdCI6bnVsbCwiVmVyc2lvbiI6NDEsIk5hbWUiOiLmjqXljZUiLCJQaG9uZSI6IjEzNTA0MTIwMDAxIiwiQXZhdGFyIjoiIiwiTGFzdExvZ2luSXAiOiIzNi4xMDEuMTI5LjIzNSIsIkxhc3RMb2dpblRpbWUiOiIyMDI0LTEwLTE0VDE1OjUxOjE4Ljk0NjQwNiswODowMCIsIklzRW1wbG95ZXJDb21wYW55IjowLCJJc0VtcGxveWVyUGVyc29uIjowLCJJc0VtcGxveWVlUGVyc29uIjoxLCJMYXN0TG9naW5Nb2RlIjoiZW1wbG95ZWVfcGVyc29uIiwiTGFzdExvZ2luUGxhdGZvcm0iOiJNaW5pIiwiT3BlbklkIjoib1dxQ2M2MS11aGxybHhzZzc4R2pGMW1Nd0FiQSIsIlVuaW9uSWQiOiJvMDlrYzZna0pTM0pwMjlsVF9yZzBqYm1CLW1vIiwiSW52aXRlVXNlcklkIjoiIiwiSXNPcGVuVm9pY2UiOiJvcGVuIiwiSXNPcGVuTWVzc2FnZSI6Im9wZW4iLCJFbXBsb3llckNvbXBhbnkiOm51bGwsIkVtcGxveWVyUGVyc29uIjpudWxsLCJFbXBsb3llZVBlcnNvbiI6bnVsbH0sIkVtcGxveWVlIjpudWxsLCJPcmRlclNlbmQiOm51bGwsIk9yZGVyQXBwbHkiOm51bGx9LCJVc2VyR2VvIjp7IklkIjoiIiwiQ3JlYXRlZEF0IjoiMDAwMS0wMS0wMVQwMDowMDowMFoiLCJVcGRhdGVkQXQiOiIwMDAxLTAxLTAxVDAwOjAwOjAwWiIsIkRlbGV0ZWRBdCI6bnVsbCwiVmVyc2lvbiI6bnVsbCwiR2VvR3BzIjpudWxsLCJVc2VySWQiOiIiLCJFbXBsb3llZUlkIjoiIiwiT3JkZXJSZWNlaXZlSWQiOiIiLCJUb3RhbERpc3RhbmNlIjowLCJEaXN0YW5jZSI6MCwiVG90YWxUaW1lIjowLCJMZWZ0VGltZSI6MH0sIkpvYiI6eyJJZCI6ImNzY2I2ZWhyZTlmNjBhMGN2Y2YwIiwiQ3JlYXRlZEF0IjoiMjAyNC0xMC0yM1QxNjoyNjozNC43OTM2NDYrMDg6MDAiLCJVcGRhdGVkQXQiOiIyMDI0LTEwLTIzVDE2OjI2OjM0Ljc5MzY0NiswODowMCIsIkRlbGV0ZWRBdCI6bnVsbCwiVmVyc2lvbiI6MSwiSm9iVHlwZUlkIjoiY3Bwb21sdTBlZzVwcmVlYTA5MzAiLCJXb3JrVGltZSI6MSwiRW1wbG95ZWVBZ2UiOiJ1c2VyX2FnZV8xOF8zMCIsIkVtcGxveWVlU2V4IjoiIiwiRW1wbG95ZWVEZWdyZWUiOiIiLCJXb3JrRXhwZXJpZW5jZSI6IiIsIlJlbWFyayI6IiIsIkpvYlR5cGUiOm51bGx9LCJKb2JFeHQiOnsiSWQiOiIiLCJDcmVhdGVkQXQiOiIwMDAxLTAxLTAxVDAwOjAwOjAwWiIsIlVwZGF0ZWRBdCI6IjAwMDEtMDEtMDFUMDA6MDA6MDBaIiwiRGVsZXRlZEF0IjpudWxsLCJWZXJzaW9uIjpudWxsLCJKb2JJZCI6IiIsIkV4dHJhV29ya1RpbWUiOjAsIlN0YXR1cyI6IiIsIlN0YXJ0VGltZSI6bnVsbCwiRW5kVGltZSI6bnVsbCwiUGF5VGltZSI6bnVsbCwiQ2xvc2VUaW1lIjpudWxsLCJUcmFuc2FjdGlvbklkIjoiIiwiUGF5VHlwZSI6IiIsIlRyYWRlTm8iOiIiLCJQcmljZSI6IjAifX0="
            }
          ]
        },
        "scheduledEventId": "5",
        "startedEventId": "6",
        "identity": "25240@JYM-HpZhanX@"
      }
    },
    {
      "eventId": "8",
      "eventTime": "2024-10-23T08:26:49.062583600Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
      "taskId": "20971791",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "JYM-HpZhanX:52e8df8d-4cb1-497e-b337-503cf76f8e9f",
          "kind": "TASK_QUEUE_KIND_STICKY",
          "normalName": "order_matching_task_queue"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "9",
      "eventTime": "2024-10-23T08:26:49.075388500Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
      "taskId": "20971795",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "8",
        "identity": "25240@JYM-HpZhanX@",
        "requestId": "aae44e80-2b94-4d83-bfff-1a242578d2d2",
        "historySizeBytes": "6170",
        "workerVersion": {
          "buildId": "5e4efe030bb177ed5afa5533417de3d6"
        }
      }
    },
    {
      "eventId": "10",
      "eventTime": "2024-10-23T08:26:49.083443800Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
      "taskId": "20971799",
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "8",
        "startedEventId": "9",
        "identity": "25240@JYM-HpZhanX@",
        "workerVersion": {
          "buildId": "5e4efe030bb177ed5afa5533417de3d6"
        },
        "sdkMetadata": {},
        "meteringMetadata": {}
      }
    },
    {
      "eventId": "11",
      "eventTime": "2024-10-23T08:26:49.083443800Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
      "taskId": "20971800",
      "activityTaskScheduledEventAttributes": {
        "activityId": "11",
        "activityType": {
          "name": "UpdateInfoActivity"
        },
        "taskQueue": {
          "name": "order_matching_task_queue",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "header": {},
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "eyJBcHBseSI6bnVsbCwiU2VuZCI6eyJJZCI6ImNzY2I2ZWhyZTlmNjBhMGN2Y2ZnIiwiQ3JlYXRlZEF0IjoiMDAwMS0wMS0wMVQwMDowMDowMFoiLCJVcGRhdGVkQXQiOiIwMDAxLTAxLTAxVDAwOjAwOjAwWiIsIkRlbGV0ZWRBdCI6bnVsbCwiVmVyc2lvbiI6bnVsbCwib3JkZXJBcHBseUlkIjoiIiwicHJpY2UiOiIwIiwib3JpZ2luYWxQcmljZSI6IjAiLCJzdGF0dXMiOiIiLCJqb2JJZCI6IiIsImlzUGVuYWxpemVkIjowLCJFbXBsb3llZUlkIjoiIiwiU3RhcnRUaW1lIjpudWxsLCJFbmRUaW1lIjpudWxsLCJDYW5jZWxlZEF0IjpudWxsLCJDYW5jZWxSZWFzb24iOiIiLCJJc0V2YWx1YXRlZCI6MCwiU2VydmljZVByaWNlIjoiMCIsIldvcmtmbG93SWQiOiIiLCJSdW5JZCI6IiIsIkNoaWxkV29ya2Zsb3dJZCI6Im1hdGNoLXdhaXRpbmctd29ya2Zsb3ctdGVzdC1hcHBseS1pZC1jc2NiNmVocmU5ZjYwYTBjdmNmZyIsIkNoaWxkUnVuSWQiOiJiMmI4NzBhMy0zM2JiLTQ4N2UtYWJmMi0yNjIwNDA2OTYxOGUiLCJFc3RpbWF0ZWREZXBhcnRUaW1lIjpudWxsLCJFc3RpbWF0ZWRBcnJpdmVUaW1lIjpudWxsLCJNYXRjaExpbWl0VGltZSI6bnVsbCwiT3JkZXJBcHBseSI6bnVsbCwiT3JkZXJSZWNlaXZlIjp7IklkIjoiIiwiQ3JlYXRlZEF0IjoiMDAwMS0wMS0wMVQwMDowMDowMFoiLCJVcGRhdGVkQXQiOiIwMDAxLTAxLTAxVDAwOjAwOjAwWiIsIkRlbGV0ZWRBdCI6bnVsbCwiVmVyc2lvbiI6bnVsbCwiT3JkZXJTZW5kSWQiOiIiLCJPcmRlckFwcGx5SWQiOiIiLCJSZWNlaXZlVXNlcklkIjoiIiwiUHJpY2UiOiIwIiwiU3RhdHVzIjoiIiwiRW1wbG95ZWVJZCI6IiIsIkFycml2ZWRUaW1lIjpudWxsLCJJc1BlbmFsaXplZCI6MCwiUGVuYWxpemVkUmVhc29uIjoiIiwiUGVuYWxpemVkUmVzdWx0IjoiIiwiUGVuYWxpemVkVGltZSI6bnVsbCwiSXNGYWNlVmFsaWQiOjAsIkNhbkNhbmNlbFRpbWUiOiIwMDAxLTAxLTAxVDAwOjAwOjAwWiIsIkNhbmNlbFJlYXNvbiI6IiIsIkNhbmNlbGVkQXQiOm51bGwsIk9yZGVyVHlwZSI6IiIsIlJlY2VpdmVVc2VyR3BzIjpudWxsLCJSZWNlaXZlVXNlckFkZHIiOiIiLCJJc0dyYWJiaW5nIjowLCJGaW5pc2hUaW1lIjpudWxsLCJJc0V2YWx1YXRlZCI6MCwiQ3RsU3RhdHVzIjoiIiwiUmVjZWl2ZVVzZXIiOm51bGwsIkVtcGxveWVlIjpudWxsLCJPcmRlclNlbmQiOm51bGwsIk9yZGVyQXBwbHkiOm51bGx9LCJPcmRlclJld2FyZCI6eyJJZCI6IiIsIkNyZWF0ZWRBdCI6IjAwMDEtMDEtMDFUMDA6MDA6MDBaIiwiVXBkYXRlZEF0IjoiMDAwMS0wMS0wMVQwMDowMDowMFoiLCJEZWxldGVkQXQiOm51bGwsIlZlcnNpb24iOm51bGwsIk9yZGVyUmVjZWl2ZUlkIjoiIiwiT3JkZXJTZW5kSWQiOiIiLCJPcmRlckFwcGx5SWQiOiIiLCJQcmljZSI6IjAiLCJQYXlUaW1lIjpudWxsLCJDbG9zZVRpbWUiOm51bGwsIlN0YXR1cyI6IiIsIlRyYW5zYWN0aW9uSWQiOiIiLCJQYXlUeXBlIjoiIiwiVHJhZGVObyI6IiIsIk9yZGVyUmVjZWl2ZSI6bnVsbCwiT3JkZXJTZW5kIjpudWxsLCJPcmRlckFwcGx5IjpudWxsfSwiSm9iIjpudWxsfSwiUmVjZWl2ZSI6bnVsbCwiVXNlckdlbyI6bnVsbCwiSm9iIjpudWxsLCJKb2JFeHQiOm51bGx9"
            }
          ]
        },
        "scheduleToCloseTimeout": "60s",
        "scheduleToStartTimeout": "60s",
        "startToCloseTimeout": "60s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "10",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s"
        },
        "useWorkflowBuildId": true
      }
    },
    {
      "eventId": "12",
      "eventTime": "2024-10-23T08:26:49.090362Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
      "taskId": "20971806",
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "11",
        "identity": "25240@JYM-HpZhanX@",
        "requestId": "fa12df5c-0633-465d-b127-d95c72113694",
        "attempt": 1,
        "workerVersion": {
          "buildId": "5e4efe030bb177ed5afa5533417de3d6"
        }
      }
    },
    {
      "eventId": "13",
      "eventTime": "2024-10-23T08:26:49.162297600Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
      "taskId": "20971807",
      "activityTaskCompletedEventAttributes": {
        "scheduledEventId": "11",
        "startedEventId": "12",
        "identity": "25240@JYM-HpZhanX@"
      }
    },
    {
      "eventId": "14",
      "eventTime": "2024-10-23T08:26:49.162297600Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
      "taskId": "20971808",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "JYM-HpZhanX:52e8df8d-4cb1-497e-b337-503cf76f8e9f",
          "kind": "TASK_QUEUE_KIND_STICKY",
          "normalName": "order_matching_task_queue"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "15",
      "eventTime": "2024-10-23T08:26:49.169851500Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
      "taskId": "20971812",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "14",
        "identity": "25240@JYM-HpZhanX@",
        "requestId": "8a00a7c9-41b1-403a-9b8e-f34d054ae913",
        "historySizeBytes": "8448",
        "workerVersion": {
          "buildId": "5e4efe030bb177ed5afa5533417de3d6"
        }
      }
    },
    {
      "eventId": "16",
      "eventTime": "2024-10-23T08:26:49.179021100Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
      "taskId": "20971816",
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "14",
        "startedEventId": "15",
        "identity": "25240@JYM-HpZhanX@",
        "workerVersion": {
          "buildId": "5e4efe030bb177ed5afa5533417de3d6"
        },
        "sdkMetadata": {},
        "meteringMetadata": {}
      }
    },
    {
      "eventId": "17",
      "eventTime": "2024-10-23T08:26:49.179021100Z",
      "eventType": "EVENT_TYPE_TIMER_STARTED",
      "taskId": "20971817",
      "timerStartedEventAttributes": {
        "timerId": "17",
        "startToFireTimeout": "117.531284900s",
        "workflowTaskCompletedEventId": "16"
      }
    },
    {
      "eventId": "18",
      "eventTime": "2024-10-23T08:28:46.726059300Z",
      "eventType": "EVENT_TYPE_TIMER_FIRED",
      "taskId": "20971820",
      "timerFiredEventAttributes": {
        "timerId": "17",
        "startedEventId": "17"
      }
    },
    {
      "eventId": "19",
      "eventTime": "2024-10-23T08:28:46.733078800Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
      "taskId": "20971821",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "JYM-HpZhanX:52e8df8d-4cb1-497e-b337-503cf76f8e9f",
          "kind": "TASK_QUEUE_KIND_STICKY",
          "normalName": "order_matching_task_queue"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "20",
      "eventTime": "2024-10-23T08:28:46.740082100Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
      "taskId": "20971825",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "19",
        "identity": "16828@JYM-HpZhanX@",
        "requestId": "90295bd6-a89f-47fa-9a90-062b8651b743",
        "historySizeBytes": "8854",
        "workerVersion": {
          "buildId": "5e4efe030bb177ed5afa5533417de3d6"
        }
      }
    },
    {
      "eventId": "21",
      "eventTime": "2024-10-23T08:28:46.757679100Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_FAILED",
      "taskId": "20971829",
      "workflowTaskFailedEventAttributes": {
        "scheduledEventId": "19",
        "startedEventId": "20",
        "cause": "WORKFLOW_TASK_FAILED_CAUSE_NON_DETERMINISTIC_ERROR",
        "failure": {
          "message": "[TMPRL1100] unknown command CommandType: Timer, ID: 17, possible causes are nondeterministic workflow definition code or incompatible change in the workflow definition",
          "source": "GoSDK",
          "stackTrace": "process event for order_matching_task_queue [panic]:\ngo.temporal.io/sdk/internal.panicIllegalState(...)\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_command_state_machine.go:440\ngo.temporal.io/sdk/internal.(*commandsHelper).getCommand(0xc00658e080, {0x4, {0xc006655728, 0x2}})\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_command_state_machine.go:929 +0x3a5\ngo.temporal.io/sdk/internal.(*commandsHelper).handleTimerStarted(0x1f08f841348?, {0xc006655728?, 0xc000601008?})\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_command_state_machine.go:1385 +0x25\ngo.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc006588108, 0xc0066aae70, 0xa0?, 0x0)\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_event_handlers.go:1175 +0x3f1\ngo.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc006598000, 0xc0066aced0)\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_task_handlers.go:1100 +0x1608\ngo.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc0042f0000, 0xc0066aced0, 0xc006598000, 0xc00658a5a0)\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_task_handlers.go:889 +0x3c5\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc00391c240, 0xc0066aced0)\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_task_pollers.go:357 +0x3c3\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc00391c240, {0x1adcba0, 0xc0066aced0})\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_task_pollers.go:321 +0x78\ngo.temporal.io/sdk/internal.(*baseWorker).processTask(0xc0043ba160, {0x1add6e0, 0xc0066a8780})\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_worker_base.go:518 +0x153\ngo.temporal.io/sdk/internal.(*baseWorker).processTaskAsync.func1()\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_worker_base.go:369 +0x45\ncreated by go.temporal.io/sdk/internal.(*baseWorker).processTaskAsync in goroutine 71\n\tE:/Project/GolangProject/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_worker_base.go:365 +0xa5",
          "applicationFailureInfo": {
            "type": "PanicError",
            "nonRetryable": true
          }
        },
        "identity": "16828@JYM-HpZhanX@"
      }
    }
  ]
}
4

History is not helpful. Have you checked the workflow code for determinism?