Timer Started But Never Fired - Stuck 24+ Hours

Community Support
,
1

Problem

A 2-second timer started at 2026-01-31T12:34:20Z (event 23) but never fired. Workflow has been stuck for over 24 hours with no TIMER_FIRED event.

Environment

  • Workflow Type: ChildWorkflow

  • SDK: temporal-go v1.29.1

  • Server: v1.22

Event Sequence

  1. Events 1-19: Workflow started, executed PreChildWorkflowActivityV1, LisaRedirectUserV3, and MapperV2 activities successfully

  2. Events 20-22: Workflow task scheduled/started/completed

  3. Event 23: Timer started with 2s timeout - NEVER FIRED

image
image1488×386 54.5 KB

Full Event History

{
  "events": [
    {
      "eventId": "1",
      "eventTime": "2026-01-31T12:34:19.760239243Z",
      "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_STARTED",
      "taskId": "2555691697",
      "workflowExecutionStartedEventAttributes": {
        "workflowType": {
          "name": "ChildWorkflowV1"
        },
        "parentWorkflowNamespace": "lending_prod",
        "parentWorkflowNamespaceId": "5bdf3a14-7b70-49c6-8a17-9c431279e1dc",
        "parentWorkflowExecution": {
          "workflowId": "ABCD_OFFER_SELECTION_79e8834d-d48d-4f17-b7d9-a5bea2152c6d",
          "runId": "d218b560-17ef-451d-a1b8-16db1d0ee53f"
        },
        "parentInitiatedEventId": "327",
        "taskQueue": {
          "name": "abcd_builder_queue_v1",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg==",
                "encodingDecoded": "json/plain"
              },
              "data": {
                "workflowId": "PREFR_REDIRECT",
                "invoke": "sync",
                "onParentComplete": "terminate"
              }
            },
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg==",
                "encodingDecoded": "json/plain"
              },
              "data": {}
            }
          ]
        },
        "workflowExecutionTimeout": "0s",
        "workflowRunTimeout": "0s",
        "workflowTaskTimeout": "10s",
        "originalExecutionRunId": "538fcfc5-e96f-4613-a33a-bdece098c4cb",
        "firstExecutionRunId": "538fcfc5-e96f-4613-a33a-bdece098c4cb",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 1,
          "maximumInterval": "100s",
          "maximumAttempts": 1
        },
        "attempt": 1,
        "workflowId": "d218b560-17ef-451d-a1b8-16db1d0ee53f_327"
      }
    },
    {
      "eventId": "2",
      "eventTime": "2026-01-31T12:34:19.803074667Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
      "taskId": "2555691701",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "abcd_builder_queue_v1",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "3",
      "eventTime": "2026-01-31T12:34:19.825242365Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
      "taskId": "2555691704",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "2",
        "identity": "prod_6022f7b1-4163-4301-8bf7-1791ef670b20",
        "requestId": "d7b39605-a63d-44eb-8c67-ff49c93d7ab6",
        "historySizeBytes": "2182"
      }
    },
    {
      "eventId": "4",
      "eventTime": "2026-01-31T12:34:19.850707707Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
      "taskId": "2555691708",
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "2",
        "startedEventId": "3",
        "identity": "prod_6022f7b1-4163-4301-8bf7-1791ef670b20",
        "workerVersion": {
          "buildId": "e18889eb0c18e7383bc75dd50965ab46"
        },
        "sdkMetadata": {
          "langUsedFlags": [3],
          "sdkName": "temporal-go",
          "sdkVersion": "1.29.1"
        }
      }
    },
    {
      "eventId": "5",
      "eventTime": "2026-01-31T12:34:19.850759424Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
      "taskId": "2555691709",
      "activityTaskScheduledEventAttributes": {
        "activityId": "5",
        "activityType": {
          "name": "PreChildWorkflowActivityV1"
        },
        "taskQueue": {
          "name": "abcd_builder_queue_v1",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "600s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "4",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 1,
          "maximumInterval": "30s",
          "maximumAttempts": 3
        },
        "useWorkflowBuildId": true
      }
    },
    {
      "eventId": "6",
      "eventTime": "2026-01-31T12:34:19.874119281Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
      "taskId": "2555691715",
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "5",
        "identity": "prod_7bef6633-e772-43aa-b2c0-a0fe94a32a6a",
        "requestId": "b480acba-4112-42d4-813f-054cb981dc5c",
        "attempt": 1
      }
    },
    {
      "eventId": "7",
      "eventTime": "2026-01-31T12:34:19.921660777Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
      "taskId": "2555691716",
      "activityTaskCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "[LARGE_WORKFLOW_DEFINITION_REDACTED]"
            }
          ]
        },
        "scheduledEventId": "5",
        "startedEventId": "6",
        "identity": "prod_7bef6633-e772-43aa-b2c0-a0fe94a32a6a"
      }
    },
    {
      "eventId": "8",
      "eventTime": "2026-01-31T12:34:19.921667005Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
      "taskId": "2555691717",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "962878e21584:89dcd662-bc80-4d7b-9b01-830ddf3061cd",
          "kind": "TASK_QUEUE_KIND_STICKY",
          "normalName": "abcd_builder_queue_v1"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "9",
      "eventTime": "2026-01-31T12:34:19.948395603Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
      "taskId": "2555691721",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "8",
        "identity": "prod_6022f7b1-4163-4301-8bf7-1791ef670b20",
        "requestId": "fab3ceb7-5491-499a-a4ab-c7c2dea9d645",
        "historySizeBytes": "11394"
      }
    },
    {
      "eventId": "10",
      "eventTime": "2026-01-31T12:34:19.973557643Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
      "taskId": "2555691725",
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "8",
        "startedEventId": "9",
        "identity": "prod_6022f7b1-4163-4301-8bf7-1791ef670b20",
        "workerVersion": {
          "buildId": "e18889eb0c18e7383bc75dd50965ab46"
        }
      }
    },
    {
      "eventId": "11",
      "eventTime": "2026-01-31T12:34:19.973604831Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
      "taskId": "2555691726",
      "activityTaskScheduledEventAttributes": {
        "activityId": "11",
        "activityType": {
          "name": "LisaRedirectUserV3"
        },
        "taskQueue": {
          "name": "abcd_builder_queue_v1",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "600s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "10",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 1,
          "maximumInterval": "30s",
          "maximumAttempts": 3
        },
        "useWorkflowBuildId": true
      }
    },
    {
      "eventId": "12",
      "eventTime": "2026-01-31T12:34:19.996331370Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
      "taskId": "2555691731",
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "11",
        "identity": "prod_3eb29f66-6eec-4b58-b54e-75158452a025",
        "requestId": "7e3f9826-ddcc-41dd-b950-b1412f4b94b6",
        "attempt": 1
      }
    },
    {
      "eventId": "13",
      "eventTime": "2026-01-31T12:34:20.138212305Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
      "taskId": "2555691732",
      "activityTaskCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": {}
            }
          ]
        },
        "scheduledEventId": "11",
        "startedEventId": "12",
        "identity": "prod_3eb29f66-6eec-4b58-b54e-75158452a025"
      }
    },
    {
      "eventId": "14",
      "eventTime": "2026-01-31T12:34:20.138219410Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
      "taskId": "2555691733",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "962878e21584:89dcd662-bc80-4d7b-9b01-830ddf3061cd",
          "kind": "TASK_QUEUE_KIND_STICKY",
          "normalName": "abcd_builder_queue_v1"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "15",
      "eventTime": "2026-01-31T12:34:20.162045169Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
      "taskId": "2555691737",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "14",
        "identity": "prod_6022f7b1-4163-4301-8bf7-1791ef670b20",
        "requestId": "2817a36f-3790-421b-84c3-b1098c5f15a8",
        "historySizeBytes": "14782"
      }
    },
    {
      "eventId": "16",
      "eventTime": "2026-01-31T12:34:20.185241631Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
      "taskId": "2555691741",
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "14",
        "startedEventId": "15",
        "identity": "prod_6022f7b1-4163-4301-8bf7-1791ef670b20",
        "workerVersion": {
          "buildId": "e18889eb0c18e7383bc75dd50965ab46"
        }
      }
    },
    {
      "eventId": "17",
      "eventTime": "2026-01-31T12:34:20.185289007Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
      "taskId": "2555691742",
      "activityTaskScheduledEventAttributes": {
        "activityId": "17",
        "activityType": {
          "name": "MapperV2"
        },
        "taskQueue": {
          "name": "abcd_builder_queue_v1",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "600s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "16",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 1,
          "maximumInterval": "100s",
          "maximumAttempts": 1
        },
        "useWorkflowBuildId": true
      }
    },
    {
      "eventId": "18",
      "eventTime": "2026-01-31T12:34:20.213799021Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
      "taskId": "2555691747",
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "17",
        "identity": "prod_f94382eb-f3b0-4906-a1b0-b0745f5ce9aa",
        "requestId": "9aa82457-5fe0-4b04-896d-a939f462f21c",
        "attempt": 1
      }
    },
    {
      "eventId": "19",
      "eventTime": "2026-01-31T12:34:20.231488987Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
      "taskId": "2555691748",
      "activityTaskCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": {}
            }
          ]
        },
        "scheduledEventId": "17",
        "startedEventId": "18",
        "identity": "prod_f94382eb-f3b0-4906-a1b0-b0745f5ce9aa"
      }
    },
    {
      "eventId": "20",
      "eventTime": "2026-01-31T12:34:20.231495379Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
      "taskId": "2555691749",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "962878e21584:89dcd662-bc80-4d7b-9b01-830ddf3061cd",
          "kind": "TASK_QUEUE_KIND_STICKY",
          "normalName": "abcd_builder_queue_v1"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "21",
      "eventTime": "2026-01-31T12:34:20.255255965Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
      "taskId": "2555691753",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "20",
        "identity": "prod_6022f7b1-4163-4301-8bf7-1791ef670b20",
        "requestId": "fc752f4b-669b-4148-b38b-ee68dd868444",
        "historySizeBytes": "18557"
      }
    },
    {
      "eventId": "22",
      "eventTime": "2026-01-31T12:34:20.276099320Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
      "taskId": "2555691757",
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "20",
        "startedEventId": "21",
        "identity": "prod_6022f7b1-4163-4301-8bf7-1791ef670b20",
        "workerVersion": {
          "buildId": "e18889eb0c18e7383bc75dd50965ab46"
        }
      }
    },
    {
      "eventId": "23",
      "eventTime": "2026-01-31T12:34:20.276125371Z",
      "eventType": "EVENT_TYPE_TIMER_STARTED",
      "taskId": "2555691758",
      "timerStartedEventAttributes": {
        "timerId": "23",
        "startToFireTimeout": "2s",
        "workflowTaskCompletedEventId": "22"
      }
    }
  ]
}
2

Any findings on this thread?

3

Following up this thread. Is there something we are missing?

5

sounds like timer queue on specific shard is really backed up or data corruption happened.

Is this execution still in running state? did timer eventually fire?
If is still running and timer never fired, can you try refresh its tasks

via old tctl:

tctl adm wf rt -h

you can get namespace id via
tctl --namespace <ns_name> n desc

via tdbg ( temporal/tools/tdbg at main · temporalio/temporal · GitHub )
tdbg -n <namespace_name> w rt -wid <workflow_id>

if this does not unblock it,

try describing this execution
tdbg w d --wid <workflow_id>

this should give you if works, the history host and shard id of this execution
try bouncing the shard:

tdbg s close-shard --shard-id <shard_id>

if still stuck there is some steps we might need to do to manually repair db and then bounce shard. let me know if things so far help unblock execution or not

it would also help if we look at metrics
p100 task latency around this time you see timer started event in your json history
histogram_quantile(1, sum by (le, operation) (rate(task_latency_bucket[1m])))

server also emits task_latency_queueand task_errorsif you can look big before and after timestamp of timer started event

would also help to look at persistence errors around that time
sum(rate(persistence_error_with_type[1m])) by (operation)

server v1.22

this is pretty old version. 1.22.0 was released 2023. whats preventing server version upgrades on your end?