Stuck workflows

Community Support
1

It looks like the activity is scheduled to be picked up by a worker, but 4 days have now elapsed, and no worker has pulled the activity to work on it. We did a Temporal container upgrade during the time this is running, would this be a contributing factor?

Is there some way to intervene so that we may force a worker to pick up the activity?

Here is a json with the workflow with just the temporal portions:

{
  "events": [
    {
      "eventId": "1",
      "eventTime": "2024-03-04T20:05:36.000Z",
      "eventType": "WorkflowExecutionStarted",
      "version": "0",
      "taskId": "132121756",
      "workflowExecutionStartedEventAttributes": {
        "workflowType": {
          "name": "sp-workflow"
        },
        "parentWorkflowNamespace": "",
        "parentInitiatedEventId": "0",
        "taskQueue": {
          "name": "sp-workflow-engine",
          "kind": "Normal"
        },
        "input": "",
        "firstExecutionRunId": "4c92ac87-a547-43cc-b4a7-c495b8d9bbbb",
        "attempt": 1,
        "workflowExecutionExpirationTime": "2025-03-04T20:05:36.000Z",
        "cronSchedule": "",
        "firstWorkflowTaskBackoff": "0s",
        "memo": "",
        "tenantId": "",
        "header": {
          "fields": {}
        }
      }
    },
    {
      "eventId": "2",
      "eventTime": "2024-03-04T20:05:36.000Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "132121757",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "sp-workflow-engine",
          "kind": "Normal"
        },
        "startToCloseTimeout": "20s",
        "attempt": 1
      }
    },
    {
      "eventId": "3",
      "eventTime": "2024-03-04T20:06:06.000Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "132121763",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "2",
        "identity": "4c4d2372-f653-4fe0-bca4-8c652e2f0f20",
        "requestId": "0d2fa3fc-1996-4757-9185-38aed5824a35"
      }
    },
    {
      "eventId": "4",
      "eventTime": "2024-03-04T20:06:06.000Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "132121767",
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "2",
        "startedEventId": "3",
        "identity": "4c4d2372-f653-4fe0-bca4-8c652e2f0f20",
        "binaryChecksum": "c4db573f700a4b0b2601bd9c1acf3952"
      }
    },
    {
      "eventId": "5",
      "eventTime": "2024-03-04T20:06:06.000Z",
      "eventType": "ActivityTaskScheduled",
      "version": "0",
      "taskId": "132121768",
      "activityTaskScheduledEventAttributes": {
        "activityId": "5",
        "activityType": {
          "name": "sp:internal:operator"
        },
        "namespace": "",
        "taskQueue": {
          "name": "sp-workflow-engine",
          "kind": "Normal"
        },
        "header": {
          "fields": {}
        },
        "scheduleToCloseTimeout": "31536000s",
        "scheduleToStartTimeout": "31536000s",
        "startToCloseTimeout": "300s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "4",
        "retryPolicy": {
          "nonRetryableErrorTypes": [],
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "120s",
          "maximumAttempts": 11
        }
      }
    }
  ]
}
3

Can you describe this workflow execution either via DescribeWorkflowExecution api or cli
temporal workflow describe -w $wfid
or
tctl wf desc -w $wfid

another thing you could try is to refresh task (note only do this in case of these types of incidents, not on regular basis):
tctl adm wf rt -w $wfid

if that does not do it you could reset this workflow execution to events 2 or 3 causing the activity task to be rescheduled in the new execution reset creates (note this running one will get terminated by reset and it will create a new one with same wfid.