ActivityTaskScheduled but never invoked

Community Support
1

I deployed a new worker and one of our scheduled workflows seems to be stuck not executing an activity. Subsequent scheduled workflows after the update are running fine.

Workflow change I did reorder activities that get invoked inside the workflow. So maybe this is a workflow history compatibility thing? But based on the temporal describe output below I don’t have any context for why this workflow is stuck.

Workflow described

temporal workflow describe --workflow-id=load-borrow-blorp-workflow-2023-07-28T16:30:00Z
{
  "executionConfig": {
    "taskQueue": {
      "name": "BLORP_FIRMS_TASK_QUEUE",
      "kind": "Normal"
    },
    "workflowExecutionTimeout": "0s",
    "workflowRunTimeout": "0s",
    "defaultWorkflowTaskTimeout": "10s"
  },
  "workflowExecutionInfo": {
    "execution": {
      "workflowId": "load-borrow-blorp-workflow-2023-07-28T16:30:00Z",
      "runId": "9fd5f8ff-8804-4ede-8608-536889330e1a"
    },
    "type": {
      "name": "LoadBorrowWorkflow"
    },
    "startTime": "2023-07-28T16:30:00.039809459Z",
    "status": "Running",
    "historyLength": "11",
    "executionTime": "2023-07-28T16:30:00.039809459Z",
    "memo": {

    },
    "searchAttributes": {
      "indexedFields": {
        "BuildIds": "[\"unversioned\",\"unversioned:7eb99bc1ab0ae615d2583ff5b33c72e8\"]",
        "TemporalScheduledById": "load-borrow-blorp-schedule",
        "TemporalScheduledStartTime": "2023-07-28T16:30:00Z"
      }
    },
    "autoResetPoints": {
      "points": [
        {
          "binaryChecksum": "7eb99bc1ab0ae615d2583ff5b33c72e8",
          "runId": "9fd5f8ff-8804-4ede-8608-536889330e1a",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2023-07-28T16:32:37.882844996Z",
          "resettable": true
        }
      ]
    },
    "stateTransitionCount": "8",
    "historySizeBytes": "1633",
    "mostRecentWorkerVersionStamp": {
      "buildId": "7eb99bc1ab0ae615d2583ff5b33c72e8"
    }
  },
  "pendingActivities": [
    {
      "activityId": "11",
      "activityType": {
        "name": "GetFirmConfigActivity"
      },
      "state": "Started",
      "lastStartedTime": "2023-07-28T16:32:37.943682038Z",
      "attempt": 1,
      "expirationTime": "0001-01-01T00:00:00Z",
      "lastWorkerIdentity": "1@f50090d0e30e@"
    }
  ]
}

Activity timeouts

	options := workflow.ActivityOptions{
		StartToCloseTimeout:    time.Minute * 60 * 12,
		ScheduleToStartTimeout: time.Minute * 60 * 1,
	}

I’d love guidance on best practices when updating workers after changes like this.

2

If you change the order of activities, the workflow will get stuck. Look at the workflow execution history in UI/CLI,. The last WorkflowTaskFailed event should contain more information.

3

The workflow finally failed and had more information and yes it seems the order caused the wrong activity result to go into the wrong activity. Which makes sense I guess. Pasted the error below in case it helps others.

{
  "message": "error getting firm operation config: payload item 0: unable to decode: json: cannot unmarshal bool into Go value of type temporal.FirmOperationConfig",
  "source": "GoSDK",
  "stackTrace": "",
  "encodedAttributes": null,
  "cause": {
    "message": "payload item 0: unable to decode: json: cannot unmarshal bool into Go value of type temporal.FirmOperationConfig",
    "source": "GoSDK",
    "stackTrace": "",
    "encodedAttributes": null,
    "cause": {
      "message": "unable to decode: json: cannot unmarshal bool into Go value of type temporal.FirmOperationConfig",
      "source": "GoSDK",
      "stackTrace": "",
      "encodedAttributes": null,
      "cause": {
        "message": "unable to decode",
        "source": "GoSDK",
        "stackTrace": "",
        "encodedAttributes": null,
        "cause": null,
        "applicationFailureInfo": {
          "type": "",
          "nonRetryable": false,
          "details": null
        }
      },
      "applicationFailureInfo": {
        "type": "wrapError",
        "nonRetryable": false,
        "details": null
      }
    },
    "applicationFailureInfo": {
      "type": "wrapError",
      "nonRetryable": false,
      "details": null
    }
  },
  "applicationFailureInfo": {
    "type": "wrapError",
    "nonRetryable": false,
    "details": null
  }
}