Scheduling of activity gets stuck for about 10 minutes before starting

Community Support
1

Hi,

I’m trying to implement a workflow that has about 9 activity steps (53+ events).

Every second time I restart my workflow, it gets stuck in the same step for about 10 minutes and then continues execution.
Canceling the workflow and triggering its execution again does schedule the activity nearly immediatly.
My problem is that stuck activity scheduling which takes too long.

(cut, account too new to embed images)

In the image you can see the activity that gets stuck.

I have got one single service that implements the workflow and some of the activities that are called in the workflow, especially the one that gets stuck for 10 minutes on every second(ish) workflow execution.

This is how it looks like when the activity is stuck.
(cut, account too new to embed images)

My activity context setup:
The startToClose timeout is set to 10 minutes.

func stuckPackageActivityContext(ctx workflow.Context, startToCloseTime time.Duration, attempts ...int32) workflow.Context {
	var attemptNum int32 = 5
	if len(attempts) > 0 {
		attemptNum = attempts[0]
	}

	// RetryPolicy specifies how to automatically handle retries if an Activity fails.
	retrypolicy := &temporal.RetryPolicy{
		InitialInterval:    time.Second,
		BackoffCoefficient: 2.0,
		MaximumInterval:    100 * time.Second,
		MaximumAttempts:    attemptNum,
	}

	options := workflow.ActivityOptions{
		// Timeout options specify when to automatically timeout Activity functions.
		StartToCloseTimeout: startToCloseTime,
		// Optionally provide a customized RetryPolicy.
		// Temporal retries failed Activities by default.
		RetryPolicy: retrypolicy,
		TaskQueue:   stuckpackageactivity.TASK_QUEUE_NAME,
	}

	// Apply the options.
	return workflow.WithActivityOptions(ctx, options)
}

If you need more information, please ask.

Any suggestions on how to debug this are much appreciated, as I’m somewhat new to temporal, at least in regard to debugging such issues.

Any help appreciated.
Thanks :slight_smile:

2

(cut, account too new to embed images)

Could you paste the json history of this execution here please?

3

Sorry for the late answer.

This is where it gets stuck:

{
  "events": [
    {
      "eventId": "1",
      "eventTime": "2024-02-08T11:44:07.134368909Z",
      "eventType": "WorkflowExecutionStarted",
      "version": "0",
      "taskId": "9437184",
      "workerMayIgnore": false,
      "workflowExecutionStartedEventAttributes": {
        "workflowType": {
          "name": "CesarProvision"
        },
        "parentWorkflowNamespace": "",
        "parentWorkflowNamespaceId": "",
        "parentWorkflowExecution": null,
        "parentInitiatedEventId": "0",
        "taskQueue": {
          "name": "CESAR_BRIDGE_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            },
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "workflowExecutionTimeout": "0s",
        "workflowRunTimeout": "0s",
        "workflowTaskTimeout": "10s",
        "continuedExecutionRunId": "",
        "initiator": "Unspecified",
        "continuedFailure": null,
        "lastCompletionResult": null,
        "originalExecutionRunId": "c56f0a62-f378-4db9-966b-cefd661a35c5",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "firstExecutionRunId": "c56f0a62-f378-4db9-966b-cefd661a35c5",
        "retryPolicy": null,
        "attempt": 1,
        "workflowExecutionExpirationTime": null,
        "cronSchedule": "",
        "firstWorkflowTaskBackoff": "0s",
        "memo": null,
        "searchAttributes": null,
        "prevAutoResetPoints": null,
        "header": {
          "fields": {}
        },
        "parentInitiatedEventVersion": "0",
        "workflowId": "cesar-provision-57edfabf-5a93-4606-bf75-3b455ea63cc3",
        "sourceVersionStamp": null
      }
    },
    {
      "eventId": "2",
      "eventTime": "2024-02-08T11:44:07.134447458Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "9437185",
      "workerMayIgnore": false,
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "CESAR_BRIDGE_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "3",
      "eventTime": "2024-02-08T11:44:07.198764446Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "9437190",
      "workerMayIgnore": false,
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "2",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "e745845d-0c2c-4bba-a0c3-59db17a9bea6",
        "suggestContinueAsNew": false,
        "historySizeBytes": "774"
      }
    },
    {
      "eventId": "4",
      "eventTime": "2024-02-08T11:44:07.234633781Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "9437194",
      "workerMayIgnore": false,
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "2",
        "startedEventId": "3",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "binaryChecksum": "",
        "workerVersion": {
          "buildId": "e35c66928e1642979bc05783460d8c66",
          "bundleId": "",
          "useVersioning": false
        },
        "sdkMetadata": {
          "coreUsedFlags": [],
          "langUsedFlags": [
            3
          ]
        },
        "meteringMetadata": {
          "nonfirstLocalActivityExecutionAttempts": 0
        }
      }
    },
    {
      "eventId": "5",
      "eventTime": "2024-02-08T11:44:07.234735824Z",
      "eventType": "ActivityTaskScheduled",
      "version": "0",
      "taskId": "9437195",
      "workerMayIgnore": false,
      "activityTaskScheduledEventAttributes": {
        "activityId": "5",
        "activityType": {
          "name": "CatalogGetAgentOsByHostGroup"
        },
        "taskQueue": {
          "name": "CATALOG_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "header": {
          "fields": {}
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            },
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            },
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            },
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "600s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "4",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s",
          "maximumAttempts": 5,
          "nonRetryableErrorTypes": []
        },
        "useCompatibleVersion": false
      }
    },
    {
      "eventId": "6",
      "eventTime": "2024-02-08T11:44:07.268010445Z",
      "eventType": "ActivityTaskStarted",
      "version": "0",
      "taskId": "9437201",
      "workerMayIgnore": false,
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "5",
        "identity": "1@catalog-service-5588d8f97-ct5cw@",
        "requestId": "c2ffda3c-a28a-4552-9e98-70ef0c2d5e93",
        "attempt": 1,
        "lastFailure": null
      }
    },
    {
      "eventId": "7",
      "eventTime": "2024-02-08T11:44:07.303140426Z",
      "eventType": "ActivityTaskCompleted",
      "version": "0",
      "taskId": "9437202",
      "workerMayIgnore": false,
      "activityTaskCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduledEventId": "5",
        "startedEventId": "6",
        "identity": "1@catalog-service-5588d8f97-ct5cw@",
        "workerVersion": null
      }
    },
    {
      "eventId": "8",
      "eventTime": "2024-02-08T11:44:07.303155547Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "9437203",
      "workerMayIgnore": false,
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "cesar-bridge-79898fbd9d-stwmc:3eb2e5d2-2044-4f2e-8ad7-99a4bcad2492",
          "kind": "Sticky",
          "normalName": "CESAR_BRIDGE_TASK_QUEUE"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "9",
      "eventTime": "2024-02-08T11:44:07.335497194Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "9437207",
      "workerMayIgnore": false,
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "8",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "6d2b2762-646c-4ce8-88ac-14e0492d4621",
        "suggestContinueAsNew": false,
        "historySizeBytes": "2854"
      }
    },
    {
      "eventId": "10",
      "eventTime": "2024-02-08T11:44:07.368982774Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "9437211",
      "workerMayIgnore": false,
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "8",
        "startedEventId": "9",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "binaryChecksum": "",
        "workerVersion": {
          "buildId": "e35c66928e1642979bc05783460d8c66",
          "bundleId": "",
          "useVersioning": false
        },
        "sdkMetadata": {
          "coreUsedFlags": [],
          "langUsedFlags": []
        },
        "meteringMetadata": {
          "nonfirstLocalActivityExecutionAttempts": 0
        }
      }
    },
    {
      "eventId": "11",
      "eventTime": "2024-02-08T11:44:07.369104685Z",
      "eventType": "ActivityTaskScheduled",
      "version": "0",
      "taskId": "9437212",
      "workerMayIgnore": false,
      "activityTaskScheduledEventAttributes": {
        "activityId": "11",
        "activityType": {
          "name": "CesarCatalogGetInventory"
        },
        "taskQueue": {
          "name": "CESAR_BRIDGE_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "header": {
          "fields": {}
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "600s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "10",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s",
          "maximumAttempts": 5,
          "nonRetryableErrorTypes": []
        },
        "useCompatibleVersion": true
      }
    },
    {
      "eventId": "12",
      "eventTime": "2024-02-08T11:44:07.403741273Z",
      "eventType": "ActivityTaskStarted",
      "version": "0",
      "taskId": "9437217",
      "workerMayIgnore": false,
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "11",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "79627dfa-027b-4a3d-85d6-407022f6d615",
        "attempt": 1,
        "lastFailure": null
      }
    },
    {
      "eventId": "13",
      "eventTime": "2024-02-08T11:44:07.501870011Z",
      "eventType": "ActivityTaskCompleted",
      "version": "0",
      "taskId": "9437218",
      "workerMayIgnore": false,
      "activityTaskCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduledEventId": "11",
        "startedEventId": "12",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "workerVersion": null
      }
    },
    {
      "eventId": "14",
      "eventTime": "2024-02-08T11:44:07.501884264Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "9437219",
      "workerMayIgnore": false,
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "cesar-bridge-79898fbd9d-stwmc:3eb2e5d2-2044-4f2e-8ad7-99a4bcad2492",
          "kind": "Sticky",
          "normalName": "CESAR_BRIDGE_TASK_QUEUE"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "15",
      "eventTime": "2024-02-08T11:44:07.536775236Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "9437223",
      "workerMayIgnore": false,
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "14",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "656c4cf6-f322-4e41-9db2-37a6a71420ac",
        "suggestContinueAsNew": false,
        "historySizeBytes": "3847"
      }
    },
    {
      "eventId": "16",
      "eventTime": "2024-02-08T11:44:07.569910973Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "9437227",
      "workerMayIgnore": false,
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "14",
        "startedEventId": "15",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "binaryChecksum": "",
        "workerVersion": {
          "buildId": "e35c66928e1642979bc05783460d8c66",
          "bundleId": "",
          "useVersioning": false
        },
        "sdkMetadata": {
          "coreUsedFlags": [],
          "langUsedFlags": []
        },
        "meteringMetadata": {
          "nonfirstLocalActivityExecutionAttempts": 0
        }
      }
    },
    {
      "eventId": "17",
      "eventTime": "2024-02-08T11:44:07.569986249Z",
      "eventType": "ActivityTaskScheduled",
      "version": "0",
      "taskId": "9437228",
      "workerMayIgnore": false,
      "activityTaskScheduledEventAttributes": {
        "activityId": "17",
        "activityType": {
          "name": "CesarBridgeStageProvisionPackage"
        },
        "taskQueue": {
          "name": "CESAR_BRIDGE_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "header": {
          "fields": {}
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            },
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "600s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "16",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s",
          "maximumAttempts": 5,
          "nonRetryableErrorTypes": []
        },
        "useCompatibleVersion": true
      }
    }
  ]
}
4

And this one is after those ~10 minutes of waiting for the job to get scheduled:

{
  "events": [
    {
      "eventId": "18",
      "eventTime": "2024-02-08T11:54:08.608306004Z",
      "eventType": "ActivityTaskStarted",
      "version": "0",
      "taskId": "9437236",
      "workerMayIgnore": false,
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "17",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "77cedb73-0220-4224-9237-4495a92b8201",
        "attempt": 2,
        "lastFailure": {
          "message": "activity StartToClose timeout",
          "source": "Server",
          "stackTrace": "",
          "encodedAttributes": null,
          "cause": null,
          "timeoutFailureInfo": {
            "timeoutType": "StartToClose",
            "lastHeartbeatDetails": null
          }
        }
      }
    },
    {
      "eventId": "19",
      "eventTime": "2024-02-08T11:54:09.283565026Z",
      "eventType": "ActivityTaskCompleted",
      "version": "0",
      "taskId": "9437237",
      "workerMayIgnore": false,
      "activityTaskCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduledEventId": "17",
        "startedEventId": "18",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "workerVersion": null
      }
    },
    {
      "eventId": "20",
      "eventTime": "2024-02-08T11:54:09.283588225Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "9437238",
      "workerMayIgnore": false,
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "cesar-bridge-79898fbd9d-stwmc:3eb2e5d2-2044-4f2e-8ad7-99a4bcad2492",
          "kind": "Sticky",
          "normalName": "CESAR_BRIDGE_TASK_QUEUE"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "21",
      "eventTime": "2024-02-08T11:54:09.328423127Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "9437242",
      "workerMayIgnore": false,
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "20",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "24401a07-f5c0-4c73-9826-1aecfda82849",
        "suggestContinueAsNew": false,
        "historySizeBytes": "5098"
      }
    },
    {
      "eventId": "22",
      "eventTime": "2024-02-08T11:54:09.376790218Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "9437246",
      "workerMayIgnore": false,
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "20",
        "startedEventId": "21",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "binaryChecksum": "",
        "workerVersion": {
          "buildId": "e35c66928e1642979bc05783460d8c66",
          "bundleId": "",
          "useVersioning": false
        },
        "sdkMetadata": {
          "coreUsedFlags": [],
          "langUsedFlags": []
        },
        "meteringMetadata": {
          "nonfirstLocalActivityExecutionAttempts": 0
        }
      }
    },
    {
      "eventId": "23",
      "eventTime": "2024-02-08T11:54:09.377098819Z",
      "eventType": "ActivityTaskScheduled",
      "version": "0",
      "taskId": "9437247",
      "workerMayIgnore": false,
      "activityTaskScheduledEventAttributes": {
        "activityId": "23",
        "activityType": {
          "name": "JobCesarProvisionCreate"
        },
        "taskQueue": {
          "name": "JOB_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "header": {
          "fields": {}
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "1800s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "22",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s",
          "maximumAttempts": 5,
          "nonRetryableErrorTypes": []
        },
        "useCompatibleVersion": false
      }
    },
    {
      "eventId": "24",
      "eventTime": "2024-02-08T11:54:09.412822072Z",
      "eventType": "ActivityTaskStarted",
      "version": "0",
      "taskId": "9437252",
      "workerMayIgnore": false,
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "23",
        "identity": "1@job-service-fd6b78779-2klhz@",
        "requestId": "723795dc-7c7a-42d7-869a-f88f6cb9ffdb",
        "attempt": 1,
        "lastFailure": null
      }
    },
    {
      "eventId": "25",
      "eventTime": "2024-02-08T11:54:09.445136836Z",
      "eventType": "ActivityTaskCompleted",
      "version": "0",
      "taskId": "9437253",
      "workerMayIgnore": false,
      "activityTaskCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduledEventId": "23",
        "startedEventId": "24",
        "identity": "1@job-service-fd6b78779-2klhz@",
        "workerVersion": null
      }
    },
    {
      "eventId": "26",
      "eventTime": "2024-02-08T11:54:09.445145179Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "9437254",
      "workerMayIgnore": false,
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "cesar-bridge-79898fbd9d-stwmc:aa654300-ea1b-4edb-b2ac-86f15b45fb52",
          "kind": "Sticky",
          "normalName": "CESAR_BRIDGE_TASK_QUEUE"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "27",
      "eventTime": "2024-02-08T11:54:09.479223866Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "9437258",
      "workerMayIgnore": false,
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "26",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "f8ca40aa-66ff-496f-b7a4-2c3fa85fbb91",
        "suggestContinueAsNew": false,
        "historySizeBytes": "6989"
      }
    },
    {
      "eventId": "28",
      "eventTime": "2024-02-08T11:54:09.516783224Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "9437262",
      "workerMayIgnore": false,
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "26",
        "startedEventId": "27",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "binaryChecksum": "",
        "workerVersion": {
          "buildId": "e35c66928e1642979bc05783460d8c66",
          "bundleId": "",
          "useVersioning": false
        },
        "sdkMetadata": {
          "coreUsedFlags": [],
          "langUsedFlags": []
        },
        "meteringMetadata": {
          "nonfirstLocalActivityExecutionAttempts": 0
        }
      }
    },
    {
      "eventId": "29",
      "eventTime": "2024-02-08T11:54:09.516962002Z",
      "eventType": "ActivityTaskScheduled",
      "version": "0",
      "taskId": "9437263",
      "workerMayIgnore": false,
      "activityTaskScheduledEventAttributes": {
        "activityId": "29",
        "activityType": {
          "name": "CesarBridgeSetProvisionStatus"
        },
        "taskQueue": {
          "name": "CESAR_BRIDGE_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "header": {
          "fields": {}
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "600s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "28",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s",
          "maximumAttempts": 5,
          "nonRetryableErrorTypes": []
        },
        "useCompatibleVersion": true
      }
    },
    {
      "eventId": "30",
      "eventTime": "2024-02-08T11:54:09.554236248Z",
      "eventType": "ActivityTaskStarted",
      "version": "0",
      "taskId": "9437268",
      "workerMayIgnore": false,
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "29",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "098b051a-405e-4125-8e20-5c9d01b628dc",
        "attempt": 1,
        "lastFailure": null
      }
    },
    {
      "eventId": "31",
      "eventTime": "2024-02-08T11:54:09.612973342Z",
      "eventType": "ActivityTaskCompleted",
      "version": "0",
      "taskId": "9437269",
      "workerMayIgnore": false,
      "activityTaskCompletedEventAttributes": {
        "result": null,
        "scheduledEventId": "29",
        "startedEventId": "30",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "workerVersion": null
      }
    },
    {
      "eventId": "32",
      "eventTime": "2024-02-08T11:54:09.612985552Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "9437270",
      "workerMayIgnore": false,
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "cesar-bridge-79898fbd9d-stwmc:aa654300-ea1b-4edb-b2ac-86f15b45fb52",
          "kind": "Sticky",
          "normalName": "CESAR_BRIDGE_TASK_QUEUE"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "33",
      "eventTime": "2024-02-08T11:54:09.651251400Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "9437274",
      "workerMayIgnore": false,
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "32",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "392e2707-4f01-4b66-bc8a-973e8d987cf9",
        "suggestContinueAsNew": false,
        "historySizeBytes": "7870"
      }
    },
    {
      "eventId": "34",
      "eventTime": "2024-02-08T11:54:09.688035502Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "9437278",
      "workerMayIgnore": false,
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "32",
        "startedEventId": "33",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "binaryChecksum": "",
        "workerVersion": {
          "buildId": "e35c66928e1642979bc05783460d8c66",
          "bundleId": "",
          "useVersioning": false
        },
        "sdkMetadata": {
          "coreUsedFlags": [],
          "langUsedFlags": []
        },
        "meteringMetadata": {
          "nonfirstLocalActivityExecutionAttempts": 0
        }
      }
    },
    {
      "eventId": "35",
      "eventTime": "2024-02-08T11:54:09.688278862Z",
      "eventType": "ActivityTaskScheduled",
      "version": "0",
      "taskId": "9437279",
      "workerMayIgnore": false,
      "activityTaskScheduledEventAttributes": {
        "activityId": "35",
        "activityType": {
          "name": "JobAwait"
        },
        "taskQueue": {
          "name": "JOB_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "header": {
          "fields": {}
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "600s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "34",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s",
          "maximumAttempts": 0,
          "nonRetryableErrorTypes": []
        },
        "useCompatibleVersion": false
      }
    },
    {
      "eventId": "36",
      "eventTime": "2024-02-08T11:54:09.713540792Z",
      "eventType": "ActivityTaskStarted",
      "version": "0",
      "taskId": "9437284",
      "workerMayIgnore": false,
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "35",
        "identity": "1@job-service-fd6b78779-2klhz@",
        "requestId": "7eb7071b-9c42-444b-87c2-bb23b16f1099",
        "attempt": 1,
        "lastFailure": null
      }
    },
    {
      "eventId": "37",
      "eventTime": "2024-02-08T11:54:50.926773888Z",
      "eventType": "ActivityTaskCompleted",
      "version": "0",
      "taskId": "9437285",
      "workerMayIgnore": false,
      "activityTaskCompletedEventAttributes": {
        "result": null,
        "scheduledEventId": "35",
        "startedEventId": "36",
        "identity": "1@job-service-fd6b78779-2klhz@",
        "workerVersion": null
      }
    },
    {
      "eventId": "38",
      "eventTime": "2024-02-08T11:54:50.926789479Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "9437286",
      "workerMayIgnore": false,
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "cesar-bridge-79898fbd9d-stwmc:aa654300-ea1b-4edb-b2ac-86f15b45fb52",
          "kind": "Sticky",
          "normalName": "CESAR_BRIDGE_TASK_QUEUE"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "39",
      "eventTime": "2024-02-08T11:54:50.950758287Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "9437290",
      "workerMayIgnore": false,
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "38",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "88c268fc-ee95-4004-ad88-ad9b106d0908",
        "suggestContinueAsNew": false,
        "historySizeBytes": "8613"
      }
    },
    {
      "eventId": "40",
      "eventTime": "2024-02-08T11:54:50.983402378Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "9437294",
      "workerMayIgnore": false,
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "38",
        "startedEventId": "39",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "binaryChecksum": "",
        "workerVersion": {
          "buildId": "e35c66928e1642979bc05783460d8c66",
          "bundleId": "",
          "useVersioning": false
        },
        "sdkMetadata": {
          "coreUsedFlags": [],
          "langUsedFlags": []
        },
        "meteringMetadata": {
          "nonfirstLocalActivityExecutionAttempts": 0
        }
      }
    },
    {
      "eventId": "41",
      "eventTime": "2024-02-08T11:54:50.983500802Z",
      "eventType": "ActivityTaskScheduled",
      "version": "0",
      "taskId": "9437295",
      "workerMayIgnore": false,
      "activityTaskScheduledEventAttributes": {
        "activityId": "41",
        "activityType": {
          "name": "JobStatus"
        },
        "taskQueue": {
          "name": "JOB_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "header": {
          "fields": {}
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "1800s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "40",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s",
          "maximumAttempts": 5,
          "nonRetryableErrorTypes": []
        },
        "useCompatibleVersion": false
      }
    },
    {
      "eventId": "42",
      "eventTime": "2024-02-08T11:54:51.019865793Z",
      "eventType": "ActivityTaskStarted",
      "version": "0",
      "taskId": "9437300",
      "workerMayIgnore": false,
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "41",
        "identity": "1@job-service-fd6b78779-2klhz@",
        "requestId": "6d4bbafc-a519-44c9-b79e-2a25538335a4",
        "attempt": 1,
        "lastFailure": null
      }
    },
    {
      "eventId": "43",
      "eventTime": "2024-02-08T11:54:51.049961993Z",
      "eventType": "ActivityTaskCompleted",
      "version": "0",
      "taskId": "9437301",
      "workerMayIgnore": false,
      "activityTaskCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduledEventId": "41",
        "startedEventId": "42",
        "identity": "1@job-service-fd6b78779-2klhz@",
        "workerVersion": null
      }
    },
    {
      "eventId": "44",
      "eventTime": "2024-02-08T11:54:51.049979842Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "9437302",
      "workerMayIgnore": false,
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "cesar-bridge-79898fbd9d-stwmc:aa654300-ea1b-4edb-b2ac-86f15b45fb52",
          "kind": "Sticky",
          "normalName": "CESAR_BRIDGE_TASK_QUEUE"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "45",
      "eventTime": "2024-02-08T11:54:51.086599785Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "9437306",
      "workerMayIgnore": false,
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "44",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "c7fd843e-8f88-480c-9021-67342b5dd8cc",
        "suggestContinueAsNew": false,
        "historySizeBytes": "9960"
      }
    },
    {
      "eventId": "46",
      "eventTime": "2024-02-08T11:54:51.117439540Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "9437310",
      "workerMayIgnore": false,
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "44",
        "startedEventId": "45",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "binaryChecksum": "",
        "workerVersion": {
          "buildId": "e35c66928e1642979bc05783460d8c66",
          "bundleId": "",
          "useVersioning": false
        },
        "sdkMetadata": {
          "coreUsedFlags": [],
          "langUsedFlags": []
        },
        "meteringMetadata": {
          "nonfirstLocalActivityExecutionAttempts": 0
        }
      }
    },
    {
      "eventId": "47",
      "eventTime": "2024-02-08T11:54:51.117544053Z",
      "eventType": "ActivityTaskScheduled",
      "version": "0",
      "taskId": "9437311",
      "workerMayIgnore": false,
      "activityTaskScheduledEventAttributes": {
        "activityId": "47",
        "activityType": {
          "name": "CesarBridgeSetProvisionStatus"
        },
        "taskQueue": {
          "name": "CESAR_BRIDGE_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "header": {
          "fields": {}
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "600s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "46",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s",
          "maximumAttempts": 5,
          "nonRetryableErrorTypes": []
        },
        "useCompatibleVersion": true
      }
    },
    {
      "eventId": "48",
      "eventTime": "2024-02-08T11:54:51.153050213Z",
      "eventType": "ActivityTaskStarted",
      "version": "0",
      "taskId": "9437316",
      "workerMayIgnore": false,
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "47",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "5a314feb-59d8-4191-8f4b-8581ddc9c56a",
        "attempt": 1,
        "lastFailure": null
      }
    },
    {
      "eventId": "49",
      "eventTime": "2024-02-08T11:54:51.223559942Z",
      "eventType": "ActivityTaskCompleted",
      "version": "0",
      "taskId": "9437317",
      "workerMayIgnore": false,
      "activityTaskCompletedEventAttributes": {
        "result": null,
        "scheduledEventId": "47",
        "startedEventId": "48",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "workerVersion": null
      }
    },
    {
      "eventId": "50",
      "eventTime": "2024-02-08T11:54:51.223570674Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "9437318",
      "workerMayIgnore": false,
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "cesar-bridge-79898fbd9d-stwmc:aa654300-ea1b-4edb-b2ac-86f15b45fb52",
          "kind": "Sticky",
          "normalName": "CESAR_BRIDGE_TASK_QUEUE"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "51",
      "eventTime": "2024-02-08T11:54:51.260602762Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "9437322",
      "workerMayIgnore": false,
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "50",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "7d84e51d-8316-453f-8053-66f027b04721",
        "suggestContinueAsNew": false,
        "historySizeBytes": "10835"
      }
    },
    {
      "eventId": "52",
      "eventTime": "2024-02-08T11:54:51.287642009Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "9437326",
      "workerMayIgnore": false,
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "50",
        "startedEventId": "51",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "binaryChecksum": "",
        "workerVersion": {
          "buildId": "e35c66928e1642979bc05783460d8c66",
          "bundleId": "",
          "useVersioning": false
        },
        "sdkMetadata": {
          "coreUsedFlags": [],
          "langUsedFlags": []
        },
        "meteringMetadata": {
          "nonfirstLocalActivityExecutionAttempts": 0
        }
      }
    },
    {
      "eventId": "53",
      "eventTime": "2024-02-08T11:54:51.287754674Z",
      "eventType": "ActivityTaskScheduled",
      "version": "0",
      "taskId": "9437327",
      "workerMayIgnore": false,
      "activityTaskScheduledEventAttributes": {
        "activityId": "53",
        "activityType": {
          "name": "CesarBridgeCleanupStagedProvisionPackage"
        },
        "taskQueue": {
          "name": "CESAR_BRIDGE_TASK_QUEUE",
          "kind": "Normal",
          "normalName": ""
        },
        "header": {
          "fields": {}
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "600s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "52",
        "retryPolicy": {
          "initialInterval": "1s",
          "backoffCoefficient": 2,
          "maximumInterval": "100s",
          "maximumAttempts": 5,
          "nonRetryableErrorTypes": []
        },
        "useCompatibleVersion": true
      }
    },
    {
      "eventId": "54",
      "eventTime": "2024-02-08T11:54:51.321333588Z",
      "eventType": "ActivityTaskStarted",
      "version": "0",
      "taskId": "9437332",
      "workerMayIgnore": false,
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "53",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "12c56c69-5a09-40d9-a4c1-c2ddc29a654a",
        "attempt": 1,
        "lastFailure": null
      }
    },
    {
      "eventId": "55",
      "eventTime": "2024-02-08T11:54:51.370453585Z",
      "eventType": "ActivityTaskCompleted",
      "version": "0",
      "taskId": "9437333",
      "workerMayIgnore": false,
      "activityTaskCompletedEventAttributes": {
        "result": null,
        "scheduledEventId": "53",
        "startedEventId": "54",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "workerVersion": null
      }
    },
    {
      "eventId": "56",
      "eventTime": "2024-02-08T11:54:51.370470744Z",
      "eventType": "WorkflowTaskScheduled",
      "version": "0",
      "taskId": "9437334",
      "workerMayIgnore": false,
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "cesar-bridge-79898fbd9d-stwmc:aa654300-ea1b-4edb-b2ac-86f15b45fb52",
          "kind": "Sticky",
          "normalName": "CESAR_BRIDGE_TASK_QUEUE"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "57",
      "eventTime": "2024-02-08T11:54:51.397294390Z",
      "eventType": "WorkflowTaskStarted",
      "version": "0",
      "taskId": "9437338",
      "workerMayIgnore": false,
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "56",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "requestId": "9d082c5c-32cc-4895-9ccc-6f41536931d8",
        "suggestContinueAsNew": false,
        "historySizeBytes": "11640"
      }
    },
    {
      "eventId": "58",
      "eventTime": "2024-02-08T11:54:51.431113788Z",
      "eventType": "WorkflowTaskCompleted",
      "version": "0",
      "taskId": "9437342",
      "workerMayIgnore": false,
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "56",
        "startedEventId": "57",
        "identity": "1@cesar-bridge-79898fbd9d-stwmc@",
        "binaryChecksum": "",
        "workerVersion": {
          "buildId": "e35c66928e1642979bc05783460d8c66",
          "bundleId": "",
          "useVersioning": false
        },
        "sdkMetadata": {
          "coreUsedFlags": [],
          "langUsedFlags": []
        },
        "meteringMetadata": {
          "nonfirstLocalActivityExecutionAttempts": 0
        }
      }
    },
    {
      "eventId": "59",
      "eventTime": "2024-02-08T11:54:51.431246536Z",
      "eventType": "WorkflowExecutionCompleted",
      "version": "0",
      "taskId": "9437343",
      "workerMayIgnore": false,
      "workflowExecutionCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "*censored*"
            }
          ]
        },
        "workflowTaskCompletedEventId": "58",
        "newExecutionRunId": ""
      }
    }
  ]
}
5

well,
the service goes Out Of Memory and is killed :smiley:
Pretty happy with temporal recovering from such state.