Temporal Activity Poll & Start Delays - Issues under Load

Community Support
,
1

Overview : Spring Application Running with Simple Workflow with Multiple Activity Tasks. The tasks are scheduled , however not picked up with the Activity Worker. The activity breaches the StartToClose Timeout and Temporal Fails activity marks for next attempt, The Second attempt is successful and completes immediately.

Cluster Setup: Temporal on Minikube , deployed with helm-charts

  • 512 Shards, however only single Single History Instance
  • 10 Partitions per Task Queue.
  • Database Postgres

Spring Boot Application : WorkflowImpl

  • StartToCloseTimeout : 10 Minutes
  • RetryInterval : 2 Seconds.

Temporal Web UI During Run,
Note : 12 Workflows out of 400+ are held up , Activity scheduled but Not Started

image
image940×541 114 KB

image
image940×342 58.8 KB

image
image940×491 79.7 KB

Temporal Web UI After all Workflows Completed. (Same Activity )

image
image940×315 50 KB

image
image940×441 63.6 KB

image
image940×422 68.1 KB

Grafana Snapshot Shared during Period.

Notes :

  • The 12 activities await at the same point for exactly 10 minutes (StartToClose Timeout Value)
  • The number of attempts shows as 2.
3

The tasks are scheduled , however not picked up with the Activity Worker.

  • The number of attempts shows as 2.

Because not every activity retry is recorded in event history, ActivityTaskStarted event is written in history after all activity retries are done (when activity completes, fails, times out after all retries).
Would look at the “Pending activities” view in your web ui or use DescribeWorkflowExecution api (or for example tctl wf desc command) to see pending activity info for this activity execution.
What was the attempt one failure? Do you see it in pending activity info while activity is running / your worker logs?

5

Workflow in Running State for 10 Minutes

./tctl  --namespace fastns  wf desc --workflow_id LWF331
{
  "executionConfig": {
    "taskQueue": {
      "name": "SIMPLE_TASK_QUEUE",
      "kind": "Normal"
    },
    "workflowExecutionTimeout": "0s",
    "workflowRunTimeout": "0s",
    "defaultWorkflowTaskTimeout": "10s"
  },
  "workflowExecutionInfo": {
    "execution": {
      "workflowId": "LWF331",
      "runId": "1f30eae0-026b-4fb2-a385-39700161ae54"
    },
    "type": {
      "name": "LoadedWorkflow"
    },
    "startTime": "2023-05-23T05:46:06.532201744Z",
    "status": "Running",
    "historyLength": "5",
    "memo": {

    },
    "searchAttributes": {
      "indexedFields": {
        "AppReference": "APPREF331"
      }
    },
    "autoResetPoints": {

    },
    "stateTransitionCount": "4"
  },
  "pendingActivities": [
    {
      "activityId": "fc904d55-0aa3-3251-ba5e-26ebfaf4c359",
      "activityType": {
        "name": "TasFunction01"
      },
      "state": "Started",
      "lastStartedTime": "2023-05-23T05:46:06.785104954Z",
      "attempt": 1,
      "expirationTime": "0001-01-01T00:00:00Z",
      "lastWorkerIdentity": "1@spring-boot-app-574cc75775-l676f"
    }
  ]
}

After Workflow Completed

./tctl --address  --namespace fastns  wf desc --workflow_id LWF331
{
  "executionConfig": {
    "taskQueue": {
      "name": "SIMPLE_TASK_QUEUE",
      "kind": "Normal"
    },
    "workflowExecutionTimeout": "0s",
    "workflowRunTimeout": "0s",
    "defaultWorkflowTaskTimeout": "10s"
  },
  "workflowExecutionInfo": {
    "execution": {
      "workflowId": "LWF331",
      "runId": "1f30eae0-026b-4fb2-a385-39700161ae54"
    },
    "type": {
      "name": "LoadedWorkflow"
    },
    "startTime": "2023-05-23T05:46:06.532201744Z",
    "closeTime": "2023-05-23T05:56:14.131851040Z",
    "status": "Completed",
    "historyLength": "65",
    "memo": {

    },
    "searchAttributes": {
      "indexedFields": {
        "AppReference": "APPREF331"
      }
    },
    "autoResetPoints": {

    },
    "stateTransitionCount": "45"
  }
}
6

Could you share the event history json of this execution (after it completes):

tctl wf show -w <wfid> -r <runid> --of myhistory.json

This will give us more details.

7

@tihomir sure will do that later today.

The activities implementation itself is very light weight and have verified that although the ‘pendingActivties.state’ = “Started” the Logs were not invoked suggesting that the Activity Implementation was not executed.

temporalio-minikube-ha-cluster/LoadedActivitiesImpl.java at main · anil-oommen/temporalio-minikube-ha-cluster · GitHub

8

./tctl --namespace fastns wf desc --workflow_id LWF295

Handling connection for 15233
{
  "executionConfig": {
    "taskQueue": {
      "name": "SIMPLE_TASK_QUEUE",
      "kind": "Normal"
    },
    "workflowExecutionTimeout": "0s",
    "workflowRunTimeout": "0s",
    "defaultWorkflowTaskTimeout": "10s"
  },
  "workflowExecutionInfo": {
    "execution": {
      "workflowId": "LWF295",
      "runId": "8e9229d4-6349-4bc0-9a34-5c73aa4ac3d0"
    },
    "type": {
      "name": "LoadedWorkflow"
    },
    "startTime": "2023-05-24T12:55:54.447049181Z",
    "status": "Running",
    "historyLength": "11",
    "memo": {

    },
    "searchAttributes": {
      "indexedFields": {
        "AppReference": "APPREF295"
      }
    },
    "autoResetPoints": {

    },
    "stateTransitionCount": "8"
  },
  "pendingActivities": [
    {
      "activityId": "5d210082-90b0-3bdc-9beb-2596c3ff559d",
      "activityType": {
        "name": "TasFunction02"
      },
      "state": "Started",
      "lastStartedTime": "2023-05-24T12:55:55.276451415Z",
      "attempt": 1,
      "expirationTime": "0001-01-01T00:00:00Z",
      "lastWorkerIdentity": "1@spring-boot-app-574cc75775-tm2vf"
    }
  ]
}
9

./tctl --namespace fastns wf show --workflow_id LWF295 -r 8e9229d4-6349-4bc0-9a34-5c73aa4ac3d0 --of myhistory.json

{
 "events": [
  {
   "eventId": "1",
   "eventTime": "2023-05-24T12:55:54.447049181Z",
   "eventType": "WorkflowExecutionStarted",
   "taskId": "27262984",
   "workflowExecutionStartedEventAttributes": {
    "workflowType": {
     "name": "LoadedWorkflow"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "workflowExecutionTimeout": "0s",
    "workflowRunTimeout": "0s",
    "workflowTaskTimeout": "10s",
    "originalExecutionRunId": "8e9229d4-6349-4bc0-9a34-5c73aa4ac3d0",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "firstExecutionRunId": "8e9229d4-6349-4bc0-9a34-5c73aa4ac3d0",
    "attempt": 1,
    "firstWorkflowTaskBackoff": "0s",
    "searchAttributes": {
     "indexedFields": {
      "AppReference": {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg==",
        "type": "VGV4dA=="
       },
       "data": "IkFQUFJFRjI5NSI="
      }
     }
    },
    "header": {

    }
   }
  },
  {
   "eventId": "2",
   "eventTime": "2023-05-24T12:55:54.447094087Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27262985",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "3",
   "eventTime": "2023-05-24T12:55:54.488587402Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27262990",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "2",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "0db86f16-cb51-4f3d-bd8c-a848046325dc",
    "historySizeBytes": "366"
   }
  },
  {
   "eventId": "4",
   "eventTime": "2023-05-24T12:55:54.611982821Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27262994",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "2",
    "startedEventId": "3",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "5",
   "eventTime": "2023-05-24T12:55:54.612017149Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "27262995",
   "activityTaskScheduledEventAttributes": {
    "activityId": "a5de4ed6-57f1-3e1e-a2c0-6b4c9235f05b",
    "activityType": {
     "name": "TasFunction01"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "600s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "4",
    "retryPolicy": {
     "initialInterval": "2s",
     "backoffCoefficient": 2,
     "maximumInterval": "2s"
    }
   }
  },
  {
   "eventId": "6",
   "eventTime": "2023-05-24T12:55:54.660549574Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "27263000",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "5",
    "identity": "1@spring-boot-app-574cc75775-jrklt",
    "requestId": "bff32b89-3bad-4d9b-85f2-bb80b1591859",
    "attempt": 1
   }
  },
  {
   "eventId": "7",
   "eventTime": "2023-05-24T12:55:55.183003573Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "27263001",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "YmluYXJ5L251bGw="
       }
      }
     ]
    },
    "scheduledEventId": "5",
    "startedEventId": "6",
    "identity": "1@spring-boot-app-574cc75775-jrklt"
   }
  },
  {
   "eventId": "8",
   "eventTime": "2023-05-24T12:55:55.183010422Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27263002",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "1@spring-boot-app-574cc75775-tm2vf:4a0b756c-b288-4477-a9af-0dc5060886fe",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "9",
   "eventTime": "2023-05-24T12:55:55.215244346Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27263006",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "8",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "4bea095b-cd75-41fd-88b7-80b4bfe36492",
    "historySizeBytes": "1031"
   }
  },
  {
   "eventId": "10",
   "eventTime": "2023-05-24T12:55:55.240820380Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27263010",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "8",
    "startedEventId": "9",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "11",
   "eventTime": "2023-05-24T12:55:55.240867407Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "27263011",
   "activityTaskScheduledEventAttributes": {
    "activityId": "5d210082-90b0-3bdc-9beb-2596c3ff559d",
    "activityType": {
     "name": "TasFunction02"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "600s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "10",
    "retryPolicy": {
     "initialInterval": "2s",
     "backoffCoefficient": 2,
     "maximumInterval": "2s"
    }
   }
  },
  {
   "eventId": "12",
   "eventTime": "2023-05-24T13:05:57.280621276Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "27263019",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "11",
    "identity": "1@spring-boot-app-574cc75775-hvrdk",
    "requestId": "ddced5cf-63cb-4699-b7c8-2436583b8e7f",
    "attempt": 2,
    "lastFailure": {
     "message": "activity StartToClose timeout",
     "source": "Server",
     "timeoutFailureInfo": {
      "timeoutType": "StartToClose"
     }
    }
   }
  },
  {
   "eventId": "13",
   "eventTime": "2023-05-24T13:05:57.793619214Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "27263020",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "YmluYXJ5L251bGw="
       }
      }
     ]
    },
    "scheduledEventId": "11",
    "startedEventId": "12",
    "identity": "1@spring-boot-app-574cc75775-hvrdk"
   }
  },
  {
   "eventId": "14",
   "eventTime": "2023-05-24T13:05:57.793625315Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27263021",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "1@spring-boot-app-574cc75775-tm2vf:4a0b756c-b288-4477-a9af-0dc5060886fe",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "15",
   "eventTime": "2023-05-24T13:05:57.815647948Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27263025",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "14",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "479f3250-71e9-4f60-b0ab-736e98f36efe",
    "historySizeBytes": "1741"
   }
  },
  {
   "eventId": "16",
   "eventTime": "2023-05-24T13:05:59.578268708Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27263029",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "14",
    "startedEventId": "15",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "17",
   "eventTime": "2023-05-24T13:05:59.578316324Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "27263030",
   "activityTaskScheduledEventAttributes": {
    "activityId": "e2be5253-1717-3308-a7d4-6c97abb84b50",
    "activityType": {
     "name": "TasFunction03"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "600s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "16",
    "retryPolicy": {
     "initialInterval": "2s",
     "backoffCoefficient": 2,
     "maximumInterval": "2s"
    }
   }
  },
  {
   "eventId": "18",
   "eventTime": "2023-05-24T13:05:59.584550036Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "27263035",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "17",
    "identity": "1@spring-boot-app-574cc75775-hvrdk",
    "requestId": "ec62b761-b09b-4576-832f-523b4a50295a",
    "attempt": 1
   }
  },
  {
   "eventId": "19",
   "eventTime": "2023-05-24T13:06:00.091522389Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "27263036",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "YmluYXJ5L251bGw="
       }
      }
     ]
    },
    "scheduledEventId": "17",
    "startedEventId": "18",
    "identity": "1@spring-boot-app-574cc75775-hvrdk"
   }
  },
  {
   "eventId": "20",
   "eventTime": "2023-05-24T13:06:00.091529227Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27263037",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "1@spring-boot-app-574cc75775-tm2vf:205963d9-ecf5-47e2-9b59-7c9b9a7df1a7",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "21",
   "eventTime": "2023-05-24T13:06:00.100315997Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27263041",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "20",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "2b39bc97-8068-4bda-a4f6-c972e0f66c21",
    "historySizeBytes": "2406"
   }
  },
  {
   "eventId": "22",
   "eventTime": "2023-05-24T13:06:03.575154825Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27263045",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "20",
    "startedEventId": "21",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "23",
   "eventTime": "2023-05-24T13:06:03.575204637Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "27263046",
   "activityTaskScheduledEventAttributes": {
    "activityId": "b3cd40d6-535a-3a57-9e3c-16211ae45b87",
    "activityType": {
     "name": "TasFunction04"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "600s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "22",
    "retryPolicy": {
     "initialInterval": "2s",
     "backoffCoefficient": 2,
     "maximumInterval": "2s"
    }
   }
  },
  {
   "eventId": "24",
   "eventTime": "2023-05-24T13:06:03.581707937Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "27263051",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "23",
    "identity": "1@spring-boot-app-574cc75775-hvrdk",
    "requestId": "48fd8b7e-e182-464b-83d2-3f4e621b5a08",
    "attempt": 1
   }
  },
  {
   "eventId": "25",
   "eventTime": "2023-05-24T13:06:04.097491773Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "27263052",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "YmluYXJ5L251bGw="
       }
      }
     ]
    },
    "scheduledEventId": "23",
    "startedEventId": "24",
    "identity": "1@spring-boot-app-574cc75775-hvrdk"
   }
  },
  {
   "eventId": "26",
   "eventTime": "2023-05-24T13:06:04.097500086Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27263053",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "1@spring-boot-app-574cc75775-tm2vf:205963d9-ecf5-47e2-9b59-7c9b9a7df1a7",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "27",
   "eventTime": "2023-05-24T13:06:04.105608854Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27263057",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "26",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "67ae9e4a-ed08-4dbf-8656-67ea352215b7",
    "historySizeBytes": "3070"
   }
  },
  {
   "eventId": "28",
   "eventTime": "2023-05-24T13:06:04.117860963Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27263061",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "26",
    "startedEventId": "27",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "29",
   "eventTime": "2023-05-24T13:06:04.117894570Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "27263062",
   "activityTaskScheduledEventAttributes": {
    "activityId": "a3f84a32-de2e-3991-9523-0650ce1cefdf",
    "activityType": {
     "name": "TasFunction05"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "600s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "28",
    "retryPolicy": {
     "initialInterval": "2s",
     "backoffCoefficient": 2,
     "maximumInterval": "2s"
    }
   }
  },
  {
   "eventId": "30",
   "eventTime": "2023-05-24T13:06:04.123599536Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "27263067",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "29",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "152ee546-ee77-4ce8-92fb-5be61462b8c5",
    "attempt": 1
   }
  },
  {
   "eventId": "31",
   "eventTime": "2023-05-24T13:06:04.634610392Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "27263068",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "YmluYXJ5L251bGw="
       }
      }
     ]
    },
    "scheduledEventId": "29",
    "startedEventId": "30",
    "identity": "1@spring-boot-app-574cc75775-tm2vf"
   }
  },
  {
   "eventId": "32",
   "eventTime": "2023-05-24T13:06:04.634622121Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27263069",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "1@spring-boot-app-574cc75775-tm2vf:205963d9-ecf5-47e2-9b59-7c9b9a7df1a7",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "33",
   "eventTime": "2023-05-24T13:06:04.644232365Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27263073",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "32",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "11268c89-b34e-4fb2-b778-21f912e279b7",
    "historySizeBytes": "3733"
   }
  },
  {
   "eventId": "34",
   "eventTime": "2023-05-24T13:06:04.658593596Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27263077",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "32",
    "startedEventId": "33",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "35",
   "eventTime": "2023-05-24T13:06:04.658639359Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "27263078",
   "activityTaskScheduledEventAttributes": {
    "activityId": "a72b71a6-3ecf-3064-80ff-504c213576d3",
    "activityType": {
     "name": "TasFunction06"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "600s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "34",
    "retryPolicy": {
     "initialInterval": "2s",
     "backoffCoefficient": 2,
     "maximumInterval": "2s"
    }
   }
  },
  {
   "eventId": "36",
   "eventTime": "2023-05-24T13:06:04.665392111Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "27263083",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "35",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "22e861e6-d60f-498b-8916-f1a0cd9efe46",
    "attempt": 1
   }
  },
  {
   "eventId": "37",
   "eventTime": "2023-05-24T13:06:05.179225034Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "27263084",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "YmluYXJ5L251bGw="
       }
      }
     ]
    },
    "scheduledEventId": "35",
    "startedEventId": "36",
    "identity": "1@spring-boot-app-574cc75775-tm2vf"
   }
  },
  {
   "eventId": "38",
   "eventTime": "2023-05-24T13:06:05.179237513Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27263085",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "1@spring-boot-app-574cc75775-tm2vf:205963d9-ecf5-47e2-9b59-7c9b9a7df1a7",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "39",
   "eventTime": "2023-05-24T13:06:05.187502848Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27263089",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "38",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "18a01e94-70c3-433c-83e4-de4f85f0591e",
    "historySizeBytes": "4398"
   }
  },
  {
   "eventId": "40",
   "eventTime": "2023-05-24T13:06:05.201656093Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27263093",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "38",
    "startedEventId": "39",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "41",
   "eventTime": "2023-05-24T13:06:05.201692136Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "27263094",
   "activityTaskScheduledEventAttributes": {
    "activityId": "49db6ea8-b287-3dcd-95c3-5a31b40333cb",
    "activityType": {
     "name": "TasFunction07"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "600s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "40",
    "retryPolicy": {
     "initialInterval": "2s",
     "backoffCoefficient": 2,
     "maximumInterval": "2s"
    }
   }
  },
  {
   "eventId": "42",
   "eventTime": "2023-05-24T13:06:05.208489038Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "27263099",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "41",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "c4e3f304-8c0d-4966-b7e6-3dcf8f278449",
    "attempt": 1
   }
  },
  {
   "eventId": "43",
   "eventTime": "2023-05-24T13:06:05.720950052Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "27263100",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "YmluYXJ5L251bGw="
       }
      }
     ]
    },
    "scheduledEventId": "41",
    "startedEventId": "42",
    "identity": "1@spring-boot-app-574cc75775-tm2vf"
   }
  },
  {
   "eventId": "44",
   "eventTime": "2023-05-24T13:06:05.720958695Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27263101",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "1@spring-boot-app-574cc75775-tm2vf:205963d9-ecf5-47e2-9b59-7c9b9a7df1a7",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "45",
   "eventTime": "2023-05-24T13:06:05.730624970Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27263105",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "44",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "9b3dde65-b74f-4c7f-90f6-05e89bd5a287",
    "historySizeBytes": "5061"
   }
  },
  {
   "eventId": "46",
   "eventTime": "2023-05-24T13:06:05.745802632Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27263109",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "44",
    "startedEventId": "45",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "47",
   "eventTime": "2023-05-24T13:06:05.745848221Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "27263110",
   "activityTaskScheduledEventAttributes": {
    "activityId": "64488e20-79e4-369a-a441-ec54e5562e67",
    "activityType": {
     "name": "TasFunction08"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "600s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "46",
    "retryPolicy": {
     "initialInterval": "2s",
     "backoffCoefficient": 2,
     "maximumInterval": "2s"
    }
   }
  },
  {
   "eventId": "48",
   "eventTime": "2023-05-24T13:06:05.751186186Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "27263115",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "47",
    "identity": "1@spring-boot-app-574cc75775-hvrdk",
    "requestId": "55068b57-b03c-46d0-9032-83e430e20c15",
    "attempt": 1
   }
  },
  {
   "eventId": "49",
   "eventTime": "2023-05-24T13:06:06.258778143Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "27263116",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "YmluYXJ5L251bGw="
       }
      }
     ]
    },
    "scheduledEventId": "47",
    "startedEventId": "48",
    "identity": "1@spring-boot-app-574cc75775-hvrdk"
   }
  },
  {
   "eventId": "50",
   "eventTime": "2023-05-24T13:06:06.258794136Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27263117",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "1@spring-boot-app-574cc75775-tm2vf:205963d9-ecf5-47e2-9b59-7c9b9a7df1a7",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "51",
   "eventTime": "2023-05-24T13:06:06.267976536Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27263121",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "50",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "9d193541-36d9-4ac9-bb96-9b07379c5b00",
    "historySizeBytes": "5726"
   }
  },
  {
   "eventId": "52",
   "eventTime": "2023-05-24T13:06:06.278051326Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27263125",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "50",
    "startedEventId": "51",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "53",
   "eventTime": "2023-05-24T13:06:06.278130941Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "27263126",
   "activityTaskScheduledEventAttributes": {
    "activityId": "43cc71d6-afff-30d4-ab06-4be2a99a4289",
    "activityType": {
     "name": "TasFunction09"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "600s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "52",
    "retryPolicy": {
     "initialInterval": "2s",
     "backoffCoefficient": 2,
     "maximumInterval": "2s"
    }
   }
  },
  {
   "eventId": "54",
   "eventTime": "2023-05-24T13:06:06.284598969Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "27263131",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "53",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "a3e91783-6723-482e-8445-d1dbc8656903",
    "attempt": 1
   }
  },
  {
   "eventId": "55",
   "eventTime": "2023-05-24T13:06:06.795382517Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "27263132",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "YmluYXJ5L251bGw="
       }
      }
     ]
    },
    "scheduledEventId": "53",
    "startedEventId": "54",
    "identity": "1@spring-boot-app-574cc75775-tm2vf"
   }
  },
  {
   "eventId": "56",
   "eventTime": "2023-05-24T13:06:06.795391776Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27263133",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "1@spring-boot-app-574cc75775-tm2vf:205963d9-ecf5-47e2-9b59-7c9b9a7df1a7",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "57",
   "eventTime": "2023-05-24T13:06:06.812645383Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27263137",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "56",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "7d5980ea-2d87-4269-854a-d9eec51f0570",
    "historySizeBytes": "6392"
   }
  },
  {
   "eventId": "58",
   "eventTime": "2023-05-24T13:06:06.823477882Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27263141",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "56",
    "startedEventId": "57",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "59",
   "eventTime": "2023-05-24T13:06:06.823514002Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "27263142",
   "activityTaskScheduledEventAttributes": {
    "activityId": "52363a2f-57ee-3ca6-aea4-c176d7454f96",
    "activityType": {
     "name": "TasFunction10"
    },
    "taskQueue": {
     "name": "SIMPLE_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "600s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "58",
    "retryPolicy": {
     "initialInterval": "2s",
     "backoffCoefficient": 2,
     "maximumInterval": "2s"
    }
   }
  },
  {
   "eventId": "60",
   "eventTime": "2023-05-24T13:06:06.828826255Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "27263147",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "59",
    "identity": "1@spring-boot-app-574cc75775-jrklt",
    "requestId": "61df898b-d762-4fb4-905e-dcb248ef864c",
    "attempt": 1
   }
  },
  {
   "eventId": "61",
   "eventTime": "2023-05-24T13:06:07.334703753Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "27263148",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "YmluYXJ5L251bGw="
       }
      }
     ]
    },
    "scheduledEventId": "59",
    "startedEventId": "60",
    "identity": "1@spring-boot-app-574cc75775-jrklt"
   }
  },
  {
   "eventId": "62",
   "eventTime": "2023-05-24T13:06:07.334711447Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "27263149",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "1@spring-boot-app-574cc75775-tm2vf:205963d9-ecf5-47e2-9b59-7c9b9a7df1a7",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "63",
   "eventTime": "2023-05-24T13:06:07.343936159Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "27263153",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "62",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "requestId": "af325b1d-691e-4c58-88fd-82dc3c0591f1",
    "historySizeBytes": "7059"
   }
  },
  {
   "eventId": "64",
   "eventTime": "2023-05-24T13:06:07.355801533Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "27263157",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "62",
    "startedEventId": "63",
    "identity": "1@spring-boot-app-574cc75775-tm2vf",
    "meteringMetadata": {

    }
   }
  },
  {
   "eventId": "65",
   "eventTime": "2023-05-24T13:06:07.355828106Z",
   "eventType": "WorkflowExecutionCompleted",
   "taskId": "27263158",
   "workflowExecutionCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IkxXRjI5NSI="
      }
     ]
    },
    "workflowTaskCompletedEventId": "64"
   }
  }
 ]
}