Web UI with Elasticsearch No history and Stacktrace

Community Support
1

Hello Everyone,

We have configured Elasticsearch with our temporal instance. From the WebUI the workflow is being listed. The only problem we have no is that we can not see History and StackTrace from the UI!!!
Here is our dynamic configs, Any idea what could the poblem be!

    frontend.enableClientVersionCheck:
      - value: true
        constraints: {}
    history.persistenceMaxQPS:
      - value: 3000
        constraints: {}
    frontend.persistenceMaxQPS:
      - value: 3000
        constraints: {}
    frontend.historyMgrNumConns:
      - value: 10
        constraints: {}
    frontend.sendRawWorkflowHistory:
      - value: true
        constraints: {}
    frontend.throttledLogRPS:
      - value: 20
        constraints: {}
    history.historyMgrNumConns:
      - value: 50
        constraints: {}
    system.advancedVisibilityWritingMode:
      - value: "on"
        constraints: {}
    system.enableReadVisibilityFromES:
      - value: true
        constraints: {}
    system.enableWriteToSecondaryAdvancedVisibility:
      - value: true
        constraints: {}
    system.enableReadFromSecondaryAdvancedVisibility:
      - value: true
        constraints: {}
    history.defaultActivityRetryPolicy:
      - value:
          InitialIntervalInSeconds: 1
          MaximumIntervalCoefficient: 100.0
          BackoffCoefficient: 2.0
          MaximumAttempts: 0
    history.defaultWorkflowRetryPolicy:
      - value:
          InitialIntervalInSeconds: 1
          MaximumIntervalCoefficient: 100.0
          BackoffCoefficient: 2.0
          MaximumAttempts: 0
2

The only problem we have no is that we can not see History and StackTrace from the UI

Does your Web UI show any errors on the history page or is it just blank? Could you share a screenshot?
So listing works, and do you get info on the Summary page when you click on one of your listed runids?

Regarding the stack trace, its only available for running workflows. Once a workflow completes, it would not longer be shown.

What do you get when you run:
tctl workflow list --query "WorkflowId=<workflow_id>"

If that gives a failure but you get a result via:

tctl workflow describe --workflow_id <workflow_id>

Then its most likely ES configuration issue.

3

Hello @tihomir,

Thank you for replying. Yes you are right. The first command is giving this error

tctl --ns cigar workflow list --query “WorkflowId=process-transaction-workflow-0adeeb0b-ba47-4579-b75d-c7ae7b0dc159”
Error: Failed to list workflow.
Error Details: invalid query: malformed SQL query: syntax error at position 69 near ‘0’
(‘export TEMPORAL_CLI_SHOW_STACKS=1’ to see stack traces)

The second one is working
tctl --ns cigar workflow describe --workflow_id process-transaction-workflow-0adeeb0b-ba47-4579-b75d-c7ae7b0dc159

How can I idenify the issue?

And from the UI the is no error it only shows no result

4

Screen Shot 2022-04-02 at 5.07.09 PM
Screen Shot 2022-04-02 at 5.07.09 PM1920×1051 24.3 KB

5

Sorry the first command is working it was just syntax issue.

tctl --ns cigar workflow list --query “WorkflowId=‘process-transaction-workflow-0adeeb0b-ba47-4579-b75d-c7ae7b0dc159’”
WORKFLOW TYPE | WORKFLOW ID | RUN ID | TASK QUEUE | START TIME | EXECUTION TIME | END TIME
ProcessTransaction | process-transaction-workflow-0adeeb0b-ba47-4579-b75d-c7ae7b0dc159 | aba53473-7af6-455f-a884-9f93f47b6633 | ACCOUNTS_WORKER_TASK_QUEUE | 21:30:20 | 21:30:20 | 00:00:00

But still no result in the history in the web UI

6

But still no result in the history in the web UI

What do you get when you run:

tctl wf show -w <wfid> -r <runid> --output_filename myhistory.json

What is then in myhistory.json?

7

@tihomir it shows the full history

{
 "events": [
  {
   "eventId": "1",
   "eventTime": "2022-03-31T21:30:20.156538828Z",
   "eventType": "WorkflowExecutionStarted",
   "taskId": "134381610",
   "workflowExecutionStartedEventAttributes": {
    "workflowType": {
     "name": "ProcessTransaction"
    },
    "taskQueue": {
     "name": "ACCOUNTS_WORKER_TASK_QUEUE",
     "kind": "Normal"
    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IjBhZGVlYjBiLWJhNDctNDU3OS1iNzVkLWM3YWU3YjBkYzE1OSI="
      }
     ]
    },
    "workflowExecutionTimeout": "0s",
    "workflowRunTimeout": "0s",
    "workflowTaskTimeout": "10s",
    "originalExecutionRunId": "aba53473-7af6-455f-a884-9f93f47b6633",
    "identity": "1@accounts-5bf46cf54-wzv48@",
    "firstExecutionRunId": "aba53473-7af6-455f-a884-9f93f47b6633",
    "attempt": 1,
    "firstWorkflowTaskBackoff": "0s",
    "header": {

    }
   }
  },
  {
   "eventId": "2",
   "eventTime": "2022-03-31T21:30:20.156573934Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "134381611",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "ACCOUNTS_WORKER_TASK_QUEUE",
     "kind": "Normal"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "3",
   "eventTime": "2022-03-31T21:30:20.387055754Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "134381616",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "2",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "1cda48bb-eaba-49b6-bfd5-12cad08d7c84"
   }
  },
  {
   "eventId": "4",
   "eventTime": "2022-03-31T21:30:20.497426615Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "134381619",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "2",
    "startedEventId": "3",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "binaryChecksum": "8a87fc12fbf7170f6b22991bfa119e88"
   }
  },
  {
   "eventId": "5",
   "eventTime": "2022-03-31T21:30:20.497503510Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "134381620",
   "activityTaskScheduledEventAttributes": {
    "activityId": "5",
    "activityType": {
     "name": "GetTransaction"
    },
    "taskQueue": {
     "name": "ACCOUNTS_WORKER_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IjBhZGVlYjBiLWJhNDctNDU3OS1iNzVkLWM3YWU3YjBkYzE1OSI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "60s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "4",
    "retryPolicy": {
     "initialInterval": "1s",
     "backoffCoefficient": 2,
     "maximumInterval": "300s",
     "maximumAttempts": 300
    }
   }
  },
  {
   "eventId": "6",
   "eventTime": "2022-03-31T21:30:20.632790600Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "134381625",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "5",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "160840d4-ee27-4397-818c-3a8c97becade",
    "attempt": 1
   }
  },
  {
   "eventId": "7",
   "eventTime": "2022-03-31T21:30:20.735292747Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "134381626",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "Y2lnYXIuYWNjb3VudHMudjEuVHJhbnNhY3Rpb24="
       },
       "data": "eyJpZCI6IjBhZGVlYjBiLWJhNDctNDU3OS1iNzVkLWM3YWU3YjBkYzE1OSIsImFjY291bnRJZCI6IjZiYjQwZjVmLWM4MzUtNGNlNi1hNGE4LWE4NTkxNGI5OGE2ZiIsInJlZmVyZW5jZSI6IjZiY2U3NzI1LTNkMjMtNDgyNS1iYWU5LWU4YjY0NzI5YTllZCIsImRyQ3IiOiJDUkVESVQiLCJ0eXBlIjoiTk9PTl9UT1BVUF9NQURBX0NBUkQiLCJhbW91bnQiOiI1MDAwMDAiLCJwb3N0ZWRBdCI6IjIwMjItMDMtMzFUMjE6MzA6MjAuMTM4NDMwWiIsInN0YXR1cyI6IlBPU1RFRCIsImNyZWF0ZWRBdCI6IjIwMjItMDMtMzFUMjE6MzA6MjAuMTM4NDMwWiIsInVwZGF0ZWRBdCI6IjIwMjItMDMtMzFUMjE6MzA6MjAuMjU4NjM1WiIsImRldGFpbHMiOnsiY2FyZF9udW1iZXIiOiI1MjAwMDAqKioqKiowMDA3IiwicGF5bWVudF9pZCI6IjdjY2Y0ZmE3LThlYjktNDVhYS04NTJhLTQ5MTExOTFmZDJkNyIsInBheW1lbnRfbWV0aG9kX2lkIjoiNWRiZDFmYmItZDE2Ny00MjQ4LTkwMzItMTkxMmUyMzBjNGI0IiwicGF5bWVudF90eF9pZCI6IjZiY2U3NzI1LTNkMjMtNDgyNS1iYWU5LWU4YjY0NzI5YTllZCJ9fQ=="
      }
     ]
    },
    "scheduledEventId": "5",
    "startedEventId": "6",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@"
   }
  },
  {
   "eventId": "8",
   "eventTime": "2022-03-31T21:30:20.735301604Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "134381627",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "accounts-worker-9b59f8797-cwmh8:cd09ad10-dcd1-4f32-8fce-b54e489bdd5f",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "9",
   "eventTime": "2022-03-31T21:30:20.830553932Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "134381631",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "8",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "8b132859-d481-4d6f-bc4f-3838c8ed0dce"
   }
  },
  {
   "eventId": "10",
   "eventTime": "2022-03-31T21:30:20.952258711Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "134381635",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "8",
    "startedEventId": "9",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "binaryChecksum": "8a87fc12fbf7170f6b22991bfa119e88"
   }
  },
  {
   "eventId": "11",
   "eventTime": "2022-03-31T21:30:20.952305860Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "134381636",
   "activityTaskScheduledEventAttributes": {
    "activityId": "11",
    "activityType": {
     "name": "GetAccountDetailsById"
    },
    "taskQueue": {
     "name": "ACCOUNTS_WORKER_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wbGFpbg=="
       },
       "data": "IjZiYjQwZjVmLWM4MzUtNGNlNi1hNGE4LWE4NTkxNGI5OGE2ZiI="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "60s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "10",
    "retryPolicy": {
     "initialInterval": "1s",
     "backoffCoefficient": 2,
     "maximumInterval": "300s",
     "maximumAttempts": 300
    }
   }
  },
  {
   "eventId": "12",
   "eventTime": "2022-03-31T21:30:21.092556725Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "134381642",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "11",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "e1adc116-bd76-4294-b281-884c0a451a56",
    "attempt": 1
   }
  },
  {
   "eventId": "13",
   "eventTime": "2022-03-31T21:30:21.433793620Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "134381643",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "Y2lnYXIuYWNjb3VudHMudjEuQWNjb3VudA=="
       },
       "data": "eyJpZCI6IjZiYjQwZjVmLWM4MzUtNGNlNi1hNGE4LWE4NTkxNGI5OGE2ZiIsImN1c3RvbWVySWQiOiI4MDZiNjkwNy1hMzJiLTRiZWEtODZiOS1jYWUxOGIyMzYyMWIiLCJjdXJyZW5jeSI6IjY4MiIsImF2YWlsYWJsZUJhbGFuY2UiOiI1NDcxMjUiLCJhdmFpbGFibGVCYWxhbmNlSW5CYXNlQ3VycmVuY3kiOiI1NDcxMjUiLCJiYWxhbmNlIjoiNTQ3MTI1IiwiYmFsYW5jZUluQmFzZUN1cnJlbmN5IjoiNTQ3MTI1IiwiY3JlYXRlZEF0IjoiMjAyMi0wMy0yNFQxNDoxODoxNi4zNjg3MDFaIn0="
      }
     ]
    },
    "scheduledEventId": "11",
    "startedEventId": "12",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@"
   }
  },
  {
   "eventId": "14",
   "eventTime": "2022-03-31T21:30:21.433801294Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "134381644",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "accounts-worker-9b59f8797-cwmh8:cd09ad10-dcd1-4f32-8fce-b54e489bdd5f",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "15",
   "eventTime": "2022-03-31T21:30:21.569007592Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "134381649",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "14",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "f6dd5820-1d38-4c53-8a44-66773bac9d08"
   }
  },
  {
   "eventId": "16",
   "eventTime": "2022-03-31T21:30:21.665008759Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "134381653",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "14",
    "startedEventId": "15",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "binaryChecksum": "8a87fc12fbf7170f6b22991bfa119e88"
   }
  },
  {
   "eventId": "17",
   "eventTime": "2022-03-31T21:30:21.665049787Z",
   "eventType": "MarkerRecorded",
   "taskId": "134381654",
   "markerRecordedEventAttributes": {
    "markerName": "LocalActivity",
    "details": {
     "data": {
      "payloads": [
       {
        "metadata": {
         "encoding": "anNvbi9wbGFpbg=="
        },
        "data": "eyJBY3Rpdml0eUlEIjoiMSIsIkFjdGl2aXR5VHlwZSI6Ik1hcFRvTm90aWZpY2F0aW9uTWVzc2FnZSIsIlJlcGxheVRpbWUiOiIyMDIyLTAzLTMxVDIxOjMwOjIxLjU3MDE5MTA3NFoiLCJBdHRlbXB0IjoxLCJCYWNrb2ZmIjowfQ=="
       }
      ]
     },
     "result": {
      "payloads": [
       {
        "metadata": {
         "encoding": "anNvbi9wbGFpbg=="
        },
        "data": "eyJtZXNzYWdlSWQiOiJ0b3B1cC13aXRoLWNhcmQiLCJ0ZW1wbGF0ZURhdGEiOnsiQWNjb3VudCI6IlNBUiBBY2NvdW50IiwiQW1vdW50IjoiNTAwMDAwIiwiQ2FyZE51bWJlciI6IjUyMDAwMCoqKioqKjAwMDciLCJDdXJyZW5jeSI6IjY4MiIsIlRpbWUiOiIyMDIyLTAzLTMxVDIxOjMwOjIwWiJ9fQ=="
       }
      ]
     }
    },
    "workflowTaskCompletedEventId": "16"
   }
  },
  {
   "eventId": "18",
   "eventTime": "2022-03-31T21:30:21.665069344Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "134381655",
   "activityTaskScheduledEventAttributes": {
    "activityId": "18",
    "activityType": {
     "name": "SendNotificationToUsersByCustomerId"
    },
    "taskQueue": {
     "name": "NOTIFICATIONS_WORKER_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "Y2lnYXIubm90aWZpY2F0aW9ucy52MS5TZW5kTm90aWZpY2F0aW9uVG9Vc2Vyc0J5Q3VzdG9tZXJJZFJlcXVlc3Q="
       },
       "data": "eyJjdXN0b21lcklkIjoiODA2YjY5MDctYTMyYi00YmVhLTg2YjktY2FlMThiMjM2MjFiIiwibm90aWZpY2F0aW9uQ2hhbm5lbHMiOlsiU01TIiwiUFVTSF9OT1RJRklDQVRJT04iXSwibWVzc2FnZUlkIjoidG9wdXAtd2l0aC1jYXJkIiwibWVzc2FnZVRlbXBsYXRlRGF0YSI6eyJBY2NvdW50IjoiU0FSIEFjY291bnQiLCJBbW91bnQiOiI1MDAwMDAiLCJDYXJkTnVtYmVyIjoiNTIwMDAwKioqKioqMDAwNyIsIkN1cnJlbmN5IjoiNjgyIiwiVGltZSI6IjIwMjItMDMtMzFUMjE6MzA6MjBaIn19"
      }
     ]
    },
    "scheduleToCloseTimeout": "10s",
    "scheduleToStartTimeout": "10s",
    "startToCloseTimeout": "10s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "16",
    "retryPolicy": {
     "initialInterval": "1s",
     "backoffCoefficient": 2,
     "maximumInterval": "300s",
     "maximumAttempts": 300
    }
   }
  },
  {
   "eventId": "19",
   "eventTime": "2022-03-31T21:30:21.665093650Z",
   "eventType": "MarkerRecorded",
   "taskId": "134381656",
   "markerRecordedEventAttributes": {
    "markerName": "LocalActivity",
    "details": {
     "data": {
      "payloads": [
       {
        "metadata": {
         "encoding": "anNvbi9wbGFpbg=="
        },
        "data": "eyJBY3Rpdml0eUlEIjoiMiIsIkFjdGl2aXR5VHlwZSI6IkJ1aWxkUG9zdEpvdXJuYWxFbnRyeVJlcXVlc3QiLCJSZXBsYXlUaW1lIjoiMjAyMi0wMy0zMVQyMTozMDoyMS41NzA2ODk2NjFaIiwiQXR0ZW1wdCI6MSwiQmFja29mZiI6MH0="
       }
      ]
     },
     "result": {
      "payloads": [
       {
        "metadata": {
         "encoding": "anNvbi9wcm90b2J1Zg==",
         "messageType": "Y2lnYXIuYm9va3MudjEuUG9zdEpvdXJuYWxFbnRyeVJlcXVlc3Q="
        },
        "data": "eyJzaGVsZkNvZGUiOiJUd2VlcV9Db0EiLCJyZWZlcmVuY2UiOiIwYWRlZWIwYi1iYTQ3LTQ1NzktYjc1ZC1jN2FlN2IwZGMxNTkiLCJ0eXBlQ29kZSI6Ik5PT05fVE9QVVBfTUFEQV9DQVJEIiwiZGV0YWlscyI6ImV5SnBaQ0k2SWpCaFpHVmxZakJpTFdKaE5EY3RORFUzT1MxaU56VmtMV00zWVdVM1lqQmtZekUxT1NJc0ltRmpZMjkxYm5SSlpDSTZJalppWWpRd1pqVm1MV000TXpVdE5HTmxOaTFoTkdFNExXRTROVGt4TkdJNU9HRTJaaUlzSW5KbFptVnlaVzVqWlNJNklqWmlZMlUzTnpJMUxUTmtNak10TkRneU5TMWlZV1U1TFdVNFlqWTBOekk1WVRsbFpDSXNJbVJ5UTNJaU9pSkRVa1ZFU1ZRaUxDSjBlWEJsSWpvaVRrOVBUbDlVVDFCVlVGOU5RVVJCWDBOQlVrUWlMQ0poYlc5MWJuUWlPaUkxTURBd01EQWlMQ0p3YjNOMFpXUkJkQ0k2SWpJd01qSXRNRE10TXpGVU1qRTZNekE2TWpBdU1UTTRORE13V2lJc0luTjBZWFIxY3lJNklsQlBVMVJGUkNJc0ltTnlaV0YwWldSQmRDSTZJakl3TWpJdE1ETXRNekZVTWpFNk16QTZNakF1TVRNNE5ETXdXaUlzSW5Wd1pHRjBaV1JCZENJNklqSXdNakl0TURNdE16RlVNakU2TXpBNk1qQXVNalU0TmpNMVdpSXNJbVJsZEdGcGJITWlPbnNpWTJGeVpGOXVkVzFpWlhJaU9pSTFNakF3TURBcUtpb3FLaW93TURBM0lpd2ljR0Y1YldWdWRGOXBaQ0k2SWpkalkyWTBabUUzTFRobFlqa3RORFZoWVMwNE5USmhMVFE1TVRFeE9URm1aREprTnlJc0luQmhlVzFsYm5SZmJXVjBhRzlrWDJsa0lqb2lOV1JpWkRGbVltSXRaREUyTnkwME1qUTRMVGt3TXpJdE1Ua3hNbVV5TXpCak5HSTBJaXdpY0dGNWJXVnVkRjkwZUY5cFpDSTZJalppWTJVM056STFMVE5rTWpNdE5EZ3lOUzFpWVdVNUxXVTRZalkwTnpJNVlUbGxaQ0o5ZlE9PSIsImJvb2tFbnRyaWVzIjpbeyJib29rQ29kZSI6IjEwMDEwMzA0MDEiLCJhbW91bnQiOiI1MDAwMDAiLCJkckNyIjoiREVCSVQifSx7ImJvb2tDb2RlIjoiMjAwMTAxMDEwMSIsImFtb3VudCI6IjUwMDAwMCIsImRyQ3IiOiJDUkVESVQifV19"
       }
      ]
     }
    },
    "workflowTaskCompletedEventId": "16"
   }
  },
  {
   "eventId": "20",
   "eventTime": "2022-03-31T21:30:21.665104410Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "134381657",
   "activityTaskScheduledEventAttributes": {
    "activityId": "20",
    "activityType": {
     "name": "PostJournalEntry"
    },
    "taskQueue": {
     "name": "ACCOUNTS_WORKER_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "Y2lnYXIuYm9va3MudjEuUG9zdEpvdXJuYWxFbnRyeVJlcXVlc3Q="
       },
       "data": "eyJzaGVsZkNvZGUiOiJUd2VlcV9Db0EiLCJyZWZlcmVuY2UiOiIwYWRlZWIwYi1iYTQ3LTQ1NzktYjc1ZC1jN2FlN2IwZGMxNTkiLCJ0eXBlQ29kZSI6Ik5PT05fVE9QVVBfTUFEQV9DQVJEIiwiZGV0YWlscyI6ImV5SnBaQ0k2SWpCaFpHVmxZakJpTFdKaE5EY3RORFUzT1MxaU56VmtMV00zWVdVM1lqQmtZekUxT1NJc0ltRmpZMjkxYm5SSlpDSTZJalppWWpRd1pqVm1MV000TXpVdE5HTmxOaTFoTkdFNExXRTROVGt4TkdJNU9HRTJaaUlzSW5KbFptVnlaVzVqWlNJNklqWmlZMlUzTnpJMUxUTmtNak10TkRneU5TMWlZV1U1TFdVNFlqWTBOekk1WVRsbFpDSXNJbVJ5UTNJaU9pSkRVa1ZFU1ZRaUxDSjBlWEJsSWpvaVRrOVBUbDlVVDFCVlVGOU5RVVJCWDBOQlVrUWlMQ0poYlc5MWJuUWlPaUkxTURBd01EQWlMQ0p3YjNOMFpXUkJkQ0k2SWpJd01qSXRNRE10TXpGVU1qRTZNekE2TWpBdU1UTTRORE13V2lJc0luTjBZWFIxY3lJNklsQlBVMVJGUkNJc0ltTnlaV0YwWldSQmRDSTZJakl3TWpJdE1ETXRNekZVTWpFNk16QTZNakF1TVRNNE5ETXdXaUlzSW5Wd1pHRjBaV1JCZENJNklqSXdNakl0TURNdE16RlVNakU2TXpBNk1qQXVNalU0TmpNMVdpSXNJbVJsZEdGcGJITWlPbnNpWTJGeVpGOXVkVzFpWlhJaU9pSTFNakF3TURBcUtpb3FLaW93TURBM0lpd2ljR0Y1YldWdWRGOXBaQ0k2SWpkalkyWTBabUUzTFRobFlqa3RORFZoWVMwNE5USmhMVFE1TVRFeE9URm1aREprTnlJc0luQmhlVzFsYm5SZmJXVjBhRzlrWDJsa0lqb2lOV1JpWkRGbVltSXRaREUyTnkwME1qUTRMVGt3TXpJdE1Ua3hNbVV5TXpCak5HSTBJaXdpY0dGNWJXVnVkRjkwZUY5cFpDSTZJalppWTJVM056STFMVE5rTWpNdE5EZ3lOUzFpWVdVNUxXVTRZalkwTnpJNVlUbGxaQ0o5ZlE9PSIsImJvb2tFbnRyaWVzIjpbeyJib29rQ29kZSI6IjEwMDEwMzA0MDEiLCJhbW91bnQiOiI1MDAwMDAiLCJkckNyIjoiREVCSVQifSx7ImJvb2tDb2RlIjoiMjAwMTAxMDEwMSIsImFtb3VudCI6IjUwMDAwMCIsImRyQ3IiOiJDUkVESVQifV19"
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "60s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "16",
    "retryPolicy": {
     "initialInterval": "1s",
     "backoffCoefficient": 2,
     "maximumInterval": "300s",
     "maximumAttempts": 300
    }
   }
  },
  {
   "eventId": "21",
   "eventTime": "2022-03-31T21:30:21.765946271Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "134381663",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "18",
    "identity": "1@notifications-worker-7cf9d8cfd7-f7m2c@",
    "requestId": "5921ff80-a881-4c13-b687-e10bbe6a16ee",
    "attempt": 1
   }
  },
  {
   "eventId": "22",
   "eventTime": "2022-03-31T21:30:21.933149779Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "134381664",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "Y2lnYXIubm90aWZpY2F0aW9ucy52MS5TZW5kTm90aWZpY2F0aW9uVG9Vc2Vyc0J5Q3VzdG9tZXJJZFJlc3BvbnNl"
       },
       "data": "e30="
      }
     ]
    },
    "scheduledEventId": "18",
    "startedEventId": "21",
    "identity": "1@notifications-worker-7cf9d8cfd7-f7m2c@"
   }
  },
  {
   "eventId": "23",
   "eventTime": "2022-03-31T21:30:21.933158335Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "134381665",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "accounts-worker-9b59f8797-cwmh8:cd09ad10-dcd1-4f32-8fce-b54e489bdd5f",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "24",
   "eventTime": "2022-03-31T21:30:21.815615361Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "134381670",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "20",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "b86e3914-9074-41fb-8269-13e4f8f80486",
    "attempt": 1
   }
  },
  {
   "eventId": "25",
   "eventTime": "2022-03-31T21:30:22.183380839Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "134381671",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "Y2lnYXIuYm9va3MudjEuUG9zdEpvdXJuYWxFbnRyeVJlc3BvbnNl"
       },
       "data": "eyJqb3VybmFsRW50cnlJZCI6IjgzNDNmMzQwLTBlNDEtNDA3Mi04NzgxLTc2N2Y4N2MzNDMxNyJ9"
      }
     ]
    },
    "scheduledEventId": "20",
    "startedEventId": "24",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@"
   }
  },
  {
   "eventId": "26",
   "eventTime": "2022-03-31T21:30:22.376511580Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "134381673",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "23",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "05765282-b435-47f8-9db7-133dd5002988"
   }
  },
  {
   "eventId": "27",
   "eventTime": "2022-03-31T21:30:22.500007041Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "134381676",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "23",
    "startedEventId": "26",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "binaryChecksum": "8a87fc12fbf7170f6b22991bfa119e88"
   }
  },
  {
   "eventId": "28",
   "eventTime": "2022-03-31T21:30:22.500076862Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "134381677",
   "activityTaskScheduledEventAttributes": {
    "activityId": "28",
    "activityType": {
     "name": "SetTransactionPostedToGL"
    },
    "taskQueue": {
     "name": "ACCOUNTS_WORKER_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "Y2lnYXIuYWNjb3VudHMudjEuU2V0VHJhbnNhY3Rpb25Qb3N0ZWRUb0dMUmVxdWVzdA=="
       },
       "data": "eyJ0cmFuc2FjdGlvbklkIjoiMGFkZWViMGItYmE0Ny00NTc5LWI3NWQtYzdhZTdiMGRjMTU5In0="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "60s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "27",
    "retryPolicy": {
     "initialInterval": "1s",
     "backoffCoefficient": 2,
     "maximumInterval": "300s",
     "maximumAttempts": 300
    }
   }
  },
  {
   "eventId": "29",
   "eventTime": "2022-03-31T21:30:22.500102600Z",
   "eventType": "MarkerRecorded",
   "taskId": "134381678",
   "markerRecordedEventAttributes": {
    "markerName": "LocalActivity",
    "details": {
     "data": {
      "payloads": [
       {
        "metadata": {
         "encoding": "anNvbi9wbGFpbg=="
        },
        "data": "eyJBY3Rpdml0eUlEIjoiMyIsIkFjdGl2aXR5VHlwZSI6IkJ1aWxkUGF5bWVudE9yZGVyUmVxdWVzdHMiLCJSZXBsYXlUaW1lIjoiMjAyMi0wMy0zMVQyMTozMDoyMi4zNzY4NzU0OTNaIiwiQXR0ZW1wdCI6MSwiQmFja29mZiI6MH0="
       }
      ]
     },
     "result": {
      "payloads": [
       {
        "metadata": {
         "encoding": "anNvbi9wbGFpbg=="
        },
        "data": "W3siZnJvbV9iYW5rX2FjY291bnRfZ2xfY29kZSI6IjEwMDEwMTAxMDMiLCJ0b19iYW5rX2FjY291bnRfZ2xfY29kZSI6IjEwMDEwMjAyMDEiLCJjdXJyZW5jeSI6IjY4MiIsImFtb3VudCI6NTAwMDAwLCJyZWZlcmVuY2UiOiIwYWRlZWIwYi1iYTQ3LTQ1NzktYjc1ZC1jN2FlN2IwZGMxNTkiLCJkZXNjcmlwdGlvbiI6Ik5PT05fVE9QVVBfTUFEQV9DQVJEIn1d"
       }
      ]
     }
    },
    "workflowTaskCompletedEventId": "27"
   }
  },
  {
   "eventId": "30",
   "eventTime": "2022-03-31T21:30:22.500113190Z",
   "eventType": "ActivityTaskScheduled",
   "taskId": "134381679",
   "activityTaskScheduledEventAttributes": {
    "activityId": "30",
    "activityType": {
     "name": "CreatePaymentOrder"
    },
    "taskQueue": {
     "name": "ACCOUNTS_WORKER_TASK_QUEUE",
     "kind": "Normal"
    },
    "header": {

    },
    "input": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "Y2lnYXIuYmFua3MucGF5bWVudHMudjEuQ3JlYXRlUGF5bWVudE9yZGVyUmVxdWVzdA=="
       },
       "data": "eyJmcm9tQmFua0FjY291bnRHbENvZGUiOiIxMDAxMDEwMTAzIiwidG9CYW5rQWNjb3VudEdsQ29kZSI6IjEwMDEwMjAyMDEiLCJjdXJyZW5jeSI6IjY4MiIsImFtb3VudCI6IjUwMDAwMCIsInJlZmVyZW5jZSI6IjBhZGVlYjBiLWJhNDctNDU3OS1iNzVkLWM3YWU3YjBkYzE1OSIsImRlc2NyaXB0aW9uIjoiTk9PTl9UT1BVUF9NQURBX0NBUkQifQ=="
      }
     ]
    },
    "scheduleToCloseTimeout": "0s",
    "scheduleToStartTimeout": "0s",
    "startToCloseTimeout": "60s",
    "heartbeatTimeout": "0s",
    "workflowTaskCompletedEventId": "27",
    "retryPolicy": {
     "initialInterval": "1s",
     "backoffCoefficient": 2,
     "maximumInterval": "300s",
     "maximumAttempts": 300
    }
   }
  },
  {
   "eventId": "31",
   "eventTime": "2022-03-31T21:30:22.641672719Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "134381686",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "28",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "d53e77af-af07-468d-9fb4-f097a66bd18d",
    "attempt": 1
   }
  },
  {
   "eventId": "32",
   "eventTime": "2022-03-31T21:30:22.830743937Z",
   "eventType": "ActivityTaskCompleted",
   "taskId": "134381687",
   "activityTaskCompletedEventAttributes": {
    "result": {
     "payloads": [
      {
       "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "Y2lnYXIuYWNjb3VudHMudjEuU2V0VHJhbnNhY3Rpb25Qb3N0ZWRUb0dMUmVzcG9uc2U="
       },
       "data": "e30="
      }
     ]
    },
    "scheduledEventId": "28",
    "startedEventId": "31",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@"
   }
  },
  {
   "eventId": "33",
   "eventTime": "2022-03-31T21:30:22.830753244Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "134381688",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "accounts-worker-9b59f8797-cwmh8:cd09ad10-dcd1-4f32-8fce-b54e489bdd5f",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "34",
   "eventTime": "2022-03-31T21:30:23.013312341Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "134381692",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "33",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "3e92e4c0-9b7d-4807-ac33-ec0e1aebfea1"
   }
  },
  {
   "eventId": "35",
   "eventTime": "2022-03-31T21:30:23.138140481Z",
   "eventType": "WorkflowTaskCompleted",
   "taskId": "134381696",
   "workflowTaskCompletedEventAttributes": {
    "scheduledEventId": "33",
    "startedEventId": "34",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "binaryChecksum": "8a87fc12fbf7170f6b22991bfa119e88"
   }
  },
  {
   "eventId": "36",
   "eventTime": "2022-04-01T21:46:05.969982746Z",
   "eventType": "ActivityTaskStarted",
   "taskId": "134743731",
   "activityTaskStartedEventAttributes": {
    "scheduledEventId": "30",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "71a0e0fb-d807-4f82-8d41-92c8dd5fa5b4",
    "attempt": 300,
    "lastFailure": {
     "message": "rpc error: code = Internal desc = rpc error: code = Internal desc = StatusCode: , ErrorCode: UNAUTHORIZED, Message: {\"fault\":{\"faultstring\":\"Access Token expired\",\"detail\":{\"errorcode\":\"keymanagement.service.access_token_expired\"}}}",
     "source": "GoSDK",
     "applicationFailureInfo": {
      "type": "Error"
     }
    }
   }
  },
  {
   "eventId": "37",
   "eventTime": "2022-04-01T21:46:06.198502471Z",
   "eventType": "ActivityTaskFailed",
   "taskId": "134743732",
   "activityTaskFailedEventAttributes": {
    "failure": {
     "message": "rpc error: code = Internal desc = rpc error: code = Internal desc = StatusCode: , ErrorCode: UNAUTHORIZED, Message: {\"fault\":{\"faultstring\":\"Access Token expired\",\"detail\":{\"errorcode\":\"keymanagement.service.access_token_expired\"}}}",
     "source": "GoSDK",
     "applicationFailureInfo": {
      "type": "Error"
     }
    },
    "scheduledEventId": "30",
    "startedEventId": "36",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "retryState": "MaximumAttemptsReached"
   }
  },
  {
   "eventId": "38",
   "eventTime": "2022-04-01T21:46:06.198512941Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "134743733",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "accounts-worker-9b59f8797-cwmh8:cd09ad10-dcd1-4f32-8fce-b54e489bdd5f",
     "kind": "Sticky"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "39",
   "eventTime": "2022-04-01T21:46:06.766318185Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "134743739",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "38",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "3dcbd5ba-aefb-4105-b122-f97b032caf88"
   }
  },
  {
   "eventId": "40",
   "eventTime": "2022-04-01T21:46:07.067434074Z",
   "eventType": "WorkflowTaskFailed",
   "taskId": "134743742",
   "workflowTaskFailedEventAttributes": {
    "scheduledEventId": "38",
    "startedEventId": "39",
    "cause": "WorkflowWorkerUnhandledFailure",
    "failure": {
     "message": "already set",
     "source": "GoSDK",
     "stackTrace": "coroutine 3 [panic]:\ngo.temporal.io/sdk/internal.(*futureImpl).SetValue(0xc000c7e708, {0x0, 0x0})\n\texternal/io_temporal_go_sdk/internal/internal_workflow.go:387 +0x4a\ncigar/accounts/workflow.ProcessTransaction.func4.1()\n\tpackages/accounts/workflow/workflow.go:331 +0x303\ncigar/accounts/workflow.ProcessTransaction.func4({0x1455958, 0xc0019d08a0})\n\tpackages/accounts/workflow/workflow.go:377 +0x1033",
     "applicationFailureInfo": {
      "type": "PanicError",
      "nonRetryable": true
     }
    },
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "binaryChecksum": "8a87fc12fbf7170f6b22991bfa119e88"
   }
  },
  {
   "eventId": "41",
   "eventTime": "2022-04-01T21:46:07.067529813Z",
   "eventType": "WorkflowTaskScheduled",
   "taskId": "134743743",
   "workflowTaskScheduledEventAttributes": {
    "taskQueue": {
     "name": "ACCOUNTS_WORKER_TASK_QUEUE",
     "kind": "Normal"
    },
    "startToCloseTimeout": "10s",
    "attempt": 1
   }
  },
  {
   "eventId": "42",
   "eventTime": "2022-04-01T21:48:20.348769580Z",
   "eventType": "WorkflowTaskStarted",
   "taskId": "134744200",
   "workflowTaskStartedEventAttributes": {
    "scheduledEventId": "41",
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "requestId": "75d99388-8b25-4bbc-b063-53ece2a6b7df"
   }
  },
  {
   "eventId": "43",
   "eventTime": "2022-04-01T21:48:21.217405896Z",
   "eventType": "WorkflowTaskFailed",
   "taskId": "134744208",
   "workflowTaskFailedEventAttributes": {
    "scheduledEventId": "41",
    "startedEventId": "42",
    "cause": "WorkflowWorkerUnhandledFailure",
    "failure": {
     "message": "already set",
     "source": "GoSDK",
     "stackTrace": "coroutine 3 [panic]:\ngo.temporal.io/sdk/internal.(*futureImpl).SetValue(0xc000e089a8, {0x0, 0x0})\n\texternal/io_temporal_go_sdk/internal/internal_workflow.go:387 +0x4a\ncigar/accounts/workflow.ProcessTransaction.func4.1()\n\tpackages/accounts/workflow/workflow.go:331 +0x303\ncigar/accounts/workflow.ProcessTransaction.func4({0x1455958, 0xc000251cb0})\n\tpackages/accounts/workflow/workflow.go:377 +0x1033",
     "applicationFailureInfo": {
      "type": "PanicError",
      "nonRetryable": true
     }
    },
    "identity": "1@accounts-worker-9b59f8797-cwmh8@",
    "binaryChecksum": "8a87fc12fbf7170f6b22991bfa119e88"
   }
  }
 ]
8

@tihomir any idea what the issue could be

9

Could you show your temporal-web deployment config? Weird that you can run both list and show tctl commands and they return results, but web ui does not show your workflow histories.
Does this happen for all executions?

@Ruslan wdyt?

10

@tihomir @Ruslan

Here is the deployment manifest for the web-deployment I didn’t change anything

apiVersion: apps/v1
kind: Deployment
metadata:
  name: temporal-web
  labels:
    app.kubernetes.io/name: temporal
    app.kubernetes.io/instance: temporal
    app.kubernetes.io/component: web
    app.kubernetes.io/part-of: temporal
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: temporal
      app.kubernetes.io/instance: temporal
      app.kubernetes.io/component: web
  template:
    metadata:
      labels:
        app.kubernetes.io/name: temporal
        app.kubernetes.io/instance: temporal
        app.kubernetes.io/component: web
        app.kubernetes.io/part-of: temporal
    spec:
      serviceAccountName: temporal-web
      containers:
        - name: temporal-web
          image: temporal/web:1.14.0
          imagePullPolicy: IfNotPresent
          env:
            - name: TEMPORAL_GRPC_ENDPOINT
              value: "temporal-frontend:7233"
          ports:
            - name: http
              containerPort: 8088
              protocol: TCP

Yes, it’s happening for all workflows.

By the way our deployment on k8s, we running a single instance from each temporal server component (frontend, matching, history, and worker)

And here is the dynamic config mounted to all of them

frontend.enableClientVersionCheck:
  - value: true
    constraints: {}
history.persistenceMaxQPS:
  - value: 3000
    constraints: {}
frontend.persistenceMaxQPS:
  - value: 3000
    constraints: {}
frontend.historyMgrNumConns:
  - value: 10
    constraints: {}
frontend.sendRawWorkflowHistory:
  - value: true
    constraints: {}
frontend.throttledLogRPS:
  - value: 20
    constraints: {}
history.historyMgrNumConns:
  - value: 50
    constraints: {}
system.advancedVisibilityWritingMode:
  - value: "on"
    constraints: {}
system.enableWriteToSecondaryAdvancedVisibility:
  - value: true
    constraints: {}
system.enableReadFromSecondaryAdvancedVisibility:
  - value: true
    constraints: {}
history.defaultActivityRetryPolicy:
  - value:
      InitialIntervalInSeconds: 1
      MaximumIntervalCoefficient: 100.0
      BackoffCoefficient: 2.0
      MaximumAttempts: 0
history.defaultWorkflowRetryPolicy:
  - value:
      InitialIntervalInSeconds: 1
      MaximumIntervalCoefficient: 100.0
      BackoffCoefficient: 2.0
      MaximumAttempts: 0
11

what does the network tab show for the History request? Especially the Headers and Response/Preview Sub-tabs

image
image1071×677 86.3 KB

12

Hello @Ruslan

Here is the cURL dump from the browser

curl 'http://localhost:8088/api/namespaces/cigar/workflows/remediate-process-transaction-trigger-workflow/0a2f0be3-0a06-4512-892d-38206948e507/history?waitForNewEvent=true' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:98.0) Gecko/20100101 Firefox/98.0' -H 'Accept: */*' -H 'Accept-Language: en-GB,en;q=0.5' -H 'Accept-Encoding: gzip, deflate' -H 'Referer: http://localhost:8088/namespaces/cigar/workflows/remediate-process-transaction-trigger-workflow/0a2f0be3-0a06-4512-892d-38206948e507/summary' -H 'Accepts: application/json' -H 'Connection: keep-alive' -H 'Cookie: intercom-id-nptsh54m=5341fc2d-ce86-49d9-872a-85e258aca8e9; _ga=GA1.1.853791304.1645729095; grafana_session=5d227bb107dfba4ef42f80340c92c92f; csrf-token=opZI8Z96-R1iZAd-BkV2-Wnxivfhb5LsXB1M; csrf-token.sig=HKX0YeDjALXrB8XunzvE3c1PuAU; koa.sess=eyJzZWNyZXQiOiJpRklROWoxWmlyOE1Wc1pxc2tqamc1b0siLCJfZXhwaXJlIjoxNjQ5NDAyNDk4OTk0LCJfbWF4QWdlIjo4NjQwMDAwMH0=; koa.sess.sig=YquLsKxyTCXRdvolVxUDV_REHiY' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin'

and here is the response headers and response

HTTP/1.1 200 OK
Vary: Accept-Encoding
Content-Type: application/json; charset=utf-8
Content-Encoding: gzip
Date: Thu, 07 Apr 2022 07:26:08 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked

{"rawHistory":[{"encodingType":"ENCODING_TYPE_PROTO3","data":"\n�\u0010\b\u0001\u0012\f\bΣ��\u0006\u0010���\u0002\u0018\u0001(���H2�\u0010\n$\n\"RemediateProcessTransactionTrigger*\u001e\n\u001aACCOUNTS_WORKER_TASK_QUEUE\u0010\u0001:\u0000B\u0000J\u0002\b\nR$5d84e7f2-f7c4-45a3-81d0-7c3156e94044X\u0003j�\u0001\n�\u0001\n\u0016\n\bencoding\u0012\njson/plain\u0012o{\"TotalOfTransactionsToRemediate\":0,\"TotalOfSuccess\":0,\"TotalOfFailure\":0,\"TransactionIDsFailedToRemediate\":[]}r$0a2f0be3-0a06-4512-892d-38206948e507�\u0001$aefcdea3-8b0e-4ac3-9e53-d21be9b91b07�\u0001\u0001�\u0001\t* * * * *�\u0001\u0002\b:�\u0001�\r\nh\n be8b48516b77f5eaab96975bf57d3086\u0012$aefcdea3-8b0e-4ac3-9e53-d21be9b91b07\u0018\u0004\"\f\b�ؘ�\u0006\u0010���\u0001*\f\b��\u0006\u0010����\u00010\u0001\nh\n 64e5e137a052f086930508eb936dc342\u0012$b5c09223-b989-4f28-808f-a6bfa37e3cca\u0018\u0004\"\f\b����\u0006\u0010���\u0001*\f\b����\u0006\u0010����\u00020\u0001\ng\n cd3f80659095ecbe7d4d21768d4fa908\u0012$7fbd6d18-33d7-48c3-9369-005f7d653967\u0018\u0004\"\u000b\b�ù�\u0006\u0010���V*\f\b��ג\u0006\u0010���\u00030\u0001\ng\n fbe5fa40987ff63756bf3f3f6d8328a2\u0012$b9188d49-1794-4c22-be42-54cc50593ae8\u0018\u0004\"\u000b\b��̑\u0006\u0010˖�s*\f\b���\u0006\u0010���\u00030\u0001\ng\n 1ca82fc5a28c7a5f621b30db8e18099f\u0012$5a36817a-8988-4dc0-b188-ef909b15b454\u0018\u0004\"\f\b��̑\u0006\u0010��غ\u0001*\u000b\b��\u0006\u0010���L0\u0001\ng\n 019bcb03fe9c5bcf0dd705ef707e69fd\u0012$e80470ac-5613-4773-a03e-b69ab0fc0e4d\u0018\u0004\"\u000b\b��ۑ\u0006\u0010��G*\f\b����\u0006\u0010޵��\u00010\u0001\ng\n 62cfcbfe707d12068deaa889b7112429\u0012$41e32315-9ee2-4a7e-880f-857ebe95893d\u0018\u0004\"\u000b\b��ݑ\u0006\u0010���/*\f\b����\u0006\u0010��׽\u00020\u0001\ng\n 21aa030f272bc4a59d5d9e5eb5cd9a91\u0012$931568a4-f49d-4c61-a73b-23365d900873\u0018\u0004\"\u000b\b���\u0006\u0010���\u0018*\f\b����\u0006\u0010����\u00020\u0001\nf\n d683cea7c254bde6896700fb4084dd7c\u0012$8dc41ce0-cd63-417a-8181-4f5ea355e2aa\u0018\u0004\"\u000b\b���\u0006\u0010Ǘ�y*\u000b\b����\u0006\u0010���)0\u0001\ng\n 6c636322e5bc888c74aec360908b7203\u0012$2a6450f3-5cfc-4fa1-9127-ec4e31ab9ad8\u0018\u0004\"\u000b\b���\u0006\u0010���i*\f\b����\u0006\u0010�\u00020\u0001\ng\n 36b7e9400ff59346e048969aabe2cde8\u0012$cffc98c7-6a94-4fec-a253-36de77d037a8\u0018\u0004\"\u000b\b����\u0006\u0010���\u0019*\f\b�Ġ�\u0006\u0010���\u00020\u0001\nh\n 0561afc9b7c65a6c74732c6a0034ede0\u0012$801410d3-e301-497f-a569-275863fa6075\u0018\u0004\"\f\b����\u0006\u0010�߃�\u0001*\f\b�®�\u0006\u0010���\u00030\u0001\ng\n 9c602d1967e298d9a88c8e29d13f9b53\u0012$b02e5c6d-c26a-4ca9-b940-cf5520dd761b\u0018\u0004\"\u000b\b�Ȑ�\u0006\u0010��G*\f\b�⮓\u0006\u0010����\u00010\u0001\ng\n 8a87fc12fbf7170f6b22991bfa119e88\u0012$e6eaf4b5-efc3-4893-8cba-88c8858a42b1\u0018\u0004\"\u000b\b����\u0006\u0010���8*\f\b����\u0006\u0010�驁\u00030\u0001\ng\n 14f11e32a0ca3e00b84b36e803595c0c\u0012$4a3f7f4e-c2c1-4ab8-afb1-9723746d16da\u0018\u0004\"\f\b����\u0006\u0010�ʝ�\u0001*\u000b\b����\u0006\u0010��+0\u0001\nh\n 3e21ef7ab674a1378c1c1a30707f4903\u0012$7ea938b2-57aa-4362-ae40-1c9c5ddfb6f9\u0018\u0004\"\f\b�\u0006\u0010�֠�\u0001*\f\b��ϓ\u0006\u0010�ՙ�\u00010\u0001�\u0001\u0000","data_base64":"CpoQCAESDAjOo7qSBhCQ8PO1AhgBKLnB5kgygBAKJAoiUmVtZWRpYXRlUHJvY2Vzc1RyYW5zYWN0aW9uVHJpZ2dlcioeChpBQ0NPVU5UU19XT1JLRVJfVEFTS19RVUVVRRABOgBCAEoCCApSJDVkODRlN2YyLWY3YzQtNDVhMy04MWQwLTdjMzE1NmU5NDA0NFgDaowBCokBChYKCGVuY29kaW5nEgpqc29uL3BsYWluEm97IlRvdGFsT2ZUcmFuc2FjdGlvbnNUb1JlbWVkaWF0ZSI6MCwiVG90YWxPZlN1Y2Nlc3MiOjAsIlRvdGFsT2ZGYWlsdXJlIjowLCJUcmFuc2FjdGlvbklEc0ZhaWxlZFRvUmVtZWRpYXRlIjpbXX1yJDBhMmYwYmUzLTBhMDYtNDUxMi04OTJkLTM4MjA2OTQ4ZTUwN4IBJGFlZmNkZWEzLThiMGUtNGFjMy05ZTUzLWQyMWJlOWI5MWIwN5ABAaIBCSogKiAqICogKqoBAgg6wgGTDQpoCiBiZThiNDg1MTZiNzdmNWVhYWI5Njk3NWJmNTdkMzA4NhIkYWVmY2RlYTMtOGIwZS00YWMzLTllNTMtZDIxYmU5YjkxYjA3GAQiDAiB2JiRBhDXy/SJASoMCIXytpIGEPPx7PkBMAEKaAogNjRlNWUxMzdhMDUyZjA4NjkzMDUwOGViOTM2ZGMzNDISJGI1YzA5MjIzLWI5ODktNGYyOC04MDhmLWE2YmZhMzdlM2NjYRgEIgwIgPmikQYQrOaU1wEqDAiBk8GSBhC5j6jWAjABCmcKIGNkM2Y4MDY1OTA5NWVjYmU3ZDRkMjE3NjhkNGZhOTA4EiQ3ZmJkNmQxOC0zM2Q3LTQ4YzMtOTM2OS0wMDVmN2Q2NTM5NjcYBCILCJHDuZEGEJH6yFYqDAiR3deSBhCntPSKAzABCmcKIGZiZTVmYTQwOTg3ZmY2Mzc1NmJmM2YzZjZkODMyOGEyEiRiOTE4OGQ0OS0xNzk0LTRjMjItYmU0Mi01NGNjNTA1OTNhZTgYBCILCIanzJEGEMuWynMqDAiGweqSBhCG9eOQAzABCmcKIDFjYTgyZmM1YTI4YzdhNWY2MjFiMzBkYjhlMTgwOTlmEiQ1YTM2ODE3YS04OTg4LTRkYzAtYjE4OC1lZjkwOWIxNWI0NTQYBCIMCPHtzJEGENjt2LoBKgsI8ofrkgYQy8jlTDABCmcKIDAxOWJjYjAzZmU5YzViY2YwZGQ3MDVlZjcwN2U2OWZkEiRlODA0NzBhYy01NjEzLTQ3NzMtYTAzZS1iNjlhYjBmYzBlNGQYBCILCJXw25EGEJ/plEcqDAiWivqSBhDeta3gATABCmcKIDYyY2ZjYmZlNzA3ZDEyMDY4ZGVhYTg4OWI3MTEyNDI5EiQ0MWUzMjMxNS05ZWUyLTRhN2UtODgwZi04NTdlYmU5NTg5M2QYBCILCJ+53ZEGEOvQ9y8qDAif0/uSBhD/3de9AjABCmcKIDIxYWEwMzBmMjcyYmM0YTU5ZDVkOWU1ZWI1Y2Q5YTkxEiQ5MzE1NjhhNC1mNDlkLTRjNjEtYTczYi0yMzM2NWQ5MDA4NzMYBCILCIn/55EGELz9lxgqDAiJmYaTBhCK8//TAjABCmYKIGQ2ODNjZWE3YzI1NGJkZTY4OTY3MDBmYjQwODRkZDdjEiQ4ZGM0MWNlMC1jZDYzLTQxN2EtODE4MS00ZjVlYTM1NWUyYWEYBCILCIjn65EGEMeXq3kqCwiJgYqTBhCDjv4pMAEKZwogNmM2MzYzMjJlNWJjODg4Yzc0YWVjMzYwOTA4YjcyMDMSJDJhNjQ1MGYzLTVjZmMtNGZhMS05MTI3LWVjNGUzMWFiOWFkOBgEIgsI+4DzkQYQ9NLmaSoMCPuakZMGEO6esJYCMAEKZwogMzZiN2U5NDAwZmY1OTM0NmUwNDg5NjlhYWJlMmNkZTgSJGNmZmM5OGM3LTZhOTQtNGZlYy1hMjUzLTM2ZGU3N2QwMzdhOBgEIgsIlaqCkgYQ/qqHGSoMCJXEoJMGEN/x8rwCMAEKaAogMDU2MWFmYzliN2M2NWE2Yzc0NzMyYzZhMDAzNGVkZTASJDgwMTQxMGQzLWUzMDEtNDk3Zi1hNTY5LTI3NTg2M2ZhNjA3NRgEIgwI/6iQkgYQ5t+D7wEqDAj/wq6TBhCWo/K6AzABCmcKIDljNjAyZDE5NjdlMjk4ZDlhODhjOGUyOWQxM2Y5YjUzEiRiMDJlNWM2ZC1jMjZhLTRjYTktYjk0MC1jZjU1MjBkZDc2MWIYBCILCIzIkJIGEOWu5kcqDAiY4q6TBhCwhbm7ATABCmcKIDhhODdmYzEyZmJmNzE3MGY2YjIyOTkxYmZhMTE5ZTg4EiRlNmVhZjRiNS1lZmMzLTQ4OTMtOGNiYS04OGM4ODU4YTQyYjEYBCILCOCAk5IGEJ/w6TgqDAjgmrGTBhCj6amBAzABCmcKIDE0ZjExZTMyYTBjYTNlMDBiODRiMzZlODAzNTk1YzBjEiQ0YTNmN2Y0ZS1jMmMxLTRhYjgtYWZiMS05NzIzNzQ2ZDE2ZGEYBCIMCIX/n5IGEOzKnZcBKgsIhpm+kwYQ/uesKzABCmgKIDNlMjFlZjdhYjY3NGExMzc4YzFjMWEzMDcwN2Y0OTAzEiQ3ZWE5MzhiMi01N2FhLTQzNjItYWU0MC0xYzljNWRkZmI2ZjkYBCIMCKDusJIGEJzWoNUBKgwIpIjPkwYQ/9WZ6AEwAcoBAA=="},{"encodingType":"ENCODING_TYPE_PROTO3","data":"\n?\b\u0002\u0012\f\b����\u0006\u0010����\u0002\u0018\u0005(���HR&\n\u001e\n\u001aACCOUNTS_WORKER_TASK_QUEUE\u0010\u0001\u0012\u0002\b\n\u0018\u0001","data_base64":"Cj8IAhIMCIikupIGEJvd8u4CGAUo4cPmSFImCh4KGkFDQ09VTlRTX1dPUktFUl9UQVNLX1FVRVVFEAESAggKGAE="}],"history":{"events":[]},"nextPageToken":"","archived":false}
13

hmm so curl returned empty events history array

Aso for fair comparison, seems tctl workflow describe and curl were executed against different workflow runs as the workflow IDs are different?
For tctl example the workflow id was process-transaction-workflow-0adeeb0b-ba47-4579-b75d-c7ae7b0dc159
for web - remediate-process-transaction-trigger-workflow

could you execute this for the same workflow run as the curl example

tctl workflow show -wid remediate-process-transaction-trigger-workflow -rid  0a2f0be3-0a06-4512-892d-38206948e507

tctl workflow describe-wid remediate-process-transaction-trigger-workflow -rid  0a2f0be3-0a06-4512-892d-38206948e507

This will help to understand whether the empty array comes from temporal server and the web

14

Hello @Ruslan

apologies for the inconvenient.
Here is the result of the first command:

   1  WorkflowExecutionStarted    {WorkflowType:{Name:RemediateProcessTransactionTrigger}, ParentInitiatedEventId:0,                                                       
                                  TaskQueue:{Name:ACCOUNTS_WORKER_TASK_QUEUE, Kind:Normal}, WorkflowExecutionTimeout:0s, WorkflowRunTimeout:0s,                            
                                  WorkflowTaskTimeout:10s, ContinuedExecutionRunId:5d84e7f2-f7c4-45a3-81d0-7c3156e94044, Initiator:CronSchedule,                           
                                  LastCompletionResult:[{"TotalOfTransactionsToRemediate":0,"TotalOfSuccess":0,"TotalOfFailure":0,"TransactionIDsFailedToRemediate":[]}],  
                                  OriginalExecutionRunId:0a2f0be3-0a06-4512-892d-38206948e507, FirstExecutionRunId:aefcdea3-8b0e-4ac3-9e53-d21be9b91b07, Attempt:1,        
                                  CronSchedule:* * * * *, FirstWorkflowTaskBackoff:58s, PrevAutoResetPoints:{Points:[{BinaryChecksum:be8b48516b77f5eaab96975bf57d3086,     
                                  RunId:aefcdea3-8b0e-4ac3-9e53-d21be9b91b07, FirstWorkflowTaskCompletedId:4, CreateTime:2022-03-07 16:00:01.289220055 +0000 UTC,          
                                  ExpireTime:2022-04-06 16:00:05.523974899 +0000 UTC, Resettable:true},...15 more]}}                                                       
   2  WorkflowTaskScheduled       {TaskQueue:{Name:ACCOUNTS_WORKER_TASK_QUEUE,                                                                                             
                                  Kind:Normal}, StartToCloseTimeout:10s,                                                                                                   
                                  Attempt:1}                                                                                                                               
   3  WorkflowTaskStarted         {ScheduledEventId:2,                                                                                                                     
                                  Identity:1@accounts-worker-64fc68bd89-vx2mt@,                                                                                            
                                  RequestId:edcc4b7b-238f-4b39-a845-d04b1cc618fa}                                                                                          
   4  WorkflowTaskCompleted       {ScheduledEventId:2, StartedEventId:3,                                                                                                   
                                  Identity:1@accounts-worker-64fc68bd89-vx2mt@,                                                                                            
                                  BinaryChecksum:3e21ef7ab674a1378c1c1a30707f4903}                                                                                         
   5  ActivityTaskScheduled       {ActivityId:5,                                                                                                                           
                                  ActivityType:{Name:GetTransactionsNeedTriggeringProcessTxWorkflow},                                                                      
                                  TaskQueue:{Name:ACCOUNTS_WORKER_TASK_QUEUE, Kind:Normal},                                                                                
                                  Input:[{}], ScheduleToCloseTimeout:10s,                                                                                                  
                                  ScheduleToStartTimeout:10s, StartToCloseTimeout:10s,                                                                                     
                                  HeartbeatTimeout:0s, WorkflowTaskCompletedEventId:4,                                                                                     
                                  RetryPolicy:{InitialInterval:1s, BackoffCoefficient:2,                                                                                   
                                  MaximumInterval:1m40s, MaximumAttempts:10,                                                                                               
                                  NonRetryableErrorTypes:[]}}                                                                                                              
   6  ActivityTaskStarted         {ScheduledEventId:5,                                                                                                                     
                                  Identity:1@accounts-worker-64fc68bd89-vx2mt@,                                                                                            
                                  RequestId:db7e6b36-8bbd-4f24-b2a6-ef66f7a2cf58,                                                                                          
                                  Attempt:1}                                                                                                                               
   7  ActivityTaskCompleted       {Result:[{}],                                                                                                                            
                                  ScheduledEventId:5, StartedEventId:6,                                                                                                    
                                  Identity:1@accounts-worker-64fc68bd89-vx2mt@}                                                                                            
   8  WorkflowTaskScheduled       {TaskQueue:{Name:accounts-worker-64fc68bd89-vx2mt:7db86d5e-0556-4df0-a3ef-64cdb8ddcfd2,                                                  
                                  Kind:Sticky}, StartToCloseTimeout:10s, Attempt:1}                                                                                        
   9  WorkflowTaskStarted         {ScheduledEventId:8,                                                                                                                     
                                  Identity:1@accounts-worker-64fc68bd89-vx2mt@,                                                                                            
                                  RequestId:e97e988f-25a0-458b-ad1f-2c972396df1c}                                                                                          
  10  WorkflowTaskCompleted       {ScheduledEventId:8, StartedEventId:9,                                                                                                   
                                  Identity:1@accounts-worker-64fc68bd89-vx2mt@,                                                                                            
                                  BinaryChecksum:3e21ef7ab674a1378c1c1a30707f4903}                                                                                         
  11  WorkflowExecutionCompleted  {Result:[{"TotalOfTransactionsToRemediate":0,"TotalOfSuccess":0,"TotalOfFailure":0,"TransactionIDsFailedToRemediate":[]}],               
                                  WorkflowTaskCompletedEventId:10, NewExecutionRunId:39dab373-631f-43d0-a4ad-63a3d976bb19}

And here for the second command:

{
  "executionConfig": {
    "taskQueue": {
      "name": "ACCOUNTS_WORKER_TASK_QUEUE",
      "kind": "Normal"
    },
    "workflowExecutionTimeout": "0s",
    "workflowRunTimeout": "0s",
    "defaultWorkflowTaskTimeout": "10s"
  },
  "workflowExecutionInfo": {
    "execution": {
      "workflowId": "remediate-process-transaction-trigger-workflow",
      "runId": "0a2f0be3-0a06-4512-892d-38206948e507"
    },
    "type": {
      "name": "RemediateProcessTransactionTrigger"
    },
    "startTime": "2022-04-07T07:25:02.649918480Z",
    "closeTime": "2022-04-07T07:26:12.422153870Z",
    "status": "Completed",
    "historyLength": "11",
    "memo": {

    },
    "autoResetPoints": {
      "points": [
        {
          "binaryChecksum": "be8b48516b77f5eaab96975bf57d3086",
          "runId": "aefcdea3-8b0e-4ac3-9e53-d21be9b91b07",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-07T16:00:01.289220055Z",
          "expireTime": "2022-04-06T16:00:05.523974899Z",
          "resettable": true
        },
        {
          "binaryChecksum": "64e5e137a052f086930508eb936dc342",
          "runId": "b5c09223-b989-4f28-808f-a6bfa37e3cca",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-09T14:41:04.451228460Z",
          "expireTime": "2022-04-08T14:41:05.717883321Z",
          "resettable": true
        },
        {
          "binaryChecksum": "cd3f80659095ecbe7d4d21768d4fa908",
          "runId": "7fbd6d18-33d7-48c3-9369-005f7d653967",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-13T21:26:41.181550353Z",
          "expireTime": "2022-04-12T21:26:41.828185127Z",
          "resettable": true
        },
        {
          "binaryChecksum": "fbe5fa40987ff63756bf3f3f6d8328a2",
          "runId": "b9188d49-1794-4c22-be42-54cc50593ae8",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-17T10:55:02.242387787Z",
          "expireTime": "2022-04-16T10:55:02.840497798Z",
          "resettable": true
        },
        {
          "binaryChecksum": "1ca82fc5a28c7a5f621b30db8e18099f",
          "runId": "5a36817a-8988-4dc0-b188-ef909b15b454",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-17T13:26:09.391526104Z",
          "expireTime": "2022-04-16T13:26:10.161047627Z",
          "resettable": true
        },
        {
          "binaryChecksum": "019bcb03fe9c5bcf0dd705ef707e69fd",
          "runId": "e80470ac-5613-4773-a03e-b69ab0fc0e4d",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-20T09:47:01.149238943Z",
          "expireTime": "2022-04-19T09:47:02.470506206Z",
          "resettable": true
        },
        {
          "binaryChecksum": "62cfcbfe707d12068deaa889b7112429",
          "runId": "41e32315-9ee2-4a7e-880f-857ebe95893d",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-20T16:55:59.100526187Z",
          "expireTime": "2022-04-19T16:55:59.666234623Z",
          "resettable": true
        },
        {
          "binaryChecksum": "21aa030f272bc4a59d5d9e5eb5cd9a91",
          "runId": "931568a4-f49d-4c61-a73b-23365d900873",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-22T16:55:37.050724540Z",
          "expireTime": "2022-04-21T16:55:37.713030026Z",
          "resettable": true
        },
        {
          "binaryChecksum": "d683cea7c254bde6896700fb4084dd7c",
          "runId": "8dc41ce0-cd63-417a-8181-4f5ea355e2aa",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-23T10:16:40.254462919Z",
          "expireTime": "2022-04-22T10:16:41.088049411Z",
          "resettable": true
        },
        {
          "binaryChecksum": "6c636322e5bc888c74aec360908b7203",
          "runId": "2a6450f3-5cfc-4fa1-9127-ec4e31ab9ad8",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-24T19:03:23.221882740Z",
          "expireTime": "2022-04-23T19:03:23.583798638Z",
          "resettable": true
        },
        {
          "binaryChecksum": "36b7e9400ff59346e048969aabe2cde8",
          "runId": "cffc98c7-6a94-4fec-a253-36de77d037a8",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-27T16:47:17.052548990Z",
          "expireTime": "2022-04-26T16:47:17.664582367Z",
          "resettable": true
        },
        {
          "binaryChecksum": "0561afc9b7c65a6c74732c6a0034ede0",
          "runId": "801410d3-e301-497f-a569-275863fa6075",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-30T08:27:43.501280742Z",
          "expireTime": "2022-04-29T08:27:43.928813462Z",
          "resettable": true
        },
        {
          "binaryChecksum": "9c602d1967e298d9a88c8e29d13f9b53",
          "runId": "b02e5c6d-c26a-4ca9-b940-cf5520dd761b",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-30T09:34:04.150574949Z",
          "expireTime": "2022-04-29T09:34:16.393102Z",
          "resettable": true
        },
        {
          "binaryChecksum": "8a87fc12fbf7170f6b22991bfa119e88",
          "runId": "e6eaf4b5-efc3-4893-8cba-88c8858a42b1",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-03-30T20:41:04.119175199Z",
          "expireTime": "2022-04-29T20:41:04.808088739Z",
          "resettable": true
        },
        {
          "binaryChecksum": "14f11e32a0ca3e00b84b36e803595c0c",
          "runId": "4a3f7f4e-c2c1-4ab8-afb1-9723746d16da",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-04-02T07:47:17.317154668Z",
          "expireTime": "2022-05-02T07:47:18.090911742Z",
          "resettable": true
        },
        {
          "binaryChecksum": "3e21ef7ab674a1378c1c1a30707f4903",
          "runId": "7ea938b2-57aa-4362-ae40-1c9c5ddfb6f9",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-04-05T12:33:36.447228700Z",
          "expireTime": "2022-05-05T12:33:40.486959871Z",
          "resettable": true
        }
      ]
    },
    "stateTransitionCount": "8"
  }
}
15

@Ruslan Any idea

16

@Ruslan Did you get the chance to look at it !

17

@Mashail_Almuzaini
just for curiosity, do you get the same issue in Chrome? Any JS errors in dev tools by chance?
this is a pretty weird error

18

@tihomir yes it’s the same I tried from both firefox and chrome

19

@Mashail_Almuzaini thanks for the info.
Can you check the value of TEMPORAL_GRPC_ENDPOINT:

env:
            - name: TEMPORAL_GRPC_ENDPOINT
              value: "temporal-frontend:7233"

see the helm template from Temporal helm charts here. Could you also show your service config please?

20

Here is from inside the pod

echo $TEMPORAL_GRPC_ENDPOINT
temporal-frontend:7233