Missing data on workflow running on Temporal Web UI

Community Support
1

Hi all

we have doing development process and testing,
sometimes we need to look at temporal-ui
several days ago we have workflow running at 2022-07-07

{“logLevel”:“debug”,“Attempt”:1,“Namespace”:“first-namespace”,“RunID”:“320955a4-b43a-4f0e-93f7-2a84dbcc844f”,“TaskQueue”:“QUEUE”,“WorkerID”:“1@wf-35-ll6jp@”,“WorkflowID”:“TPa0220707052919841e4cda0”,“WorkflowType”:“wftype”,“logTimestamp”:“2022-07-07T17:29:19+07:00”,“message”:“Workflow for ID TPa0220707052919841e4cda0 starts.”}

in the service pods it’s running, but when we check on the temporal web ui at first-namespace
it’s not showing at all
the workflow is still Running and not terminated/ completed

We integrated it with elasticsearch.
I say it’s intermittent because sometimes when a new workflow generated it still shown on the web ui.

Any suggestion, What should we check, is this expected behavior ?

image
image1952×287 19.2 KB

2

the workflow is still Running and not terminated/ completed

Can you see the workflow history
tctl wf show -w <wfid> -r <runid> --output_filename myhistory.json
Whats the last event?

What’s the status of the workflow that you see via command:
tctl wf desc -w <wfid>

Visibility is eventually consistent, Typically there is a couple of seconds delay from when workflow state is updated until its reflected in visibility records and shows on the ui.
One reason it might not show up is latencies in visibility queue. If you have server metrics set up check with Grafana:


histogram_quantile(0.95, sum(rate(task_latency_bucket{operation=~"VisibilityTask.*", service_name="history"}[1m])) by (operation, le))

this measures visibility tasks processing latency.

3

tctl wf show -w -r --output_filename myhistory.json

image
image844×298 69.9 KB

the last event is timer started

tctl wf desc -w 


{
  "executionConfig": {
    "taskQueue": {
      "name": "QUEUE",
      "kind": "Normal"
    },
    "workflowExecutionTimeout": "0s",
    "workflowRunTimeout": "0s",
    "defaultWorkflowTaskTimeout": "10s"
  },
  "workflowExecutionInfo": {
    "execution": {
      "workflowId": "TPa0220707052919841e4cda0",
      "runId": "320955a4-b43a-4f0e-93f7-2a84dbcc844f"
    },
    "type": {
      "name": "WorkflowName"
    },
    "startTime": "2022-07-07T10:29:19.869376716Z",
    "status": "Running",
    "historyLength": "116",
    "memo": {

    },
    "searchAttributes": {
      "indexedFields": {
      }
    },
    "autoResetPoints": {
      "points": [
        {
          "binaryChecksum": "70be49be3444fb42753437a9667920aa",
          "runId": "320955a4-b43a-4f0e-93f7-2a84dbcc844f",
          "firstWorkflowTaskCompletedId": "4",
          "createTime": "2022-07-07T10:29:19.890224399Z",
          "resettable": true
        }
      ]
    },
    "stateTransitionCount": "63"
  }
}

So this should be shown on web ui right because we can still query it through tctl, will try to setup the metric first, but what should we check next ?
Thank You

4

So this should be shown on web ui right

Imo yes, unless maybe you are looking at a different namespace in the UI, or ES maybe having issues.
Are you able to check ES container logs to see if there is anything that stands out? Any error logs spit out by other temporal services?