Workflow scan out of sync after deletion?

Community Support
, ,
1

Hello, after deletion of workflows using DeleteWorkflowExecution call, some of the workflows seem to remain in a weird state.

The state is:

Workflow isn’t found directly in elasticsearch (see pic).
Workflow isn’t in Cassandra either (Error: Failed to get history on workflow id: 23839328-5ade-4d36-97ee-9ff155001e6g)
Workflow is found using scan query

 tctl wf scan --query 'WorkflowId="23839328-5ade-4d36-97ee-9ff155001e6d"'
       WORKFLOW TYPE       |             WORKFLOW ID              |                RUN ID                | TASK QUEUE | START TIME | EXECUTION TIME  
  ConfigDrivenSagaWorkflow | 23839328-5ade-4d36-97ee-9ff155001e6d | 5b9458b2-4bbc-41ab-9e21-f7bd0346442b | 16:56:44   | 16:56:44

Is there some sort of cache? I don’t understand how this could be the case, where does it get the data for the deleted workflow? Is it some sort of nuance in elasticsearch?

image
image1057×498 45.4 KB

2

Scan pulls data from your visibility store (seems you have advanced vis with ES configured)
What server version are you using?
If you have server metrics configured check visibility latencies:

histogram_quantile(0.95, sum(rate(task_latency_bucket{operation=~"VisibilityTask.*", service_name="history"}[1m])) by (operation, le))

1 Like
3

Thank you, it’s true our ES was lagging badly.