Workflow scan out of sync after deletion?

Hello, after deletion of workflows using DeleteWorkflowExecution call, some of the workflows seem to remain in a weird state.

The state is:

Workflow isn’t found directly in elasticsearch (see pic).
Workflow isn’t in Cassandra either (Error: Failed to get history on workflow id: 23839328-5ade-4d36-97ee-9ff155001e6g)
Workflow is found using scan query

 tctl wf scan --query 'WorkflowId="23839328-5ade-4d36-97ee-9ff155001e6d"'
       WORKFLOW TYPE       |             WORKFLOW ID              |                RUN ID                | TASK QUEUE | START TIME | EXECUTION TIME  
  ConfigDrivenSagaWorkflow | 23839328-5ade-4d36-97ee-9ff155001e6d | 5b9458b2-4bbc-41ab-9e21-f7bd0346442b | 16:56:44   | 16:56:44    

Is there some sort of cache? I don’t understand how this could be the case, where does it get the data for the deleted workflow? Is it some sort of nuance in elasticsearch?

Scan pulls data from your visibility store (seems you have advanced vis with ES configured)
What server version are you using?
If you have server metrics configured check visibility latencies:

histogram_quantile(0.95, sum(rate(task_latency_bucket{operation=~"VisibilityTask.*", service_name="history"}[1m])) by (operation, le))

1 Like

Thank you, it’s true our ES was lagging badly.