Temporal UI not displaying correct status for a workflow

Support
#1

I’ve been running some experimental workflows using 0.27. I saturated it with 5,000 executions, each signalling to 1 of 100 queues, with the queues processing incoming requests, and completing the originating activity.

After processing, the UI shows an inconsistent result. For one workflow, sqlserver91, the list of workflows displays sqlserver91 as “Running”. However, if you go into the workflow summary, it shows “Completed”. This is inconsistent. I tried refreshing a few times and nothing happened.

workflow-ui-inconsistency
workflow-ui-inconsistency1883×1026 149 KB

Any ideas?

Regards,

Sean

1 Like
#3

Hey @sdonovan,
The API which drives list view on the UI is powered by Temporal visibility API which provides eventual consistent semantics. As oppose to the summary page which uses highly consistent semantics.
So they can go out of sync for a short period of time. But having this out of sync for extended period of time is definitely not expected. This usually means something on the server is back logged or potential bug on the server. In general we do not recommend running any high scale scenario on using visibility backed by cassandra or mysql as they have known scalability problems. For any high scale production deployment we generally recommend elastic search backend.

Can you provide some information about the your setup:

  • Which visibility store are you using? Cassandra? MySQL? Elastic Search?
  • Can you describe your cluster setup in general?
  • In case of elastic search do you see any backlog on Kafka?
  • Are you seeing any errors on server logs during your run? Can you share those with us?
  • Can you describe your scale run?