Tracking Schedule buffered Workflow Executions

Community Support
1

I triggered a large backfill in a Temporal Schedule that would result in a few thousand Workflow Executions. The ScheduleOverlapPolicy for the backfill was set to BUFFER_ALL so as not to overwhelm the workers with all the backfill executions and potentially starving other workflows.

However, once the backfill was triggered, I had no way to know whether it was still on-going, except by tracking the Workflow Executions as they came up. My question is: is there a way to keep track of buffered Workflow Executions to avoid double-triggering a backfill and measure backfill progress?

Thanks!

2

Just wanted to add that the large BUFFER_ALL backfill apparently stopped at some point. Not entirely sure if there’s a cap on the amount of executions that can be buffered or what else could explain that.

3

Hi @tomasfarias,

Are you running Temporal with OSS or Cloud? For Cloud, please file a support ticket. For OSS, this forum is definitely the right place to get support.

backfill apparently stopped at some point

What number do you get when you count the workflows executed by the backfill? e.g.

temporal workflow count --query 'TemporalScheduledById ="test-schedule-backfill"'

Assuming you are running self-hosted, are you able to see worker logs with “Buffer too large”? If so you may have hit the max buffer limit here.

4

Hi @tao!

Thanks for reply.

Are you running Temporal with OSS or Cloud?

We are on Cloud, so I can submit a ticket.

are you able to see worker logs with “Buffer too large”? If so you may have hit the max buffer limit here.

I will have to check our worker logs, I wasn’t aware of the max buffer limit (I assumed it existed). Is this setting configurable? I am doing a search in the docs but can’t find any references.

5

Sorry, when I say worker I actually meant the server side worker. We can verify the log for you once you share the namespace with us in the ticket.

To workaround this limit, for the time being, you may have to break the backfill into smaller batches with less than 1000 executions each.