How to Limit Concurrent Workflow Instances (Workflow-Level Semaphore)

Community Support
,
1

I have a use case where I need workflows to queue (FIFO), but I’m struggling to find a Temporal-native solution that doesn’t break activity parallelism.

Our Scenario

We have a resource-intensive pointcloud analysis workflow with a distributed architecture:

  • Python workers: Handle workflow execution, split, and merge activities

  • Node.js workers (~15 distributed instances): Handle the intensive chunk analysis activities

  • Workflow splits large files into ~100 chunks

  • Uses asyncio.gather() to schedule all 100 chunk analyses in parallel to the Node.js workers

  • Python workers merge results when complete

The Problem

When multiple users request analysis simultaneously:

  • Workflow A starts → schedules 100 analyze activities to Node.js workers

  • Workflow B starts → schedules 100 analyze activities to Node.js workers

  • Workflow C starts → schedules 100 analyze activities to Node.js workers

  • Now 300 activities are competing for the same 15 distributed Node.js workers

  • Workers pick activities randomly across all workflows

  • Result: All 3 workflows complete slowly, nobody gets results quickly

What We Want

  • FIFO queue behavior: Only ONE workflow runs at a time

  • Workflow A completes fully (all 100 chunks processed by all 15 Node.js workers in parallel)

  • Then Workflow B starts and gets all 15 Node.js workers

  • Then Workflow C starts

  • Much faster individual workflow completion + better user experience

What We’ve Tried

max_concurrent_workflow_tasks=1:

  • Doesn’t prevent multiple workflow instances from starting

  • When a workflow schedules activities to Node.js workers and waits, the workflow task completes

  • Python worker sees itself as “free” and picks up the next workflow

  • Multiple workflows end up “Running” simultaneously, all competing for Node.js workers

  • Also worried this would kill activity parallelism within a single workflow

Is there a Temporal-native way to limit concurrent workflow instances of a specific workflow type (like a workflow-level semaphore)?
So that only one workflow processes at a time, while maintaining full activity parallelism within that running workflow across distributed Node.js workers.

Our architecture has Python handling orchestration and Node.js handling the heavy computation, and we want to ensure all Node.js workers focus on one workflow at a time.

Any suggestions would be greatly appreciated!

2

Take a look at sliding window approach, sample samples-python/batch_sliding_window at main · temporalio/samples-python · GitHub

Maybe bit simpler one that typically works well for smaller rate and number of invocation requests is to use Signal/SignalWithStart to queue up processing requests in workflow state, and process them one at a time (so in both cases practically use business logic to implement your sequential execution requirement)