Seeking Guidance on Temporal Application Design, scaling workflow queues as per load

Community Support
1

We are designing an application using Temporal to handle large-scale asynchronous jobs. Each job can process up to 1 million rows , and we expect 10-15 jobs per hour .

Job Prioritization & SLAs

Jobs are categorized into three priority levels:

  • High → Must complete within 1 hour
  • Medium → Must complete within 3-4 hours
  • Low → Can take up to 6 hours

Current Approach

We have created three separate queues (high , med , low ) and assigned jobs to them based on priority. Additionally, we have configured execution limits as follows:

  1. Concurrency Control ( setMaxConcurrentWorkflowTaskExecutionSize )
  • High: 20
  • Medium: 10
  • Low: 5
  1. Worker-Level Rate Limiting ( setMaxWorkerActivitiesPerSecond )
  2. Queue-Level Rate Limiting ( setMaxTaskQueueActivitiesPerSecond )

Request for Guidance

  • Is this the correct approach for handling job prioritization and rate limiting in Temporal?
  • Would there be a better way to dynamically scale processing power based on real-time load per queue rather than hardcoding concurrency limits?
  • Problem I am seeing with this approach is even if there are no high priority jobs to process and there is some load no low priority jobs queue the processing will be slow only and might under utilize the resources.

PS: Using java sdk for development.

Any insights or best practices would be greatly appreciated. Thanks!

1 Like