I want to rate limit on a specific activity which is part of a workflow by setting WorkerOptions.TaskQueueActivitiesPerSecond to 5. However, when testing with 1,000 workflows, there are instances where the number of activities per second exceeds 5. It typically ranges from 6 to 8 and occurs roughly 10% of the time.
My test setup:
-
The activity takes 1-2s to run
-
At the start of activity, a timestamp is recorded to calculate activities per second later
Am I missing something or is there anything that I could do to strictly enforce the activities per second limit?
This is actually expected, as task queue dispatch rate limiter has a “burst” factor based on number of task queue partitions.
The way you can calculate max burst is using ceiling function for example here
Default number of task queue partitions for a task queue is 4, so
burst-per-partition = ceiling(TaskQueueActivitiesPerSecond/numOfTaskQueuePartitions)
so for your use case (im assuming you didnt change default num of task queue partitions)
burst-per-partition = ceiling(5/4) = 2
This means that at time T server can dispatch up to 2 * numOfTaskQueuePartitions = 8
activity tasks, and at time T+1 it would not dispatch any so that over time it will maintain 5/s.
If you need very strict dispatch limit (so have to stay at and under 5/s)
and the throughput of your activities is not extremely high you could lower number of task queue partitions to 1 for this activity task queue via dynamic config, see sample here
this will give you burst of 5 with single partition, so your at your limit.
Alternatively set TaskQueueActivitiesPerSecond to 4 which then would give you burst
burst-per-partition = ceiling(4/4) = 1
so thats burst of 4 across default num of partitions which will stay under you 5 limit
1 Like
Thank you for the detailed answer