Parameter `TaskQueueActivitiesPerSecond` does not work as expected

Community Support
, ,
1

I am trying to rate limit a certain activity (for example only one execution per second). For testing, all I do is print the current timestamp in that activity. I call the activity from a workflow, that does nothing else but call this activity. I made a seperate worker for that activity and the activity has it’s own task queue, which is connected to that worker. Now I set the parameter TaskQueueActivitiesPerSecond to 1, so only one activity can be executed per second. For testing I then started 20 workflows, which each just call this activity (there is a separate worker and task queue to handle the workflows).
Now I observe the following behavior. Instead of observing one execution per second, like the parameter would suggest, there are 2-10 concurrent executions, followed by a break longer than 1 second. How can I force the worker, to actually listen to the parameter and only poll one activity from the task queue per second and not exceed the set limit?

Thank you in advance for any help.

2

I’m having this issue too – is this by design, or is this a misconfiguration on my part?

3

Here’s a quick example app with the issue as well GitHub - vigneshv59/temporal-rate-limit-test. The output looks something like this:

Test activity 2023-12-07 22:42:20.938586 -0800 PST m=+13.477650834
Test activity 2023-12-07 22:42:20.953206 -0800 PST m=+13.492271209
Test activity 2023-12-07 22:42:20.965526 -0800 PST m=+13.504591042
Test activity 2023-12-07 22:42:20.994732 -0800 PST m=+13.533796667
Test activity 2023-12-07 22:42:24.946264 -0800 PST m=+17.485310709
Test activity 2023-12-07 22:42:24.963022 -0800 PST m=+17.502068626
Test activity 2023-12-07 22:42:24.995761 -0800 PST m=+17.534807417
Test activity 2023-12-07 22:42:25.021277 -0800 PST m=+17.560323834
...
4

Okay, seems like I figured out what was going on, the rate limit is set to be the rateLimit/nPartitions per task queue, so since the default is 4 partitions, it ends up sending 4 requests every 4 seconds instead of 1 request every second. Not sure if this is intended behavior or not, but setting the number of partitions to 1 fixes the issue (and I guess that’s sufficient for a low RPS anyways).

5

That indeed fixed it for me too, thanks a lot! For my cases the rates this can achieve is more than enough.

6

Task queue partitions do play a role here as mentioned.

As hypothetical example if you set MaxTaskQueueActivitiesPerSecond to 10 then
burst can be calculated as ceiling(limit / num partitions) and with default 4 task queue partitions:
ceiling(10/4) = 3 would mean burst of 3 activity tasks per task queue partition

So as approximation if lets say at second X service dispatches 12 activity tasks, X+1s would allow less than 9 to make average rate stay 10

By setting task queue partitions to 1 it would allow burst of 10 * 1.
Note this is an approximation but it shows the general idea as in over any large enough time window the service will make sure the average activity task dispatch rate will be kept to the desired config.