Hello,
How can I rate limit on a specific activity which is part of a workflow. Can you please share some code examples ?
For Example: Workflow has activities A, B, C, D. How can I configure to say at any instant the activity A should only be invoked twice in a span of 1 second ?
Temporal supports task queue rate limiting across multiple worker processes. If you want to rate limit a specific activity then make sure that it listens on its own task queue by creating a separate worker object instance. Then set WorkerOptions.MaxTaskQueueActivitiesPerSecond to 2 to get 2 activities per second rate limit.
Thanks @maxim How can I unit test to make sure the number of activities invoked are indeed the specified count ? Is there a junit support or any other way to validate this programmatically or manually ?
You have to test it against the real service (docker-compose one is OK). I don’t think unit testing work for this scenario.
@maxim we have a requirement for rate-limiting activities on an hourly basis. Also, we can’t assume equal workload distribution on the activities throughout the hour. Hence, setting the rate limit per second will not work in such cases. Do you have any recommendations on how we can achieve this?
The current thought is to explore activity interceptors and probably use Redis based rate-limited impl which will reject the activity calls if the limit exceeds. However, this approach exhausts the activity attempt count. Do you have any better recommendation ?
Temporal cannot support such a requirement at this point. So using Redis makes sense.
However, this approach exhausts the activity attempt count. Do you have any better recommendation ?
The default max attempt count is unlimited. So don’t set it.
Got it. Thanks
But whats the ordering of the activity execution. If there is no specific ordering guarantee, it can lead to starvation of activities, example, everytime a specific activity is retries, it can see rate limit has reached and has to wait for the next interval. How can you avoid such starvation if we manage rate limiting with an external system like redis ?
Temporal is not the right system if you have to guarantee a global order of activities execution in the presence of a backlog.
Sure, but is there fairness to avoid starvation? What kind of fairness guarantees can I assume to get with temporal?
Currently none. But in all practical applications that run on Temporal workers are provisioned with enough capacity to avoid a backlog.
We do realize that it is a limitation and consider features to provide “fair” task queues.
Good to know, thanks for clarifying. Please let me know if there is a feature request that I can track for when something like that will make it upstream.
I don’t think Gubernator is a good fit for rate limiting activities. The Temporal task queues rate limit doesn’t even dispatch a task to a worker. In the case of Gubernator, the task is dispatched then fails and had to be retried. This adds unnecessary load and noise to the system.
We have the similar requirements as @nkatre:
For example each customer can do A at a rate of 30 operations a day, B at a rate of 1000 a day but not more than 50 per hour, etc. We don’t have control on the rate limits (as they are enforced by a third party) and we will get penalised for hitting those limits.
A workflow per customer works assuming that a rate of signals/actions per customer is not too high.