How can I rate limit on a specific activity which is part of a workflow. Can you please share some code examples ?
For Example: Workflow has activities A, B, C, D. How can I configure to say at any instant the activity A should only be invoked twice in a span of 1 second ?
Temporal supports task queue rate limiting across multiple worker processes. If you want to rate limit a specific activity then make sure that it listens on its own task queue by creating a separate worker object instance. Then set
WorkerOptions.MaxTaskQueueActivitiesPerSecond to 2 to get 2 activities per second rate limit.
Thanks @maxim How can I unit test to make sure the number of activities invoked are indeed the specified count ? Is there a junit support or any other way to validate this programmatically or manually ?
You have to test it against the real service (docker-compose one is OK). I don’t think unit testing work for this scenario.
@maxim we have a requirement for rate-limiting activities on an hourly basis. Also, we can’t assume equal workload distribution on the activities throughout the hour. Hence, setting the rate limit per second will not work in such cases. Do you have any recommendations on how we can achieve this?
The current thought is to explore activity interceptors and probably use Redis based rate-limited impl which will reject the activity calls if the limit exceeds. However, this approach exhausts the activity attempt count. Do you have any better recommendation ?
Temporal cannot support such a requirement at this point. So using Redis makes sense.
However, this approach exhausts the activity attempt count. Do you have any better recommendation ?
The default max attempt count is unlimited. So don’t set it.
But whats the ordering of the activity execution. If there is no specific ordering guarantee, it can lead to starvation of activities, example, everytime a specific activity is retries, it can see rate limit has reached and has to wait for the next interval. How can you avoid such starvation if we manage rate limiting with an external system like redis ?
Temporal is not the right system if you have to guarantee a global order of activities execution in the presence of a backlog.
Sure, but is there fairness to avoid starvation? What kind of fairness guarantees can I assume to get with temporal?
Currently none. But in all practical applications that run on Temporal workers are provisioned with enough capacity to avoid a backlog.
We do realize that it is a limitation and consider features to provide “fair” task queues.
Good to know, thanks for clarifying. Please let me know if there is a feature request that I can track for when something like that will make it upstream.