Hi there,
I would like to ask for some advice for running activities that call external services with rate limits.
From some of the discussions here, I see that I can make use of workflow.Sleep() together with redis to hold the rate limit. At the moment in the workflow, I am making a rate-limit check before calling the activity and running workflow.Sleep() if I should backoff. Can I call workflow.Sleep() inside the activity rather than at the workflow level or do something similar?
Because the rate limits are not static, i.e. based on external load they can change. Is the above approach the best way for dealing with dynamic throttling in temporal?
Can I call workflow.Sleep() inside the activity rather than at the workflow level or do something similar?
No, you cannot call workflow APIs from an activity. To sleep in an activity you can use time.Sleep or time.After, be aware that this will occupy your activity execution slot. Alternatively, you can let activity retry to handle it but you’ll lose the control over the timing.
Is the above approach the best way for dealing with dynamic throttling in temporal?
It depends on your requirement, the traffic pattern from upstream, and how rate limit is enforced on the external service.
The task queue level throttling - WorkerOptions.MaxTaskQueueActivitiesPerSecond, is one of the easiest ways to rate limit activities. It’s typically used together with activity specific task queue.
Thanks for your response, I believe that for MaxTaskQueueActivitiesPerSecond that would be a static setting without any ability to respond dynamically to external variations to rate limit? Ideally I’m looking for a way to limit the activities per second based on errors we get back (from a 3rd party endpoint).
For example, if I am trying to create folders for 1000 different users in cloud storage, but only 30% of users will encounter rate limits. I would like to only slow down (backoff) folder creation for those who are hitting rate limit errors. (and speed them up again once they don’t hit any rate limit errors)
The standard activity exponential retry is not really enough as ultimately, the 3rd party service provides us with a specific Retry-After header which is the amount of time we should wait before retrying. And this value can be different for each execution of the Activity.