Rate limit per activity

Community Support
1

I need to implement some rate-limiting on my activities.

Ideally, I don’t want to gimp my entire workflow (using setMaxWorkerActivitiesPerSecond() for example)
because I only want to limit the activities that have outgoing HTTP requests.

My ideal solution would be that defining the maximum concurrent activities per activity class, but I can’t seem to find anything like this. What should be done in this case?

I assume the easiest way to achieve what I need is to have a separate task queue for the activities and assign the rate limit for that task queue, but since different APIs have different rate limits that would mean I need to make new task queues per API which is not a path I will willingly go down as this will need some more development and I need a quick solution.

Edit: I’ve tried separate task queues as suggested in Rate limit activities - #4 by maxim however it didn’t work as I expected.

Currently, I have the following:

  • 2 workflows that share some activities.
  • Each workflow has its own worker, which registers its own activities (so the shared activities are defined twice, in two different workers)

What I’ve tried is moving all the activities that I need to ratelimit into their own Worker, on a completely new task queue. This new worker does not register Workflow implementations, only activity implementations. This led to the workflow being unable to start the activity as “it is not registered with any worker”.

I also tried registering the workflow type in the new workers, but it seems this overrides the activities registered with the other worker. What am I missing here?

3

You need to specify the new task queue name in ActivityOptions when invoking the activity.

4

Ah, apologies. I’ve tried that and the errors are gone, however for testing I set the max to 0.1 which according to the docs specifies that we want 1 run every 10 seconds and ran 3 workflows, and the activities on the task queue seemed to be operating at normal speed.

5

For such low rates make sure to change the default number of task queue partitions to 1.

6

The 0.1 was just for me to check if the rate limiter actually works before I deploy it to production. My actual limits will be 4 limits (depending on the target project) that will range from 120 to 600.

My testing was done locally (through the default docker-compose https://github.com/temporalio/docker-compose/tree/main/dynamicconfig), which did not have anything related to the queue partitions.

7

I would recommend testing with the real cluster and rates. It is expected to work.

8

Well, I suppose worst case scenario is the same behaviour anyway. After migrating to AWS from GCP it seems that our workflows are running faster, which resulted in my project getting rate limited quite often by our other internal assets. This must work as there is no other way for me to rate-limit my outgoing http requests.

9

Sorry, I don’t understand your concern. Do you mean that Temporal task queue rate limiting doesn’t work?

10

Yes that’s exactly what I mean.

WorkerFactory factory = WorkerFactory.newInstance(client);
WorkerOptions workerOptions =
    WorkerOptions
        .newBuilder()
        .setMaxWorkerActivitiesPerSecond(0.1)
        .build();
Worker worker = factory.newWorker(HttpActivityQueues.ODOO_HTTP_TASK_QUEUE, workerOptions);

Edit: I have also tried setMaxTaskQueueActivitiesPerSecond() (since we have multiple workers) and this didn’t work either.

11

0.1 requires at most one partition. Please file a bug report if you reproduce the problem with higher rates.

12

Alrighty, like I said we don’t have anything to lose, I will be sure to file a report if it happens.
Thank you very much for your help.

13

I tried to do the same by providing a task queue for workflow and another task queue for the activity i wanted to rate limit. I ensured i gave the right task queue in workflow options and activity options.

However i observed that the rate limits are not honoured.

  1. Is the rate limit on the task queue or the task queue partition? Because i was able to see 3 tasks submitted even though i gave the max activities per task queue as 1 for the activity worker.
  2. What is the rate limit algorithm? Didn’t seem to be sliding window but something like token bucket.