Hi,
I have a use case where I want to throttle the number of request into a particular “Activity”. I throttled it using a child workflow where number of concurrent runs were limited.
I observed that these executions were randomised and not FIFO. Is there a way of achieving the same?
To limit the rate of execution of an activity use task queue rate limiting. Create a separate worker on a separate task queue for that activity and specify the rate limit for that task queue through worker.Options.TaskQueueActivitiesPerSecond. This approach doesn’t guarantee FIFO.
To guarantee FIFO you have to pass all the requests through a single workflow instance. This approach doesn’t scale as a single workflow has limited throughput.
What is your use case for FIFO?
Hi @maxim ,
Sorry for the late response. We have an external dependency with throttling. We are already using temporal workflow. We introduced throttling in our external dependency by using child workflow.
However, we saw random behaviour in the child workflow execution. They were not FIFO. This resulted in a few workflow execution within seconds and few workflow wait time for more than 10 minutes.
What is the rate of activity executions for that external dependency? it it is not high then using a single workflow that schedules it is the way to go. Otherwise, I would rely on task queue rate limiting.
What is the rate of activity executions for that external dependency?
→ 3 activities concurrently to be executed for the external dependent activity.
Time taken for the external dependent activity is not large but we see few of the workflows stuck in waiting state and workflows that are registered later are executed before.
@maxim If we don’t have FIFO, and at some time their is a high load encountered for 5-6 hours, their is a possibility that workflows get’s stucked for a long time(6 hours) as their is a limit we can scale, that would be a bad User Experience. Is this assumption correct?
Yes, if your workers are not provisioned to keep up with the load and tasks stay in the task queue for 6 hours then delays can happen.
@maxim are we working on FIFO implementation for temporal, or is their any issue doing it ?
Here is the relevant feature request.