Activity with priorities

Hello! We need to have priorities for the activity tasks. Is there any way to provide this besides the different task queues?

Priorities for what specifically? Can you provide more info?

We have activities for media processing (compression of video and audio). Some of the videos are short and simple and they should come firsts, before the long and heavy media.

Technically this can be somewhat done with the just-merged interceptor features. The ActivityInboundInterceptor can intercept activity executions and debounce or prioritize some over others. I would caution against custom queuing there as you’re still subject to MaxConcurrentActivityExecutionSize worker option.

The more ideal approach would be to just handle your own prioritization inside your activity. If, upon receipt of a long/heavy media processing job you need to defer to short/simple ones, you can.

Of course if you could use different task queues, that’d make partitioning the workers easier for scaling purposes.

It would be nice to have task queues that support priorities. But we don’t have plans to add them in the near future. Currently using multiple task queues is the only reasonable option if you need to run on a large scale.

If your rate of execution is low (one request per second or lower) then you can have a single workflow that would receive signals with media processing requests and execute activities according to whatever prioritization policy you have.

2 Likes

Setting activity with priorities is not support now.
What do this meaning in https://docs.temporal.io/docs/concepts/what-is-an-activity

  • Multiple priorities . One task queue per priority and having a worker pool per priority.

That is a list of use cases for multiple task queues. If you wanted to implement multiple priorities or otherwise different settings for different types of activities, you’d use multiple task queues.

Hello!

Even with multiple task queues and a worker-per-task queue, how do I guarantee that the highest priority task is picked first? Could you please help me with an example?

Thank you!

You cannot guarantee a certain task is picked before another in a process with multiple workers. You can limit the number of activities each worker can run concurrently though which may let one worker run more than another.

1 Like