Can temporal support multi-stage pipeline?

Community Support
1

Our workflow combines with 4 different gpu tasks and executes serially.
For some reasons, a gpu task can only support one or two concurrent requests, but the others can support more.
After looking though the workflow and acitvity options, we can only set the concurrency of the workflow and the concurrency of all activities in a workflow.
Is there any way to support multi-stage pipline in temporal(like cpu multi-stage, one step can be executed only once)? Or only set the concurrency of the specific acitvity?

2

Can you give a little more info on this please? Are you looking for setting up different concurrency options per activity type?

Currently you can set this per worker, via Worker options, for example in Java SDK:

WorkerOptions.newBuilder()
              .setMaxConcurrentActivityExecutionSize(10)
              .build();
3

Setting up different concurrency options per activity type?Yes, but temporal doesn’t support this.
Should we split the worker into small worker per activity type? Do you have better idea?

4

If you need to limit parallelism or rate of a set of activities you have to use a separate worker. Do you have a specific issue with this approach?

5

The workflows ate created to process audio. Many activities would call different gpu services which are deployed in other cluster.
The Concurrency of the gpu services is not the same and very low.

On one hand, setting a global convurrency leads to many idle connections.
On the other hand, every workflow consist of 7, 8 acvitities , or even more. It’s too much rouble for us to split they into a seprarte work.

6

I see. It looks like starting multiple workers is the only option you have.

7

Why the concurrency is setting for global worker? Is there any plan to support single activity level concurrency?

8

It’s currently a worker config that applies to all activity types. Feel free to open issue in the sdk-features repo for per-activity-type setting and let’s see if it’s a feature sdk team thinks would be feasible to add.

9

Why the concurrency is setting for global worker? Is there any plan to support single activity level concurrency?

It is already supported by using a different task queue and associated worker for that specific activity type.