Limit worker process to work on only one workflow, and limit workflow to be executed by only one worker

[maor_ts]

Hello.

I have a pipeline that involve filesystem, and so i need what each workflow(his tasks and activities) will be coupled to one proccess worker.

Also, the files are heavy, so to manage the storage well, i need that each worker will be working on one workflow only.

I tried to implement an architecture that there will be a public task queue and private task queue for each instance.

every instance will have 2 workers - one polling workflows from the public task queue, creating child workflow(which does the actual work) on the private task queue).

I limited the max concurrent workflow task queue to 1.

Im experiencing some weird behavior, mostly it takes really long time until the task being polled, and if i cancel a workflow, so the worker should take another one fro the task queue, its just not hapenning.

Whould like to hear tips, criticism about the architecture or everything comes in mind.

[antonio.perez]

Hi

I limited the max concurrent workflow task queue to 1.

this limit the number of workflow tasks, not the number of concurrent workflows.

Also, the files are heavy, so to manage the storage well, i need that each worker will be working on one workflow only.

I tried to implement an architecture that there will be a public task queue and private task queue for each instance.

I think your approach is in the right direction.

With worker configuration you can’t limit the number or concurrent workflows but the number of concurrent task. In your case I think you want to limit maxConcurrentActivityTasks to 1, and only in the private task queue to ensure only one activity tasks runs at the time.

You can control the number of concurrent child workflows with code, in the parent workflow.

[maxim]

Does Temporal Typescript support workflow session

I see. The main session feature is the ability to limit the number of such sequences of activities that can run in parallel. The session uses the following design and you can try to implement it yourself: Each process creates a separate worker with fixed parallelism (for example 1). Let’s call its task queue “process-lock”. Each process listens on a host specific task queue. For example, “host1”. To start the sequence of activities invoke “lock-process” activity at “process-lock” task queue. T…

[maor_ts]

Interesting, i didnt think about it this way, inwas sure it limit the amount of wofklows.

I will try to shift the creating of new workflows to activity.