Calling task queue from multiple languages

Hello,

Is it possible to call the same task queue from different workers?

For the context, I have a worker (worker-py) defined with task queue task-queue, workflows, and activities using Python. Now, I need to make a new workflow (workflow-ts) in different language, Typescript, and I want to start my workflow-ts using the same task queue task-queue. I think my typescript is running in worker-ts. It’s two different languages, two different setup, so Typescript and Python has different workers. I am trying to figure out is it possible to call using the same task queue in multiple languages?

Thank you so much.

Even though we make them look like a single thing, Activity Task Queues and Workflow Task Queues are really two distinct things. So you may have Activity-only Workers that are polling from the Activity Task Queue my-taskqueue, and a different set of Workflow-only Workers (possibly in a different language) that are polling tasks from the Workflow Task Queue that is also named my-taskqueue.

Regarding Activity Tasks Queues… All Workers polling from a same Activity Task Queue should be able to execute any Activity task sent to that queue. That means that all Activity Workers polling from a same queue must implement the same Activities (well, thanks to retry, it may be acceptable for this not to be the case for some transition time period, notably while deploying new Workers; this should however not be the normal situation). There is technically nothing stopping you from having Activity Workers written in different languages polling from a same Task Queue, though I would strongly recommend against doing so, for practical reasons (difficulty in maintaining exactly the same API and behavior in two completely distinct implementation; harder to diagnose issues; different behavior in case of errors; etc).

Now, regarding Workflow Tasks Queues… All Workers polling from a same Workflow Task Queue should be able to execute any Workflow Task sent to that queue. Note that a single Workflow execution implies several Workflow Tasks, each making progression in the execution of that Workflow, and possibly one or several replays, in which the Worker reexecute the Workflow code in fast forward mode while comparing/filling in events from that Workflow Execution History. Now, different SDKs will produce different Workflow History for similar code, and consequently, it is not possible to use register Workers written using different SDKs on a same Workflow Task Queue.

I hope that answer your question. If not, can you elaborate on why you want to use the same Task Queues for different Worker implementations?

1 Like

Thank you so much! I think I understand the concept better and it makes sense. Thanks for the help!