Multiple Worker Instances for Asynchronous Activity Processing.

Community Support
1

I am new to Golang and temporal so thanks for understating.

In this scenario, we have more than one identical worker, each running in a separate container. in a workflow, there is an activity that starts an asynchronous task and then waits for the result via a blocking call on a signal channel. When the asynchronous task finishes, it returns the result to a pub/sub queue to which all the running workflow instances would be subscribed.

1)is having multiple worker instances running in separate containers the best architecture for asynchronous task processing? if not, what would be the best architecture?

  1. does the worker perform autoscaling if there are many tasks in the queues to which it is subscribed?
2

I’m not sure why you need the pub/sub queue for this use case? I would suggest to directly send the signal to the workflow execution after the asynchronous job finishes instead of putting in some intermediate queue. I’m not sure what you mean by the following:

to which all the running workflow instances would be subscribed.

Answer for your questions:

  1. In general we recommend running multiple worker instances, one in each container for redundancy. Temporal workers are completely stateless so if any worker goes down, Temporal would automatically resurrect state of workflow executions on a different instance when a new event happens.
  2. Temporal itself does not perform auto-scaling of workers. We expose schedule-to-start latency metric on the client-side which is a very good indicator to monitor and scale your workers if it breaches certain threshold.
3

Thanks for your clarification. I was trying to say that we have an intermediate queue by saying to which all the running workflow instances would be subscribed. but I wasn’t very clear in my writing. Thanks again!