Identify active worker queues

Community Support
1

Is there any way to identify the names of queues being listened on by workflow or activity workers?

For example, I would like to deploy a worker that listens to a worker-defined job queue. Is there a way to query the temporal server to discover the name of the queue the worker is register against?

Motivation:
We have a large number of job-processors. They conform to the same interface but do different things (i.e. use different algorithms to accomplish the job). I’m interested in an architecture where I can add a new algorithm to the system by simply deploying a new worker. Our API would then detect that a new queue exists and expose a new endpoint that users can invoke to use the new algorithm.

2

There is no list task queue API yet. In your case I would recommend a following workaround.

  • Have a workflow that owns the list of the workers.
  • Upon startup, the worker would notify this workflow about its task queue via signal.
  • Other workflows would query the “task queue list” workflow for the list of available algorithms
3

@maxim - thank you for the quick response.

You say “no API yet”. Is such an API on the roadmap?

4

And for the workflow implementation in the workaround - you would just have a workflow that runs indefinitely? Are there implications for the size of the history over time?

Also, if I want to prune workers that are no longer active, I’d need to have some kind of heartbeat signal from the workers, which I assume will bloat the history even faster. Suggestions?

5

You say “no API yet ”. Is such an API on the roadmap?

It is a pretty commonly requested feature. So we will implement it someday.

And for the workflow implementation in the workaround - you would just have a workflow that runs indefinitely? Are there implications for the size of the history over time?

Such workflow has to call continue as new periodically to trim the history.

Also, if I want to prune workers that are no longer active, I’d need to have some kind of heartbeat signal from the workers, which I assume will bloat the history even faster. Suggestions?

The heartbeat signal is one option. Another is to use a short ScheduleToStart timeout for such queues. This way, if the worker cannot pick up the task within the timeout, it will fail. Then the caller can notify the “task queue list” workflow that the given task queue is having issues. To be robust you might need to use both techniques.