In my usage senior, there is a large number (1 million) of worker processes connect to a temporal cluster simultaneously. And each worker process poll tasks from a task queue, so there should be 1 million task queues.
When I read the official website, I think maybe there is no problem with the task queues number, the consideration is only for the connections number.
This is designed because I need to execute his own tasks on each machine. So Am I using temporal the right way? Will such a large number of worker connections affect the performance of the temporal cluster?
This is possible, but you should take some care to avoid creating even more task queues than necessary:
Scale the matching service appropriately and configure appropriate limits for concurrent pollers in frontend. Some experimentation may be required here.
I assume you’re talking about activity task queues. Make sure that these workers have only an activity worker running and not an (unused) workflow worker, or you’ll end up creating workflow task queues also.
Configure the server to use only 1 partition for these task queues. (You’ll have to do it at the namespace level.) For Temporal Cloud, you can contact us to make the necessary adjustments.
If you can reuse task queue names, that will help to reduce the amount of garbage in the database. The task queue metadata scavenger doesn’t work on cassandra.
