In the production environment, I have millions of pipelines to execute, and each pipeline corresponds to a PipelineWorkflow class with @WorkflowInterface annotations, and some of the subjobs of the pipeline correspond to a PipelineActivities class with @ActivityInterface annotations. Will there be performance pressure when millions of pipelines and sub-jobs are registered with one worker, that is, all of them share a taskqueue?
The performance limitation is mainly related to the number of actions per second across all these pipelines. Each worker would have limited throughput. So more workers might be needed to execute at higher rates.
However, for millions of pipelines, when many workers are created, an error indicating that resources exceed the limit is reported.
Open workflows that are waiting on something don’t consume worker resources. We need more information about which exact resource limit is reported to help you.