Noticed that with decrease in number of workers, all else being constant
Its understandable that workflow completion decreases with reduced workers, but why does rate of workflow starts increase?
Check your frontend requests during this time if it was increased.
sum by (operation) (rate(service_requests{service_name="frontend"}[5m]))
Workflow execution is created by server on client / worker requests, don’t think it has to directly do with number of your workers (workflow executions can be created even if none of your workers are up and polling).
Specifically, can we say that workflows can be scheduled and started even if none of the workers are up and polling?
Workflow executions can be requested (by clients for example) if workers are not running. Workflow executions can be created (started) by service if workers are not running as well.
First workflow task can be scheduled on matching task queue without workers running.
Workflow executions can be started and completed/timed out without workers running.
Workflow executions can be terminated without workers running.
Workflow execution progress (execution of your workflow/activity code) cannot be made without workers running to poll work from task queues.