Temporal is slow to start burst of 1000s of workflows

Hello,

We have a use case where we start 1000s of workflows every 15 minutes from our app. In my app logs i can see that all workflows are started in less than a second, but on temporal side, it takes about 1m20s to start every workflow.

I have consulted many topics on the forum with same-ish issue, we now have MaxConcurrentWorkflowTaskPollers: 400 and MaxConcurrentWorkflowTaskPollers: 400 in the app, and on the server, we have this dynamic config:

dynamicConfig: 
    frontend.persistenceMaxQPS: 
    - 
      constraints: {}
      value: 18000
    frontend.rps: 
    - 
      constraints: {}
      value: 4800
    history.persistenceMaxQPS: 
    - 
      constraints: {}
      value: 18000
    history.rps: 
    - 
      constraints: {}
      value: 12000
    matching.numTaskqueueReadPartitions: 
    - 
      constraints: {}
      value: 40
    matching.numTaskqueueWritePartitions: 
    - 
      constraints: {}
      value: 40
    matching.persistenceMaxQPS: 
    - 
      constraints: {}
      value: 18000
    matching.rps: 
    - 
      constraints: {}
      value: 4800

But now i can’t get faster starting time for all my workflow.
I’m monitoring pods resources (app and temporal cluster), but i do not see any significant load, except on the history service, but does not get near the limit at all.
Am i expecting to much from temporal ? I have trouble to understand where is the bottleneck in my setup.
Thanks :slight_smile:
Maxime