Hello,
We have a use case where we start 1000s of workflows every 15 minutes from our app. In my app logs i can see that all workflows are started in less than a second, but on temporal side, it takes about 1m20s to start every workflow.
I have consulted many topics on the forum with same-ish issue, we now have MaxConcurrentWorkflowTaskPollers: 400
and MaxConcurrentWorkflowTaskPollers: 400
in the app, and on the server, we have this dynamic config:
dynamicConfig:
frontend.persistenceMaxQPS:
-
constraints: {}
value: 18000
frontend.rps:
-
constraints: {}
value: 4800
history.persistenceMaxQPS:
-
constraints: {}
value: 18000
history.rps:
-
constraints: {}
value: 12000
matching.numTaskqueueReadPartitions:
-
constraints: {}
value: 40
matching.numTaskqueueWritePartitions:
-
constraints: {}
value: 40
matching.persistenceMaxQPS:
-
constraints: {}
value: 18000
matching.rps:
-
constraints: {}
value: 4800
But now i can’t get faster starting time for all my workflow.
I’m monitoring pods resources (app and temporal cluster), but i do not see any significant load, except on the history service, but does not get near the limit at all.
Am i expecting to much from temporal ? I have trouble to understand where is the bottleneck in my setup.
Thanks
Maxime