I am planning to deploy temporal in kubernetes cluster with AWS aurora RDS. I am having trouble estimating the right value of deployment configurations. Any approximations or ballpark numbers should be helpful.
Assume, I have to run 10k workflows per second with 10 activities each.
total tasks per second per workflow - workflow tasks (11) + activity tasks (10) = 21
total tasks per second = 210k
- How do I calculate the right number of shards required for my cluster to function properly?
- Any physical limitations around number of shards in SQL based datastores? Can it support 10k shards or more?
- As maxim mentioned in scaling-task-queue , we have to configure 100 partitions to support 10k tasks per second. For 210k tasks per second, 2100 partitions are required. Will the system work effectively with 2100 partitions (especially on mysql) ?
- what should be the service RPS for history, matching and frontend to avoid contextDeadlineExceeded and resource exhausted errors?
- To support such a scale, is it a good idea to increase max number of db connections by temporal services?
Thanks for helping