Hi ,
Could you please provide information on how many concurrent workflows a single Temporal shard can support ? what’s the metric to compute this ? , Is this configurable via Helm charts? Additionally, I would like to know if it’s possible to increase the number of shards post-deployment without affecting existing workflows.
Thanks.
how many concurrent workflows a single Temporal shard can support
this will highly depend on your db, max concurrency is going to be bounded by your db partition size limit and its processing capacity.
processing capacity can be measured via shard lock latency.
you can look at server metric semaphore_latency (available since server release 1.23.0, before that use lock_latency) for example p99:
histogram_quantile(0.99, sum(rate(semaphore_latency_bucket[5m])) by (le))
another thing you can look at is to adjust concurrency of persistence operations in shard context via dynamic config history.shardIOConcurrency, default is 1 (note for cassandra persistence it cannot be set to > 1 currently)
I would like to know if it’s possible to increase the number of shards post-deployment without affecting existing workflows.
no, its not possible unless you start with “fresh” db as well (have to re-index and lose your current data).
since server release 1.20 however you can use multi-cluster replication to replicate between clusters with different shard counts (have to be multiple of each other). this made possible to replicate to a cluster with a much larger shard count allowing you to eventually fail over to the larger cluster