# Temporal Shards ( configured in 'shards' table )

**URL:** <https://community.temporal.io/t/temporal-shards-configured-in-shards-table/15533>\
**Category:** Community Support\
**Tags:** java-sdk\
**Created:** [December 5, 2024, 4:51am UTC](https://community.temporal.io/t/temporal-shards-configured-in-shards-table/15533 "2024-12-05T04:51:53Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rakesh\_P](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/rakesh_p/32/5815_2.png) [@Rakesh\_P](https://community.temporal.io/u/Rakesh_P)\
**Post date:** [December 5, 2024, 4:51am UTC](https://community.temporal.io/t/temporal-shards-configured-in-shards-table/15533/1 "2024-12-05T04:51:53Z")

</div>

Hi ,

Could you please provide information on how many concurrent workflows a single Temporal shard can support ? what’s the metric to compute this ? , Is this configurable via Helm charts? Additionally, I would like to know if it’s possible to increase the number of shards post-deployment without affecting existing workflows.

Thanks.

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [December 8, 2024, 9:34pm UTC](https://community.temporal.io/t/temporal-shards-configured-in-shards-table/15533/2 "2024-12-08T21:34:03Z")

</div>

> how many concurrent workflows a single Temporal shard can support

this will highly depend on your db, max concurrency is going to be bounded by your db partition size limit and its processing capacity.

processing capacity can be measured via shard lock latency.  
you can look at server metric `semaphore_latency` (available since server release 1.23.0, before that use `lock_latency`) for example p99:

`histogram_quantile(0.99, sum(rate(semaphore_latency_bucket[5m])) by (le))`

another thing you can look at is to adjust concurrency of persistence operations in shard context via dynamic config `history.shardIOConcurrency`, default is 1 (note for cassandra persistence it cannot be set to \> 1 currently)

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [December 8, 2024, 9:39pm UTC](https://community.temporal.io/t/temporal-shards-configured-in-shards-table/15533/3 "2024-12-08T21:39:15Z")

</div>

> I would like to know if it’s possible to increase the number of shards post-deployment without affecting existing workflows.

no, its not possible unless you start with “fresh” db as well (have to re-index and lose your current data).

since server release 1.20 however you can use multi-cluster replication to replicate between clusters with different shard counts (have to be multiple of each other). this made possible to replicate to a cluster with a much larger shard count allowing you to eventually fail over to the larger cluster

---

<div class="post-metadata">

**Author:** ![nathanielobrown](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/nathanielobrown/32/6818_2.png) [@nathanielobrown](https://community.temporal.io/u/nathanielobrown)\
**Post date:** [February 8, 2025, 3:06pm UTC](https://community.temporal.io/t/temporal-shards-configured-in-shards-table/15533/4 "2025-02-08T15:06:51Z")

</div>

@tihomir I’m on Temporal version 1.23 and I have P95 `lock_latency` of under 1ms but `semaphore_latency_bucket` is typically 10ms and peaks at 90ms. The [Scaling Temporal the basics](https://temporal.io/blog/scaling-temporal-the-basics) post made me think everything is good due to low lock latency but reading your post I’m wondering if I should ignore lock latency and only pay attention to semaphore latency, what would you advise? Also, what is a tolerable semaphore latency?

For extra context, I’m on a cluster with only 8 shards (silly historical choice) which I’m worried is causing me problems but want to make sure I’m using the right metric to measure the issue.

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [February 8, 2025, 4:15pm UTC](https://community.temporal.io/t/temporal-shards-configured-in-shards-table/15533/5 "2025-02-08T16:15:01Z")

</div>

> but `semaphore_latency_bucket` is typically 10ms and peaks at 90ms

correct, `semaphore_latency` is new metric to use for monitoring processing capacity. `lock_latency` still exists but its used for something else now

> I’m wondering if I should ignore lock latency and only pay attention to semaphore latency

correct

---

<div class="post-metadata">

**Author:** ![nathanielobrown](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/nathanielobrown/32/6818_2.png) [@nathanielobrown](https://community.temporal.io/u/nathanielobrown)\
**Post date:** [February 10, 2025, 1:53pm UTC](https://community.temporal.io/t/temporal-shards-configured-in-shards-table/15533/6 "2025-02-10T13:53:23Z")

</div>

Thank you @tihomir! Should `semaphore_latency` be less than 5ms ideally less than 1ms, just like the guidance for `lock_latency`?

Also, any guidance on what we can do with `history.shardIOConcurrency`? Like how high can we set it or what metrics should we watch for to identify if we have it set optimally? I cannot find any documentation on this setting. Some guidance on how to think about what this setting does would be much appreciated. I’m using Postgres.

---

<div class="post-metadata">

**Author:** ![nathanielobrown](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/nathanielobrown/32/6818_2.png) [@nathanielobrown](https://community.temporal.io/u/nathanielobrown)\
**Post date:** [February 10, 2025, 11:54pm UTC](https://community.temporal.io/t/temporal-shards-configured-in-shards-table/15533/7 "2025-02-10T23:54:41Z")

</div>

Update: I’ve been varying `history.shardIOConcurrency` (tried values like 2 and 3) and have not noticed it have an affect on `semaphore_latency`
