# Recommendation for K8S Cluster; currently using default values

**URL:** <https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916>\
**Category:** Server Deployment\
**Tags:** helm, general-impl\
**Created:** [October 24, 2023, 9:52pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916 "2023-10-24T21:52:49Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![sebastian\_violet](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@sebastian\_violet](https://community.temporal.io/u/sebastian_violet)\
**Post date:** [October 24, 2023, 9:52pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/1 "2023-10-24T21:52:49Z")

</div>

Hello Team,

Our team has deployed Temporal using the Helm charts provided here: [GitHub - temporalio/helm-charts: Temporal Helm charts](https://github.com/temporalio/helm-charts)

Recently, we got the following error when trying to create a new schedule:

```auto
StatusRuntimeException: UNAVAILABLE: Not enough hosts to serve the request

```

After doing a bit of research, it turns out that the **`temporaltest-history`** pods were being killed with OOMKiller(137) and restarting over and over again.

The deployment is on AWS EKS, and the pods have the minimum amount or resources for memory and cpu; 0.25 units for CPU and 0.5 Gib for Memory.

As we are just starting to use Temporal, our avg/peak WPS is \< 10 WPS.

[Another topic has the following recommendations](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/3):

- **Frontend:** …Start with 3 instances of 4 cores and 4GB memory.
- **History:** …Start with 5 history instances with 8 cores and 8 GB memory.
- **Matching:** …Start with 3 matching instances each with 4 cores and 4 GB memory.
- **Worker** : …You can just start with 2 instances each with 4 cores and 4 GB memory.

However, these values seem a bit high for our current operations scale.

Can anyone give recommendations based on throughput?

---

<div class="post-metadata">

**Author:** ![sebastian\_violet](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@sebastian\_violet](https://community.temporal.io/u/sebastian_violet)\
**Post date:** [October 24, 2023, 9:53pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/2 "2023-10-24T21:53:18Z")

</div>

@samar for input.  
Thank You

---

<div class="post-metadata">

**Author:** ![samar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/samar/32/12_2.png) [@samar](https://community.temporal.io/u/samar)\
**Post date:** [October 24, 2023, 11:26pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/3 "2023-10-24T23:26:25Z")

</div>

@sebastian_violet can you share the following information for your cluster?

1. Temporal Server Version
2. Number of Shards
3. Config: Max History Cache Size (“history.cacheMaxSize”)
4. Config: Events Cache Size (“history.eventsCacheMaxSizeBytes”)

---

<div class="post-metadata">

**Author:** ![sebastian\_violet](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@sebastian\_violet](https://community.temporal.io/u/sebastian_violet)\
**Post date:** [October 25, 2023, 4:31pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/4 "2023-10-25T16:31:39Z")

</div>

Here are the values:

1. Temporal Server Version

```auto
server.image.repository=temporalio/server
server.image.tag=1.22.1

```

1. Number of Shards

```auto
server.config.numHistoryShards=512

```

1. Config: Max History Cache Size (“history.cacheMaxSize”)

- I assume the default(_128_) based on this: [History service memory usage - #4 by tihomir](https://community.temporal.io/t/history-service-memory-usage/5569/4)
- [I don’t see a place to set it in the Helm Chart](https://github.com/temporalio/helm-charts)
  - I assume we can set it here `/etc/temporal/dynamic_config`

1. Config: Events Cache Size (“history.eventsCacheMaxSizeBytes”)

- I assume the default(_512_) based on this: [History service memory usage - #4 by tihomir](https://community.temporal.io/t/history-service-memory-usage/5569/4)
- [I don’t see a place to set it in the Helm Chart](https://github.com/temporalio/helm-charts)
  - I assume we can set it here `/etc/temporal/dynamic_config`

---

<div class="post-metadata">

**Author:** ![samar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/samar/32/12_2.png) [@samar](https://community.temporal.io/u/samar)\
**Post date:** [October 26, 2023, 4:10pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/5 "2023-10-26T16:10:42Z")

</div>

These values might be too high for your setup. These cache sizes are per shard, so if you have event payloads which are larger then it could quickly add up and take up all the memory on your history service. Based on the numbers you provided above:  
MutableStateCacheSize = 512 (shards) X 512 (Default Max Items) = 256K Items  
HistoryEvent Cache Size = 512 (shards) X 512 (Default Max Items) = 256K Items

So if you use an avg of 10KB per item then these 2 caches could take upto 5 Gigs of memory. Can you share how many history service pods you have and what is the memory limit on those pods?

Can you try configuring the cache sizes to smaller number and see if this helps with the OOM?

We have workstreams in progress to make this simpler by configuring it per history host and setting the limit in bytes rather the cache items.

And yes, please use `dynamic_config` to override these limits.

---

<div class="post-metadata">

**Author:** ![sebastian\_violet](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@sebastian\_violet](https://community.temporal.io/u/sebastian_violet)\
**Post date:** [October 26, 2023, 5:02pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/6 "2023-10-26T17:02:20Z")

</div>

Hello @samar,

Currently, the history service has 3 pods each with 2 Gigs of memory.

I will try and adjust the caches to a lower number and see if that resolves it.

Thank You

---

<div class="post-metadata">

**Author:** ![sebastian\_violet](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@sebastian\_violet](https://community.temporal.io/u/sebastian_violet)\
**Post date:** [October 26, 2023, 5:57pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/7 "2023-10-26T17:57:44Z")

</div>

Here are the changes I made based on your formula above:

```auto
---
# Source: temporal/templates/server-dynamicconfigmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: "temporaltest-dynamic-config"
  labels:
    app.kubernetes.io/name: temporal
    helm.sh/chart: temporal-0.29.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/version: 1.22.1
    app.kubernetes.io/part-of: temporal
data:
  dynamic_config.yaml: |-
    history:
      cacheInitialSize:
      - value: 32
      cacheMaxSize:
      - value: 64
      eventsCacheInitialSize:
      - value: 32
      eventsCacheMaxSize:
      - value: 64

```

---

<div class="post-metadata">

**Author:** ![sebastian\_violet](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@sebastian\_violet](https://community.temporal.io/u/sebastian_violet)\
**Post date:** [October 26, 2023, 6:04pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/8 "2023-10-26T18:04:14Z")

</div>

> [@samar](#):
>
> 10KB per item

Would Item in this case be a single line(an Event) in the workflow, or the entire workflow(comprised of multiple events)? I clicked on download on a workflow, and it showed 30K for the json file.

Also, is the cache here for the workers to be able to continue from where the last execution left off, or is it for the Web UI to display results quick?

What are the ramifications of the following:

- No cache
- Small cache

Large cache we know takes up too much memory in the history pods.

Thank You

---

<div class="post-metadata">

**Author:** ![sebastian\_violet](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@sebastian\_violet](https://community.temporal.io/u/sebastian_violet)\
**Post date:** [October 27, 2023, 12:43am UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/9 "2023-10-27T00:43:51Z")

</div>

> [@sebastian\_violet](#):
>
> ```auto
> history:
> cacheInitialSize:
> - value: 32
> cacheMaxSize:
> - value: 64
> eventsCacheInitialSize:
> - value: 32
> eventsCacheMaxSize:
> - value: 64
> 
> ```

Also, this doesn’t seem to work and throws the following error:

```auto
{"level":"error","ts":"2023-10-27T00:41:20.281Z","msg":"Unable to update dynamic config.","error":"unable to decode dynamic config: yaml: unmarshal errors:\n line 2: cannot unmarshal !!map into []struct { Constraints map[string]interface {}; Value i │
│ nterface {} }","logging-call-at":"file_based_client.go:136","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/dynamicconfig.(*fileBasedClient).init.f │
│ unc1\n\t/home/builder/temporal/common/dynamicconfig/file_based_client.go:136"}

```

---

<div class="post-metadata">

**Author:** ![sebastian\_violet](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@sebastian\_violet](https://community.temporal.io/u/sebastian_violet)\
**Post date:** [October 27, 2023, 12:49am UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/10 "2023-10-27T00:49:44Z")

</div>

I see that the key itself has `.` in it. Thus for Helm `--set`, it needs to be like so:

```auto
...
  --set "server.dynamicConfig.history\.cacheInitialSize[0].value"=32 \
  --set "server.dynamicConfig.history\.cacheMaxSize[0].value"=64 \
  --set "server.dynamicConfig.history\.eventsCacheInitialSize[0].value"=32 \
  --set "server.dynamicConfig.history\.eventsCacheMaxSize[0].value"=64 \
...

```

I.e. `history\.<key>`

This gives the following:

```auto
---
# Source: temporal/templates/server-dynamicconfigmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: "temporaltest-dynamic-config"
  labels:
    app.kubernetes.io/name: temporal
    helm.sh/chart: temporal-0.29.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/version: 1.22.1
    app.kubernetes.io/part-of: temporal
data:
  dynamic_config.yaml: |-
    history.cacheInitialSize:
    - value: 32
    history.cacheMaxSize:
    - value: 64
    history.eventsCacheInitialSize:
    - value: 32
    history.eventsCacheMaxSize:
    - value: 64

```

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [October 28, 2023, 9:19pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/11 "2023-10-28T21:19:51Z")

</div>

You should be able to edit values.dynamic\_config.yaml directly, does this not work for you?

---

<div class="post-metadata">

**Author:** ![sebastian\_violet](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@sebastian\_violet](https://community.temporal.io/u/sebastian_violet)\
**Post date:** [October 30, 2023, 3:50pm UTC](https://community.temporal.io/t/recommendation-for-k8s-cluster-currently-using-default-values/9916/12 "2023-10-30T15:50:05Z")

</div>

The setup from above works perfectly. I just had to escape the `.` when using `--set` during helm install. The `.` is part of the config name and not indicative of a sub property.

```auto
history.cacheInitialSize

```

vs

```auto
history:
  cacheInitialSize

```