# What is the recommended setup for running Cadence/Temporal with Cassandra on production? **URL:** \ **Category:** Community Support\ **Tags:** cassandra, production\ **Created:** [August 31, 2020, 8:00pm UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556 "2020-08-31T20:00:48Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Akshay\_Goyal](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/akshay_goyal/32/112_2.png) [@Akshay\_Goyal](https://community.temporal.io/u/Akshay_Goyal)\ **Post date:** [August 31, 2020, 8:00pm UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/1 "2020-08-31T20:00:48Z")
We’ll be running our own Cassandra setup on Kubernetes What is the recommended - cluster size - replication factor - write level - read level - Any other parameter that I may have missed Rough scale estimates: **Current scale:** wf / sec: ~50 actions / sec: ~100 signals / sec: ~100 **Not too optimistic future needs:** wf / sec: ~150 actions / sec: ~300 signals / sec: ~300 **Very Optimistic future needs:** wf / sec: ~500 actions / sec: ~1000 signals / sec: ~1000 How many shard and pods should we go with? Any other factor we consider while running Cadence/Temporal with Cassandra on prod? ---
**Author:** ![samar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/samar/32/12_2.png) [@samar](https://community.temporal.io/u/samar)\ **Post date:** [September 1, 2020, 4:55am UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/3 "2020-09-01T04:55:23Z")
Hey @Akshay_Goyal, I can provide some high level recommendations so you have a starting point, but mostly you have to run your bench test your usecase to figure out the right sizing for your workload. Here are some of the things to consider when sizing your clusters: **Number Of Shards** Based on your future load characteristics I would recommend to start with atleast 4k history shards. Number of history shards is a setting which cannot be updated after the cluster is provisioned. For all other parameters you could start small and scale your cluster based on need with time but this one you have to think upfront about your maximum load. **Core Database** Temporal is write intensive and uses LWT feature of Cassandra. 50% of the writes are LWT. I expect your workload to be mostly bounded by database capacity. Nice thing about Cassandra is that it scales horizontally and you just increases the size of cluster for Temporal to scale on throughput. We recommend to use replication factor of 3 for core Temporal keyspace. You want to start with 5 Cassandra hosts each with 16 cores and 32 GB memory. **Visibility Database** I don’t think Cassandra implementation for visibility store would scale at the numbers you are looking at. You should start with ElasticSearch as the visibility store. **Temporal Roles** Temporal server consists of 4 roles. Although you can run all roles within same process but we highly recommend running them separately as they have completely different concerns and scale characteristics. It also makes it operationally much simpler to isolate problems in production. All of the roles are completely stateless and system scales horizontally as you spin up more instances of role once you identify any bottleneck. Here are some recommendations to use as a starting point: 1. **Frontend:** Responsible for hosting all service api. All client interaction goes through frontend and mostly scales with rps for the cluster. Start with 3 instances of 4 cores and 4GB memory. 2. **History:** This hosts the workflow state transition logic. Each history host is running a shard controller which is responsible for activating and passivating shards on that host. If you provision a cluster with 4k shards then they are distributed across all available history hosts within the cluster through shard controller. If history hosts are scalability bottleneck, you just add more history hosts to the cluster. All history hosts form its own membership ring and shards are distributed among available nodes in the hash ring. They are quite memory intensive as they host mutable state and event caches. Start with 5 history instances with 8 cores and 8 GB memory. 3. **Matching:** They are responsible for hosting TaskQueues within the system. Each TaskQueue partition is placed separately on all available matching hosts. They usually scale with the number of workers connecting for workflow or activity task, throughput of workflow/activity/query task, and number of total active TaskQueues in the system. Start with 3 matching instances each with 4 cores and 4 GB memory. 4. **Worker** : This is needed for various background logic for ElasticSearch kafka processor, CrossDC consumers, and some system workflows (archival, batch processing, etc). You can just start with 2 instances each with 4 cores and 4 GB memory. Hope this will give you a good starting point. Let’s keep updating this thread when you have more specific questions. ---
**Author:** ![Akshay\_Goyal](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/akshay_goyal/32/112_2.png) [@Akshay\_Goyal](https://community.temporal.io/u/Akshay_Goyal)\ **Post date:** [September 2, 2020, 4:19am UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/4 "2020-09-02T04:19:44Z")
Hi @samar thanks for the detailed answer. **Our Cassandra setup:** We are running our own Cassandra setup on Kubernetes .Our primary Cassandra cluster setup has 5 nodes (hosted in 1 AZ) and we have a backup of 3 nodes cluster which acts as DR (in another AZ). Have a few followup questions setting up Cassandra for Cadence/Temporal: 1. On Cassandra, what should be the read and write level? 2. What latency can we expect on the client side and does Cadence/Temporal expose any metrics for the same? 3. How does tombstoning have an impact on the performance 4. What is the recommended backup frequency? 5. Can you pls share the number of reads per second and writes per second that Cadence does with Cassandra. This will help us a lot while provisioning the infra for our Cassandra cluster (at optimal cost). ---
**Author:** ![Arjun\_Narain](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/arjun_narain/32/268_2.png) [@Arjun\_Narain](https://community.temporal.io/u/Arjun_Narain)\ **Post date:** [September 2, 2020, 5:23am UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/5 "2020-09-02T05:23:41Z")
> [@samar](#): > > hey are quite memory intensive as they host mutable state and event caches. Start with 5 history instances with 8 cores and 8 GB memory. Hi Samar, Follow up question on the memory and CPU cores. Does each of the 5 instances have to have 8 cores and 8 GB Memory? Can I split the total provisioned cores and memory into more number of history pods? ---
**Author:** ![Akshay\_Goyal](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/akshay_goyal/32/112_2.png) [@Akshay\_Goyal](https://community.temporal.io/u/Akshay_Goyal)\ **Post date:** [September 2, 2020, 5:22pm UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/6 "2020-09-02T17:22:59Z")
Hi @samar Our scale requirements change from **20-30 wps** for now to maybe ~500 wps in say next 10-12 months. We’re also very conservative about our infra costs and want to optimise it as much as possible, without impacting uptime. Q6: Now if we were to set up 4k shards, does that have any implications on the infra we provision for both Cadence components or Cassandra? Q7: What is the minimum setup that I can run to support 30-50 tps on day 1 and have the ability to scale it up to 500 tps when required. Q8: Also would love to understand what metric should we look at to identify what component we need to scale. ---
**Author:** ![samar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/samar/32/12_2.png) [@samar](https://community.temporal.io/u/samar)\ **Post date:** [September 3, 2020, 2:08am UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/7 "2020-09-03T02:08:22Z")
> On Cassandra, what should be the read and write level? I guess you mean read/write consistency. gocql driver used for cassandra defaults to LOCAL\_QUORUM and LOCAL\_SERIAL. > What latency can we expect on the client side and does Cadence/Temporal expose any metrics for the same? Temporal emits tons of metric both on server and client. Look at existing threads on the forum on this topic. Here are some to get you started: > [@Guidance on creating and interpreting Grafana dashboards](https://community.temporal.io/t/guidance-on-creating-and-interpreting-grafana-dashboards/493/3): > > Hey @rfwagner, This is a lengthy topic and we plan to provide more information on operationalizing Temporal clusters in the future. We recently started [dashboard repo](https://github.com/temporalio/dashboards) which has some basic dashboards which gives visibility into Temporal service. This is work in progress so please use this just as a reference at this point. In the future we plan to have fully supported version of grafana dashboards using PromQL. All the metric emitted by server are listed in [defs.go](https://github.com/temporalio/temporal/blob/master/common/metrics/defs.go). So if you see somethings… > [@What metrics does temporal expose out of box and how to consume this in prometheus?](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346): > > Hi, there is not much documentation around metrics, the statd which was removed too seemed very complex. I want to to understand a)what metrics does temporal expose by default b) are the metrics namespace specific? c) can i get queue /task list specific metrics? d) how to consume them in prometheus. e) if i am to develop custom metrics what’s the best way, should those be activities in workflows or interceptors? > How does tombstoning have an impact on the performance We have tested Cassandra schema very carefully and it is throughly tested for sustained load for long periods of time. Cassandra tombstone characteristics should not have an impact on performance characteristics of the server. Watch my architecture [talk](https://www.youtube.com/watch?v=5M5eiNBUf4Q&t) which touches this topic. > What is the recommended backup frequency? Ideally we recommend to run the server with cross data center replication. We have our own replication stack and does not rely on database level replication for various reasons. It is completely upto you for backup frequency for disaster scenario. Basically it comes down to your business requirements rather than any requirements enforced by Temporal. I would rely on best practice recommended by Cassandra. > Can you pls share the number of reads per second and writes per second that Cadence does with Cassandra. This will help us a lot while provisioning the infra for our Cassandra cluster (at optimal cost). We don’t have any benchmarks to share at this point. I would recommend to run a bench run specifically for your use case to generate those numbers. ---
**Author:** ![samar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/samar/32/12_2.png) [@samar](https://community.temporal.io/u/samar)\ **Post date:** [September 3, 2020, 2:13am UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/8 "2020-09-03T02:13:11Z")
> Now if we were to set up 4k shards, does that have any implications on the infra we provision for both Cadence components or Cassandra? Shards are very lightweight. There are no real implications on the cost of clusters. We have tested the system upto 16k shards. > Q7: What is the minimum setup that I can run to support 30-50 tps on day 1 and have the ability to scale it up to 500 tps when required. 30-50 tps is pretty low scale from our perspective. More than performance the size of our cluster would be driven by availability requirements. The rough numbers I provided earlier is a good place to start and going anything below those numbers may have implications on availability of the clusters during certain failure scenarios. > Q8: Also would love to understand what metric should we look at to identify what component we need to scale. Please refer to the metric posts I linked earlier. ---
**Author:** ![Akshay\_Goyal](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/akshay_goyal/32/112_2.png) [@Akshay\_Goyal](https://community.temporal.io/u/Akshay_Goyal)\ **Post date:** [September 3, 2020, 11:36pm UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/9 "2020-09-03T23:36:21Z")
Thanks for the detailed answers. I’ll run my perf and get back to you. > [@Arjun\_Narain](#): > > Follow up question on the memory and CPU cores. Does each of the 5 instances have to have 8 cores and 8 GB Memory? Can I split the total provisioned cores and memory into more number of history pods? @samar could you answer this as well please? 🙂 ---
**Author:** ![Akshay\_Goyal](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/akshay_goyal/32/112_2.png) [@Akshay\_Goyal](https://community.temporal.io/u/Akshay_Goyal)\ **Post date:** [September 5, 2020, 12:03am UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/10 "2020-09-05T00:03:11Z")
> [@samar](#): > > The rough numbers I provided earlier is a good place to start and going anything below those numbers may have implications on availability of the clusters during certain failure scenarios. Can you pls elaborate on such scenarios? We’ll be doing perf testing and would like to recreate these scenarios if possible. ---
**Author:** ![samar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/samar/32/12_2.png) [@samar](https://community.temporal.io/u/samar)\ **Post date:** [September 5, 2020, 4:13pm UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/11 "2020-09-05T16:13:46Z")
We send out a bi-weekly update on our stabilization testing which covers details on the kind of failure testing. [Here](https://docs.temporal.io/blog/temporal-transparency-5) is one which give you an idea on failure scenarios we are testing for our V1 production release. ---
**Author:** ![Raul-Ronald](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/raul-ronald/32/4548_2.png) [@Raul-Ronald](https://community.temporal.io/u/Raul-Ronald)\ **Post date:** [November 21, 2023, 5:24pm UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/13 "2023-11-21T17:24:33Z")
Are there updates to this? Are the specs described here still relevant ~3 years later? How might numbers have changed? ---
**Author:** ![parul\_rawat](https://avatars.discourse-cdn.com/v4/letter/p/9f8e36/32.png) [@parul\_rawat](https://community.temporal.io/u/parul_rawat)\ **Post date:** [September 18, 2026, 8:18am UTC](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/14 "2026-09-18T08:18:19Z")
Hi @samar and others, A little late to the party but if anyone can help us answer for our scale as well For 3 wfs: Our Temporal TPS: 30/s Our TPS is low but our waits schedules are high Wf pauses at the rate of 12/s Wf pause average: 3 months Wf resumes at the rate of: 3/s For 1 wf: At a time there are 89M journeys waiting in the system Size of each waiting journey in DB → 4-8KB → ~360-715 GB Assuming our retention policy for finished workflows is 15 days Size of each finished run in DB → ~40-80 KB → ~0.6-1.2TB Total raw storage → ~1–2 TB total raw storage for 3 wfs → ~3-6TB Any recommendation on numHistoryShards setting in Temporal?