On Cassandra, what should be the read and write level?
I guess you mean read/write consistency. gocql driver used for cassandra defaults to LOCAL_QUORUM and LOCAL_SERIAL.
What latency can we expect on the client side and does Cadence/Temporal expose any metrics for the same?
Temporal emits tons of metric both on server and client. Look at existing threads on the forum on this topic. Here are some to get you started:
How does tombstoning have an impact on the performance
We have tested Cassandra schema very carefully and it is throughly tested for sustained load for long periods of time. Cassandra tombstone characteristics should not have an impact on performance characteristics of the server. Watch my architecture talk which touches this topic.
What is the recommended backup frequency?
Ideally we recommend to run the server with cross data center replication. We have our own replication stack and does not rely on database level replication for various reasons. It is completely upto you for backup frequency for disaster scenario. Basically it comes down to your business requirements rather than any requirements enforced by Temporal. I would rely on best practice recommended by Cassandra.
Can you pls share the number of reads per second and writes per second that Cadence does with Cassandra. This will help us a lot while provisioning the infra for our Cassandra cluster (at optimal cost).
We don’t have any benchmarks to share at this point. I would recommend to run a bench run specifically for your use case to generate those numbers.