I have deployed Cassandra on 3 node GKE cluster of 16 cores and 64GB memory each. I have configured Cassandra(3 pods) with max heap size of 16GB.
As you can see in above screenshot, all the pods are using their max heap memory (16GB * 3)even when any workflows are not being executed. And we don’t see any GC occuring either.
For our workflow runs(120 ms fixed due to worker execution time):
50tps: 340.49 ms
But anything more than that, the response time goes in seconds. Hence we were suspecting cassandra to be the bottleneck.
Hence need your suggestion.