During Peak Load, Cassandra cluster performance deteriorates, though we dont see any spike in cpu utilization, Read or Write IO.
Read latencies go beyond 1 second
Not able to scale the load when we increase the number of nodes in the cassandra cluster. When we increased the number of machines by 30% (reaching 36 nodes), we could not increase the load at least by the same percentage. It was much less than expected. Any hints on what else to debug would be very appreciated!
Cluster Size: 27 nodes EC2 configurations:
machine type:m4.2xlarge cpu: 8 memory: 32 GB storage: EBS (1 TB)
Cassandra Version: 3.11.4 Replication factor : 3 Total Data: around 500GB (without replication) ( It was 180GB when we had only Customer Data)
