I ran into a performance bottleneck with IO while doing Cassandra query stress tests
query qps reached about 2K, my machine IOWaIT basically reached 30%~40%
cluster info
- C* version: 3.11.3
- disk:4T*6 hhd
- mem: 128G
- node: 20
the amount of data per node is about 6T
jvm info:
- -Xms16G
- -Xmx32G
- -Xss228k
- -Xmn4G
- -XX:MaxDirectMemorySize=32G
- -Djdk.nio.maxCachedBufferSize=1048576
cache info:
- key_cache_size_in_mb: 1024
- key_cache_save_period: 14400
- row_cache_size_in_mb: 1024
- row_cache_save_period: 14400
- counter_cache_size_in_mb: 0
- file_cache_size_in_mb: 0
table config info:
Because I need to query two tables at the same time, and they have different compaction strategy
table1:
compaction = {'class': 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 'compaction_window_size': '10', 'compaction_window_unit': 'HOURS', 'max_threshold': '32', 'min_threshold': '4', 'unsafe_aggressive_sstable_expiration': 'true'}
compression = {'chunk_length_in_kb': '4', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
table2:
compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 'sstable_size_in_mb': '800', 'unchecked_tombstone_compaction': 'true'}
compression = {'chunk_length_in_kb': '4', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
I wonder how I need to optimize my configuration to achieve the best read/write performance of Cassandra without replacing SSD disks