question

pooh avatar image
pooh asked Erick Ramirez commented

How do I optimize iowait due to performance testing?

I ran into a performance bottleneck with IO while doing Cassandra query stress tests

query qps reached about 2K, my machine IOWaIT basically reached 30%~40%

cluster info

  • C* version: 3.11.3
  • disk:4T*6 hhd
  • mem: 128G
  • node: 20

the amount of data per node is about 6T

jvm info:

  • -Xms16G
  • -Xmx32G
  • -Xss228k
  • -Xmn4G
  • -XX:MaxDirectMemorySize=32G
  • -Djdk.nio.maxCachedBufferSize=1048576

cache info:

  • key_cache_size_in_mb: 1024
  • key_cache_save_period: 14400
  • row_cache_size_in_mb: 1024
  • row_cache_save_period: 14400
  • counter_cache_size_in_mb: 0
  • file_cache_size_in_mb: 0

table config info:

Because I need to query two tables at the same time, and they have different compaction strategy

table1:

compaction = {'class': 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 'compaction_window_size': '10', 'compaction_window_unit': 'HOURS', 'max_threshold': '32', 'min_threshold': '4', 'unsafe_aggressive_sstable_expiration': 'true'}

compression = {'chunk_length_in_kb': '4', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

table2:

compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 'sstable_size_in_mb': '800', 'unchecked_tombstone_compaction': 'true'}

compression = {'chunk_length_in_kb': '4', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

I wonder how I need to optimize my configuration to achieve the best read/write performance of Cassandra without replacing SSD disks

performance
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

Unfortunately, there isn't much tuning you could do with spinning HDD disks apart from setting the readahead to 128 (see Optimize spinning disks).

If nodes are dense meaning they have more than 1TB of data, they will not perform well particularly with HDD. We recommend that you aim for a data density closer to 500GB per node.

And if you really want optimal performance, there is no substitute for SSDs but you will still need to have data density lower than 1TB.

As a side note, I assume you're using CMS because you've set a new generation size of 4GB. CMS doesn't perform well with large heap sizes. We recommend that you switch to G1 GC which performs well for heap sizes of 20GB and larger.

I also recommend that you don't allocate more than 31GB to the heap. 32GB heaps have less addressable objects than a 31GB heap. See Fabian Lange's blog post which has a nice explanation on this subject. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

pooh avatar image pooh commented ·

Because of the cost of SSD disks, they will not be replaced


There was no accurate estimate of disk capacity


There's one about heap mem, that's a good suggestion, thank you！


0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ pooh commented ·

I do understand the cost of using SSDs. But it's the price you pay for the bad performance of HDDs. :)

0 Likes 0 ·