Why are Cassandra nodes consuming more memory after upgrade to 3.11.6 from 2.2.8?

Question

question

12lovelysharma20_187429 asked · Nov 20, 2020 at 11:29 AM

Why are Cassandra nodes consuming more memory after upgrade to 3.11.6 from 2.2.8?

When I started my Cassandra cluster with version 3.11.6 the available memory was 15% and after 4,5 day it comes down to 10 % and gradually decreasing, do we need to change some setting there so that available memory is always remain constant around 14 to 15% . When I was running Cassandra cluster with version 2.2.8, the available memory was constant. Below are few details Cassandra is running with -Xms982M -Xmx982M -Xmn200M

-ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003 -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseNUMA -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xms982M -Xmx982M -Xmn200M

After certain time We are receiving following OOM errors

ERROR [CompactionExecutor:4] 2020-08-12 10:58:50,289 CassandraDaemon.java:195 - Exception in thread Thread[CompactionExecutor:4,1,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [CompactionExecutor:4] 2020-08-12 11:01:21,652 JVMStabilityInspector.java:118 - JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space

Please let me know what changes should we implement and what else is needed ?

cassandra

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-11-20T22:29:50Z

Erick Ramirez answered · Nov 20, 2020 at 10:29 PM

Without knowing much about your cluster, my initial reaction is that the heap size is way too small.

For light workloads, you should allocate at least 4GB to the heap but in production, the minimum should be 8GB for small applications with not a lot of users. To achieve this, the nodes need at least 16GB of RAM but 24-32GB is preferred.

If you're seeing off-heap memory being consumed, consider setting this in cassandra.yaml:

disk_access_mode: mmap_index_only

This will tell Cassandra to only mmap() index files and prevent data files from getting mapped to memory.

For more info, see this post I previously published -- Why nodes have increased memory use after upgrading Cassandra. Cheers!

4 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

12lovelysharma20_187429 · Nov 24, 2020 at 04:40 AM

In Cassandra 2.2.8 mmap_index_only was set to auto itself and no issue was faced related to heap. After upgrading to 3.11.6 we started facing the issue. Now after changing the node type to 4GB RAM the memory usage seems 50 % of the node, while -Xms982M -Xmx982M -Xmn200M for the last 10 days

0 ·

12lovelysharma20_187429 12lovelysharma20_187429 · Nov 24, 2020 at 04:41 AM

nodetool info tells the heap usage under 1000M

nodetool info

Heap Memory (MB) : 308.73 / 1004.00

Off Heap Memory (MB) : 26.20

Data Center : us-east

Rack : 1d

Exceptions : 1

Key Cache : entries 1594, size 175.99 KiB, capacity 50 MiB, 223583 hits, 240613 requests, 0.929 recent hit rate, 14400 save period in seconds

Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds

Counter Cache : entries 120, size 16.82 KiB, capacity 25 MiB, 427 hits, 491 requests, 0.870 recent hit rate, 7200 save period in seconds

Chunk Cache : entries 220, size 13.75 MiB, capacity 219 MiB, 163117 misses, 8115947 requests, 0.980 recent hit rate, NaN microseconds miss latency

What else should I try for checking exact heap usage by Cassandra?

0 ·

12lovelysharma20_187429 12lovelysharma20_187429 · Nov 24, 2020 at 04:48 AM

I am not so sure where the rest of the memory is getting consumed ? If you can guide me somewhere ?

0 ·

Erick Ramirez ♦♦ 12lovelysharma20_187429 · Nov 24, 2020 at 11:10 PM

How are you determining that Cassandra is using 50% of the memory?

0 ·

question