Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

sunilrpawar7_183464 avatar image
sunilrpawar7_183464 asked ·

What could be the cause of 1 node out of 12 getting too many dropped mutations?

We have 12 node Cassandra cluster (3.11.2 version). Out of 12 nodes, one node is getting too much mutation dropped stats in nodetool tpstats command.

While looking into the logs we are getting frequent StatusLogger.java messages in the system.log.

The total number of connections on each individual node is the same. (netstats -na | grep 9041).

Application is not explicitly writing any data on this particular node and the average load on the system is also very much like other nodes.

What might be the other reasons for having too much mutation dropped message on only one node and not on others?

Why we get StatusLogger.java messages in the system.log?

Specifications:-

Xmx/Xms:- 31GB

Average load on each node:- 220GB.

RAM:- 128GB

Thanks,

Sunil

cassandradropped mutations
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

Dropped mutations are an indication that the commitlog disk cannot keep up with the write requests from the application. If you've confirmed that the configuration of all nodes are identical (both C* and server configurations), then I suspect the problem lies with the commitlog disk.

I would escalate the issue with your sysadmin and/or storage admins to investigate a possible impending hardware failure on that problematic node. Cheers!

Share
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.