swang_148208 avatar image
swang_148208 asked Erick Ramirez answered

How do I find out why there are large mutations in the cluster?


Is there some ways to find out why we have so large mutations?

I have change log level to trace, and can find the following info.

But it is difficult to determine the cause of large mutations based on these logs.


TRACE [PerDiskMemtableFlushWriter_0:6] 2020-05-27 07:40:09,091 - wrote DecoratedKey(-8165226590013749090, 000fb2bb) at 46968468
TRACE [PerDiskMemtableFlushWriter_0:6] 2020-05-27 07:40:09,091 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@557039a7 at 191321
TRACE [CompactionExecutor:3] 2020-05-27 07:40:09,091 - wrote DecoratedKey(6717267206562729203, 00051626) at 667280616
TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 - wrote DecoratedKey(3947767368590274430, 000be6d3) at 78053649
TRACE [CompactionExecutor:3] 2020-05-27 07:40:09,091 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@1f1360f5 at 3287156
TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@3f369cd6 at 334920
ERROR [StreamReceiveTask:7] 2020-05-27 07:40:09,091 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Streaming error occurred on session with peer
java.lang.IllegalArgumentException: Mutation of 17.482MiB is too large for the maximum size of 16.000MiB
        at org.apache.cassandra.db.commitlog.CommitLog.add( ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.db.Keyspace.applyInternal( ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.db.Keyspace.apply( ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.StreamReceiveTask$ ~[apache-cassandra-3.11.3.jar:3.11.3]
        at java.util.concurrent.Executors$ [na:1.8.0_181]
        at [na:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor.runWorker( [na:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$ [na:1.8.0_181]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0( [apache-cassandra-3.11.3.jar:3.11.3]
        at ~[na:1.8.0_181]
DEBUG [STREAM-OUT-/] 2020-05-27 07:40:09,091 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Sending Session Failed
DEBUG [StreamReceiveTask:7] 2020-05-27 07:40:09,091 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Finishing keep-alive task.
TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 - wrote DecoratedKey(3947797804992188747, 000626d9) at 78056906
TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@55149d1c at 334931
TRACE [PerDiskMemtableFlushWriter_0:6] 2020-05-27 07:40:09,091 - wrote DecoratedKey(-8165212536319642936, 000afe4b) at 46972086
DEBUG [StreamReceiveTask:7] 2020-05-27 07:40:09,091 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Closing stream connection handler on /
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

@swang_148208 Those mutations were written by the application which is why I suggested that you discuss it with the application team to understand why you have large writes.

A large mutation could be the result of:

  • a big batch
  • an insert/update to a row that contains a large collection
  • an insert/update to a column that contains a massive amount of text
  • an insert/update with a very large blob

You'll need a good understanding of the data and the access patterns to work out how you end up with large mutations. You can't do troubleshoot it by looking at logs. You will need the help of the application team to get to the bottom of it. Cheers!

10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.