question

swang_148208 avatar image
swang_148208 asked Erick Ramirez answered

How do I find out why there are large mutations in the cluster?

[FOLLOW UP QUESTION TO #4597]

Is there some ways to find out why we have so large mutations?

I have change log level to trace, and can find the following info.

But it is difficult to determine the cause of large mutations based on these logs.

Thanks.

TRACE [PerDiskMemtableFlushWriter_0:6] 2020-05-27 07:40:09,091 BigTableWriter.java:139 - wrote DecoratedKey(-8165226590013749090, 000fb2bb) at 46968468
TRACE [PerDiskMemtableFlushWriter_0:6] 2020-05-27 07:40:09,091 BigTableWriter.java:473 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@557039a7 at 191321
TRACE [CompactionExecutor:3] 2020-05-27 07:40:09,091 BigTableWriter.java:139 - wrote DecoratedKey(6717267206562729203, 00051626) at 667280616
TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 BigTableWriter.java:139 - wrote DecoratedKey(3947767368590274430, 000be6d3) at 78053649
TRACE [CompactionExecutor:3] 2020-05-27 07:40:09,091 BigTableWriter.java:473 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@1f1360f5 at 3287156
TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 BigTableWriter.java:473 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@3f369cd6 at 334920
ERROR [StreamReceiveTask:7] 2020-05-27 07:40:09,091 StreamSession.java:593 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Streaming error occurred on session with peer 10.233.141.136
java.lang.IllegalArgumentException: Mutation of 17.482MiB is too large for the maximum size of 16.000MiB
        at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:255) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:593) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:470) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:203) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.jar:3.11.3]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
DEBUG [STREAM-OUT-/10.233.141.136:7000] 2020-05-27 07:40:09,091 ConnectionHandler.java:379 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Sending Session Failed
DEBUG [StreamReceiveTask:7] 2020-05-27 07:40:09,091 StreamSession.java:472 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Finishing keep-alive task.
TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 BigTableWriter.java:139 - wrote DecoratedKey(3947797804992188747, 000626d9) at 78056906
TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 BigTableWriter.java:473 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@55149d1c at 334931
TRACE [PerDiskMemtableFlushWriter_0:6] 2020-05-27 07:40:09,091 BigTableWriter.java:139 - wrote DecoratedKey(-8165212536319642936, 000afe4b) at 46972086
DEBUG [StreamReceiveTask:7] 2020-05-27 07:40:09,091 ConnectionHandler.java:111 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Closing stream connection handler on /10.233.141.136
cassandra
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

@swang_148208 Those mutations were written by the application which is why I suggested that you discuss it with the application team to understand why you have large writes.

A large mutation could be the result of:

  • a big batch
  • an insert/update to a row that contains a large collection
  • an insert/update to a column that contains a massive amount of text
  • an insert/update with a very large blob

You'll need a good understanding of the data and the access patterns to work out how you end up with large mutations. You can't do troubleshoot it by looking at logs. You will need the help of the application team to get to the bottom of it. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.