[FOLLOW UP QUESTION TO #4597]
Is there some ways to find out why we have so large mutations?
I have change log level to trace, and can find the following info.
But it is difficult to determine the cause of large mutations based on these logs.
Thanks.
TRACE [PerDiskMemtableFlushWriter_0:6] 2020-05-27 07:40:09,091 BigTableWriter.java:139 - wrote DecoratedKey(-8165226590013749090, 000fb2bb) at 46968468 TRACE [PerDiskMemtableFlushWriter_0:6] 2020-05-27 07:40:09,091 BigTableWriter.java:473 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@557039a7 at 191321 TRACE [CompactionExecutor:3] 2020-05-27 07:40:09,091 BigTableWriter.java:139 - wrote DecoratedKey(6717267206562729203, 00051626) at 667280616 TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 BigTableWriter.java:139 - wrote DecoratedKey(3947767368590274430, 000be6d3) at 78053649 TRACE [CompactionExecutor:3] 2020-05-27 07:40:09,091 BigTableWriter.java:473 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@1f1360f5 at 3287156 TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 BigTableWriter.java:473 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@3f369cd6 at 334920 ERROR [StreamReceiveTask:7] 2020-05-27 07:40:09,091 StreamSession.java:593 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Streaming error occurred on session with peer 10.233.141.136 java.lang.IllegalArgumentException: Mutation of 17.482MiB is too large for the maximum size of 16.000MiB at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:255) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:593) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:470) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:203) ~[apache-cassandra-3.11.3.jar:3.11.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.jar:3.11.3] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181] DEBUG [STREAM-OUT-/10.233.141.136:7000] 2020-05-27 07:40:09,091 ConnectionHandler.java:379 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Sending Session Failed DEBUG [StreamReceiveTask:7] 2020-05-27 07:40:09,091 StreamSession.java:472 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Finishing keep-alive task. TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 BigTableWriter.java:139 - wrote DecoratedKey(3947797804992188747, 000626d9) at 78056906 TRACE [PerDiskMemtableFlushWriter_0:5] 2020-05-27 07:40:09,091 BigTableWriter.java:473 - wrote index entry: org.apache.cassandra.db.RowIndexEntry@55149d1c at 334931 TRACE [PerDiskMemtableFlushWriter_0:6] 2020-05-27 07:40:09,091 BigTableWriter.java:139 - wrote DecoratedKey(-8165212536319642936, 000afe4b) at 46972086 DEBUG [StreamReceiveTask:7] 2020-05-27 07:40:09,091 ConnectionHandler.java:111 - [Stream #726a0370-9fec-11ea-8232-dfaf54f9beef] Closing stream connection handler on /10.233.141.136