Hi,
We are suffering from dropped mutations on a cluster that is writing about 150 million rows/day.
This is a 2 hour sample of the dropped mutations we have:
INFO [ScheduledTasks:1] 2020-02-18 08:02:56,908 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 3 internal and 258 cross node. Mean internal dropped latency: 2192 ms and Mean cross-node dropped latency: 2169 ms INFO [ScheduledTasks:1] 2020-02-18 08:07:32,116 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 396 cross node. Mean internal dropped latency: 2192 ms and Mean cross-node dropped latency: 2162 ms INFO [ScheduledTasks:1] 2020-02-18 08:10:47,252 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 2 internal and 414 cross node. Mean internal dropped latency: 2147 ms and Mean cross-node dropped latency: 2126 ms INFO [ScheduledTasks:1] 2020-02-18 08:17:17,610 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 63 cross node. Mean internal dropped latency: 2147 ms and Mean cross-node dropped latency: 2021 ms INFO [ScheduledTasks:1] 2020-02-18 08:25:02,864 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 13 cross node. Mean internal dropped latency: 2147 ms and Mean cross-node dropped latency: 2013 ms INFO [ScheduledTasks:1] 2020-02-18 08:32:32,993 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 1 internal and 423 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2024 ms INFO [ScheduledTasks:1] 2020-02-18 08:39:53,218 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 293 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2015 ms INFO [ScheduledTasks:1] 2020-02-18 08:47:38,413 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 108 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2018 ms INFO [ScheduledTasks:1] 2020-02-18 08:54:58,562 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 194 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2018 ms INFO [ScheduledTasks:1] 2020-02-18 09:02:28,767 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 1 internal and 123 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2084 ms INFO [ScheduledTasks:1] 2020-02-18 09:07:44,055 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 4 internal and 240 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2081 ms INFO [ScheduledTasks:1] 2020-02-18 09:13:49,388 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 189 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2092 ms INFO [ScheduledTasks:1] 2020-02-18 09:20:49,678 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 49 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2013 ms INFO [ScheduledTasks:1] 2020-02-18 09:28:35,212 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 1 internal and 48 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2013 ms INFO [ScheduledTasks:1] 2020-02-18 09:35:00,520 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 1 internal and 171 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2058 ms INFO [ScheduledTasks:1] 2020-02-18 09:42:35,663 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 3 internal and 132 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2025 ms INFO [ScheduledTasks:1] 2020-02-18 09:49:11,032 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 346 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2025 ms INFO [ScheduledTasks:1] 2020-02-18 09:56:11,311 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 37 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2019 ms INFO [ScheduledTasks:1] 2020-02-18 10:04:41,485 DroppedMessages.java:156 - MUTATION messages were dropped in the last 5 s: 0 internal and 262 cross node. Mean internal dropped latency: 2013 ms and Mean cross-node dropped latency: 2027 ms
As you can see, the amount of dropped messages are not so big, and most of them are cross-node.
I know that dropped mutations are something bad :( but looking at the logs, is it that bad in our current situation (150 Million/day)
Regards.
