question

mohdnazeem1 avatar image
mohdnazeem1 asked Erick Ramirez answered

Understanding Dropped mutations message

how do I read dropped mutations error messages - what is internal and cross node? For mutations it fails on cross-node and read_repair/read it fails on internal. What does it mean?

INFO  [ScheduledTasks:1] 2019-07-21 11:44:46,150  MessagingService.java:1281 - MUTATION messages were dropped in last 5000 ms: 0 internal and 65 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 4966 ms
INFO  [ScheduledTasks:1] 2019-07-19 05:01:10,620  MessagingService.java:1281 - READ_REPAIR messages were dropped in last 5000 ms: 9 internal and 8 cross node. Mean internal dropped latency: 6013 ms and Mean cross-node dropped latency: 8164 ms

Does internal mean local node operations - in this case mutation response from local node and cross node means the time it took to get response back from other nodes depending on the consistency level choosen?

What does Read and _Trace dropped mutations mean? There is no tracing enabled on any node in the cluster, what are these _TRACE dropped messages?

INFO  [ScheduledTasks:1] 2019-07-25 21:17:13,878  MessagingService.java:1281 - READ messages were dropped in last 5000 ms: 1 internal and 0 cross node. Mean internal dropped latency: 5960 ms and Mean cross-node dropped latency: 0 ms
INFO  [ScheduledTasks:1] 2019-07-25 20:38:43,788  MessagingService.java:1281 - _TRACE messages were dropped in last 5000 ms: 5035 internal and 0 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 0 ms
cassandra
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

@mohdnazeem1 yes, the internal count is the number of requests dropped for the node itself while the cross node count is for replicas. There are different types of messages in Cassandra:

  • MUTATION - write requests including INSERT, UPDATE and DELETE
  • READ - read requests (SELECT)
  • _TRACE - queries where tracing is enabled

The 5K dropped _TRACE messages in your example indicates tracing is enabled server-side. Disable tracing with nodetool settraceprobability -- 0. This is different from client-side TRACING. Cheers!

Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.