Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

m_youssef010_51449 avatar image
m_youssef010_51449 asked ·

Backed up Gossip Messages when removing node

I’m getting strange gossip performance when removing a node using removenode.

Gossip messages backup (pending tasks > 100) and nodes start to be marked as DOWN.

Any ideas why this is happening?


Cluster details:

4 Cassandra DCs
37 nodes/DC

Cassandra Version 2.2.14

cassandragossipclusternoderemove
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

@m_youssef010_51449 in some large clusters with nodes which are quite dense (around 1TB of data or more), we see that some nodes start appearing offline because gossip tasks queue up. This can sometimes happen because the node being decommissioned is streaming lots of data to its neighbours and those nodes (new replicas) are overwhelmed by the streaming activity. The workaround is to restart Cassandra on the affected nodes. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.