Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

satvantsingh_190085 avatar image
satvantsingh_190085 asked ·

How long does it take for nodes to mark another node as down?

A node is working fine but due some network issue any particular node not reachable , How soon others node mark that particular node as down.

I know gossip communication run every second for state information exchange , but I want to know is there any time interval when a particular node becomes down for other node permanently ?

cassandranetwork partition
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

It happens pretty quickly, usually within a few seconds. "A few" being low single-digit seconds.

What typically happens is that a node will gossip with 3 other nodes every second. One of those nodes is a seed node. Nodes always gossip with at least one seed node in order for gossip information to converge very quickly.

Imagine you live on a street with 10 other neighbours. Your neighbour at house #5 is a gossipmonger, a person who spreads gossip about other people. If every person on the street gossips with any 2 other neighbours, news will spread relatively slowly. But if every person always gossip with house #5 plus 1 other neighbour, the news will spread really fast because everyone ends up telling house #5 all the street news and this neighbour passes it on to everyone else.

The same thing happens for a down node -- a node which tries to gossip with it will quickly tell a seed node that its down and the seed node will quickly spread it to other seed nodes plus the rest of the nodes. Cassandra uses an algorithm to detect down nodes and changes in the cluster topology so that requests don't get routed to nodes which are down, overloaded or unresponsive.

For more info, see Failure detection and recovery. Cheers!

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Thank you @Erick Ramirez !

0 Likes 0 · ·