question

rangumanikar_88470 avatar image
rangumanikar_88470 asked Erick Ramirez answered

Cluster was down with "Unable to gossip with any peers"

Hi team,

We have test cluster which is having 3 nodes in AWS cloud.Today we got to know that this cluster was down completely and tried to restart the services but failing with below error.

ERROR [main] 2021-12-20 10:14:57,180 CassandraDaemon.java:803 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1545) ~[apache-cassandra-3.11.10.jar:3.11.10]
    at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:621) ~[apache-cassandra-3.11.10.jar:3.11.10]
    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:880) ~[apache-cassandra-3.11.10.jar:3.11.10]
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:738) ~[apache-cassandra-3.11.10.jar:3.11.10]
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:687) ~[apache-cassandra-3.11.10.jar:3.11.10]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:395) [apache-cassandra-3.11.10.jar:3.11.10]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633) [apache-cassandra-3.11.10.jar:3.11.10]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786) [apache-cassandra-3.11.10.jar:3.11.10]

When i checked the configuration,Some of the nodes are having different seeds which are not part of cluster.I removed that node from seeds then tried to restart the services though it is failing with same error.

Issue got resolved after adding broadcast_address paramter to cassandra.yaml.All nodes are up.

How come node has come up after adding that parameter ? Since broadcast_address will be used if cluster is deployed across multi regions. But here this cluster is deployed only us-west-2 region.

How it is differ from listen_address ?

Regards,

Mani Rangu

gossip
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

Setting the broadcast_address means that the nodes are not able to communicate with each other using their listen_address.

The listen_address is what nodes use to talk to each other and in EC2 deployment, this is set to each node's private IP. The broadcast_address needs to be set to the nodes' public IP addresses only in a multi-region configuration since nodes cannot reach nodes in other regions using their private IPs. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.