PLANNED MAINTENANCE

Hello, DataStax Community!

We want to make you aware of a few operational updates which will be carried out on the site. We are working hard to streamline the login process to integrate with other DataStax resources. As such, you will soon be prompted to update your password. Please note that your username will remain the same.

As we work to improve your user experience, please be aware that login to the DataStax Community will be unavailable for a few hours on:

  • Wednesday, July 15 16:00 PDT | 19:00 EDT | 20:00 BRT
  • Thursday, July 16 00:00 BST | 01:00 CEST | 04:30 IST | 07:00 CST | 09:00 AEST

For more info, check out the FAQ page. Thank you for being a valued member of our community.


question

Erick Ramirez avatar image
Erick Ramirez asked ·

Why can't nodes gossip sometimes when cassandra-topology.properties is loaded with GossipingPropertyFileSnitch?

Nodes are intermittently unable to gossip properly when configured with the GossipingPropertyFileSnitch.

One symptom is nodes randomly going up and down for no apparent reason.

INFO  [GossipTasks:1] 2016-04-29 02:47:32,559  Gossiper.java:1001 - InetAddress /10.1.2.3 is now DOWN
INFO  [GossipTasks:1] 2016-04-29 02:50:47,123  Gossiper.java:1001 - InetAddress /10.1.2.4 is now DOWN
INFO  [GossipTasks:1] 2016-04-29 02:54:59,640  Gossiper.java:1001 - InetAddress /10.1.2.5 is now DOWN
INFO  [SharedPool-Worker-2] 2016-04-29 03:01:23,828  Gossiper.java:987 - InetAddress /10.1.2.4 is now UP
INFO  [SharedPool-Worker-1] 2016-04-29 03:01:59,432  Gossiper.java:987 - InetAddress /10.1.2.5 is now UP
INFO  [SharedPool-Worker-7] 2016-04-29 03:02:01,839  Gossiper.java:987 - InetAddress /10.1.2.3 is now UP

Similarly, different nodes appear to be down in the nodetool status output depending on where it was ran, for example:

Datacenter: Cassandra
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address   Load       Tokens  Owns    Host ID                               Rack
DN  10.1.2.3  8.97 GB    256     ?       a50dfef5-229d-4d15-89d9-971bec01094b  rack1
UN  10.1.2.5  8.9 GB     256     ?       a16b71a2-9b95-4669-a6bd-d7326bd279e2  rack1
DN  10.1.2.4  9.09 GB    256     ?       ac01b6f9-3cb9-47ff-83c6-0404836386eb  rack1
UN  10.1.2.6  10.65 GB   256     ?       9c0ef3a2-aad7-4d06-b015-f32ddccac750  rack1
cassandragossipsnitch
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

Cause

The problem has only been seen in a very small number of clusters and is still under investigation. However, it has been identified that the problem occurs when the cassandra-topology.properties exists while nodes are configured with GossipingPropertyFileSnitch:

INFO  [main] 2016-04-29 15:31:26,039  GossipingPropertyFileSnitch.java:71 - Loaded cassandra-topology.properties for compatibility

It is important to note that the issue is very intermittent and not all vectors which trigger the problem are known yet.

Workaround

By design, the GossipingPropertyFileSnitch falls back on the PropertyFileSnitch's cassandra-topology.properties as a means to allow clusters to be migrated to GossipingPropertyFileSnitch.

If the cluster is already on GossipingPropertyFileSnitch, ensure that cassandra-topology.properties has been removed or does not exist even if there are no issues with the nodes to ensure the cluster does not encounter problems in the future.

See also

Cassandra JIRA - CASSANDRA-11508 GPFS property file should more clearly explain the relationship with PFS

Credits

This was originally published in the DataStax Support Knowledge Base as Intermittent gossip issues when using cassandra-topology.properties with GossipingPropertyFileSnitch.

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.