question

phofegger_148429 avatar image
phofegger_148429 asked Erick Ramirez commented

Why would nodes be getting "Seed gossip version is -2147483648" while adding a DC?

Hi,

I am adding a 3rd DC to the existing cassandra cluster frp migration stuff. I have updated seed hosts in the cassandra.yaml. I restartet all nodes with the new yaml file and then i started one of the new seed node and i get following WARN Message .

WARN [MessagingService-Outgoing-/192.x.x.x-Small] 2020-06-04 09:49:16,715 OutboundTcpConnection.java:436 - Seed gossip version is -2147483648; will not connect with that version.

I did a nodetool status and the new nodes is visible with Up / Normal.

I tested the connection between the nodes that should be ok.

Any Ideas many thanks in advance

Cheers

Patrick

cassandragossip
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

The warning happens because the node is not able to get the real version of gossip. Notice that -2147483648 is the minimum value possible for the Java integer type. In my experience, this is likely caused by a firewall either closing the socket or truncating the connection between the node in the new DC and the seed node.

What happens is that the nodes connect with each other then during the handshake the new node tries to get the gossip version from the seed but the connection gets interrupted/truncated (for whatever reason) and no gossip version is returned so it defaults to NO_VERSION which is:

     private static final int NO_VERSION = Integer.MIN_VALUE;

For this to happen, the firewall is not blocking the traffic. You can even confirm it with tools like telnet or nc to port 7000 (gossip port). The problem lies in what happens during the handshake -- if the firewall interferes with the connection, the handshake fails.

I'm leaning towards this being a network connectivity issue because I know from your previous posts that you are adding nodes in a new physical DC. I'd engage your network admin to assist you with the investigation. Cheers!

3 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

phofegger_148429 avatar image phofegger_148429 commented ·

Hi Erick, thank you for the answer. I checked the connection on both side. I can connect to port 7000 from the new seed to the existing node and vice versa. The connection to DC2 works but the connection to DC1 fail. i don't know. I have contacted our Network Admin.


0 Likes 0 ·
phofegger_148429 avatar image phofegger_148429 commented ·

Hi Erick, thank you for your hint. You are right, it was a network issue. I have set another route and now it works. the first route pointed to a low connection. It was weird because I have been able to connect to port 7000 on all nodes. However... I will continue with the migration

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ phofegger_148429 commented ·

That's good to hear. Cheers!

0 Likes 0 ·