question

kranthij29_188881 avatar image
kranthij29_188881 asked Erick Ramirez edited

Why does nodetool status/ring return different outputs between nodes?

nodetool status /nodetool ring is giving diff output on all the nodes. any idea why it is ?

-bash-4.2$ dse -v
6.8.0
-bash-4.2$ nodetool statusgossip
not running

gossiping is not running on one of the node.

node 3:

-bash-4.2$ nodetool ring
Datacenter: xxxxxx
==================
Address     Rack  Status State  Load       Owns Token
                                                7160673013025134991
xx.xx.48.74 rack1 Down   Normal 668.01 GiB ?    -9223372036854775808
xx.xx.48.18 rack1 Down   Normal 473.28 GiB ?    -8051969637187861068
xx.xx.48.78 rack1 Up     Normal 627.21 GiB ?    -1170601027052494348
xx.xx.48.76 rack1 Down   Normal 800.32 GiB ?    3074457345618258602
xx.xx.48.77 rack1 Up     Normal  790.8 GiB ?    7160673013025134991

nod 5 :

-bash-4.2$ nodetool ring
Datacenter: xxxxxxxx
====================
Address     Rack  Status State  Load       Owns Token
                                                7160673013025134991
xx.xx.48.74 rack1 Down   Normal 686.76 GiB ?    -9223372036854775808
xx.xx.48.18 rack1 Up     Normal 493.26 GiB ?    -8051969637187861068
xx.xx.48.78 rack1 Up     Normal 654.18 GiB ?    -1170601027052494348
xx.xx.48.76 rack1 Up     Normal 832.96 GiB ?    3074457345618258602
xx.xx.48.77 rack1 Down   Normal 790.36 GiB ?    7160673013025134991

node2:

-bash-4.2$ nodetool ring
Datacenter: xxxxx
=================
Address     Rack  Status State  Load       Owns Token
                                                7160673013025134991
xx.xx.48.74 rack1 Down   Normal 686.76 GiB ?    -9223372036854775808
xx.xx.48.18 rack1 Down   Normal 493.26 GiB ?    -8051969637187861068
xx.xx.48.78 rack1 Up     Normal 654.18 GiB ?    -1170601027052494348
xx.xx.48.76 rack1 Up     Normal 832.96 GiB ?    3074457345618258602
xx.xx.48.77 rack1 Down   Normal 790.36 GiB ?    7160673013025134991
cassandranodetool
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

@kranthij29_188881 The differences in the outputs is a result of nodes not being able to communicate with each other via the gossip protocol. You need to ensure that gossip is operational on all nodes because it's critical for the operation of the cluster.

On each node, check that DSE is listening on all ports. Here are primary ones if you're just running Cassandra:

$ netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:7199          0.0.0.0:*               LISTEN      18696/java      
tcp        0      0 10.100.22.333:8609      0.0.0.0:*               LISTEN      18696/java      
tcp        0      0 10.100.22.333:9042      0.0.0.0:*               LISTEN      18696/java      
tcp        0      0 10.100.22.333:7000      0.0.0.0:*               LISTEN      18696/java

For a complete list of ports used by DSE and the associated services, see Securing DataStax Enterprise ports. Cheers!

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

kranthij29_188881 avatar image kranthij29_188881 commented ·

@Erick RamirezThanks for your reply, i have brought down the cluster, and cleared other processes running by Cassandra user and started the cluster again which resolved the issue. Thanks.

1 Like 1 ·