Why can't nodes join the cluster when rpc_address and broadcast_address is set to the public IP?

Question

question

vikasgupta20_84283 asked · Jul 10, 2020 at 09:18 PM

Why can't nodes join the cluster when rpc_address and broadcast_address is set to the public IP?

I have asked this question before and replied by @Erick Ramirez here but I have follow up question. As suggested I updated.

broadcast_address: public_ip listen_address: private_ip rpc_address: public_ip

I dnt see native_transport_address in cassandra.yaml file so I am assuming you are talking about rpc_address. also seeds are set to public IP address but after making this change nodes are not joining the cluster. Every individual node shows up when running nodetool status on this node.

With previous configuration where listen_address: and rpc_address: was left empty and broadcast_address: section was commented, All nodes were showing up in same cluster with private IP's.

Could you please suggest me if any other changes needs to be done.

cassandra network

nodes.png (26.9 KiB)

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-07-11T10:38:39Z

Erick Ramirez answered · Jul 11, 2020 at 10:38 AM

It's impossible to guess the root cause of your issue because you haven't provided sufficient information and you've obfuscated the IP addresses the nodes are using.

My guess is that the IP addresses you've configured are invalid which is why the nodes cannot communicate with each other.

In question #6019, you've stated you're using the same public IPs on all nodes:

Currently I took public IP of one node and set this value for whole cluster...

As I advised in that post, a server cannot use an IP address it does not own. This is really a networking issue and not a Cassandra issue. I would recommend that you discuss your problem with your sysadmin team so they can assist you with working out which IP addresses you should use. Cheers!

3 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

vikasgupta20_84283 · Jul 12, 2020 at 05:50 AM

Thanks for your response. I wanted to check in case if I am doing some obvious mistake. Also as you suggested last time I have updated all cassnadra.yaml file with its own public IP's.
Nodetool screen shot from Node 1 in Rack 100 : Screen shot from Node 3 in Rack 300 :

Based on configuration all 4 nodes should join the same cluster and show under same data center with public IP address but now each individual nodes shows status UN with its public IP address and other nodes shows as ?N with its their private IP address.

0 ·

node1.png (21.1 KiB)

node1.png (21.0 KiB)

node3.png (40.7 KiB)

Erick Ramirez ♦♦ vikasgupta20_84283 · Jul 12, 2020 at 06:06 AM

When a node places other nodes in the default DC1 DC and r1 rack, it means that it can't gossip with other nodes. These are coming from this entry in cassandra-topology.properties:

# default for unknown nodes
default=DC1:r1

This is a networking issue -- the IPs you've configured are not routable between nodes. Cheers!

0 ·

vikasgupta20_84283 Erick Ramirez ♦♦ · Jul 17, 2020 at 05:38 PM

Thanks for your reply. Yes, it was a networking issue and it started working after resolving that issue. but I see intermittent behavior now. Sometimes all nodes come up joining the same cluster and sometime I see error : "nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused: connect'."
Do I need to make any changes to line :

# $env:JVM_OPTS="$env:JVM_OPTS -Djava.rmi.server.hostname=<public name>"

0 ·

question