question

vikasgupta20_84283 avatar image
vikasgupta20_84283 asked miranda commented

Why can't nodes join the cluster when rpc_address and broadcast_address is set to the public IP?

I have asked this question before and replied by @Erick Ramirez here but I have follow up question. As suggested I updated.

broadcast_address: public_ip
listen_address: private_ip
rpc_address: public_ip

I dnt see native_transport_address in cassandra.yaml file so I am assuming you are talking about rpc_address. also seeds are set to public IP address but after making this change nodes are not joining the cluster. Every individual node shows up when running nodetool status on this node.

With previous configuration where listen_address: and rpc_address: was left empty and broadcast_address: section was commented, All nodes were showing up in same cluster with private IP's.

Could you please suggest me if any other changes needs to be done.

cassandranetwork
nodes.png (26.9 KiB)
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered miranda commented

It's impossible to guess the root cause of your issue because you haven't provided sufficient information and you've obfuscated the IP addresses the nodes are using.

My guess is that the IP addresses you've configured are invalid which is why the nodes cannot communicate with each other.

In question #6019, you've stated you're using the same public IPs on all nodes:

Currently I took public IP of one node and set this value for whole cluster...

As I advised in that post, a server cannot use an IP address it does not own. This is really a networking issue and not a Cassandra issue. I would recommend that you discuss your problem with your sysadmin team so they can assist you with working out which IP addresses you should use. Cheers!

4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

vikasgupta20_84283 avatar image vikasgupta20_84283 commented ·

Thanks for your response. I wanted to check in case if I am doing some obvious mistake. Also as you suggested last time I have updated all cassnadra.yaml file with its own public IP's.
Nodetool screen shot from Node 1 in Rack 100 : Screen shot from Node 3 in Rack 300 :

Based on configuration all 4 nodes should join the same cluster and show under same data center with public IP address but now each individual nodes shows status UN with its public IP address and other nodes shows as ?N with its their private IP address.

0 Likes 0 ·
node1.png (21.1 KiB)
node1.png (21.0 KiB)
node3.png (40.7 KiB)
Erick Ramirez avatar image Erick Ramirez ♦♦ vikasgupta20_84283 commented ·

When a node places other nodes in the default DC1 DC and r1 rack, it means that it can't gossip with other nodes. These are coming from this entry in cassandra-topology.properties:

# default for unknown nodes
default=DC1:r1

This is a networking issue -- the IPs you've configured are not routable between nodes. Cheers!

0 Likes 0 ·
vikasgupta20_84283 avatar image vikasgupta20_84283 Erick Ramirez ♦♦ commented ·

Thanks for your reply. Yes, it was a networking issue and it started working after resolving that issue. but I see intermittent behavior now. Sometimes all nodes come up joining the same cluster and sometime I see error : "nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused: connect'."
Do I need to make any changes to line :

# $env:JVM_OPTS="$env:JVM_OPTS -Djava.rmi.server.hostname=<public name>"

0 Likes 0 ·
Show more comments