Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

ranjeet_ranjee avatar image
ranjeet_ranjee asked ·

Unable to connect to GOSSIP, reason: connection timed out

Hi, I am facing a Gossip error. Here you go

Gossip Setting inside cassandra.yaml both the nodes

cluster_name: tracking-1
listen_address: resepective both nodes public ip
broadcast_address: Cassandra-10 Public IP
broadcast_address: Cassandra-9 Public IP
listen_on_broadcast_address: true
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "Cassandra-9 Private IP,Cassandra-9-Public IP,Cassandra-10 Public IP,Cassandra-10-private IP"
DEBUG [NonPeriodicTasks:1] 2020-10-19 18:13:24,500  PullRequestScheduler.java:213 - Cancelling pull request to /Cassandra-10 for local schema version 092ca5d4-2b03-3804-adb2-112abb3ae18b.
DEBUG [NonPeriodicTasks:1] 2020-10-19 18:13:24,648  PullRequestScheduler.java:213 - Cancelling pull request to /Cassandra-10 for local schema version 092ca5d4-2b03-3804-adb2-112abb3ae18b.
DEBUG [CoreThread-1] 2020-10-19 18:13:24,819  OutboundMessagingConnection.java:461 - Unable to connect to GOSSIP with id 6 from /cassandra-9 to /Cassandra-10-private-ip via (/cassandra-9,/Cassandra-10-private-ip:7000), reason: connection timed out: /Cassandra-10-private-ip:7000
DEBUG [ScheduledFastTasks:1] 2020-10-19 18:13:25,020  OutboundMessagingConnection.java:396 - Connection attempt to GOSSIP with id 6 from /cassandra-9 to /Cassandra-10-private-ip via (/cassandra-9,/Cassandra-10-private-ip:7000)
DEBUG [ScheduledFastTasks:1] 2020-10-19 18:13:25,020  NettyFactory.java:231 - Creating outbound bootstrap to peer (/cassandra-9,/Cassandra-10-private-ip:7000), compression: false, encryption: disabled, coalesce: DISABLED

[UPDATE] Here you error after making changes to

broadcast_address: Public IP
listen_address: private_ip
native_transport_address: public_ip
listen_on_broadcast_address: true
native_transport_allow_older_protocols: true
native_transport_keepalive: true
native_transport_broadcast_address: public IP
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "private_ip_1, private_ip_2"
ERROR [DSE main thread] 2020-10-26 19:11:12,481  CassandraDaemon.java:901 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1772)
    at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:653)
    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:1035)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:796)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:730)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:405)
    at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:541)
    at org.apache.cassandra.service.CassandraDaemon.activate0(CassandraDaemon.java:733)
    at org.apache.cassandra.service.CassandraDaemon.access$100(CassandraDaemon.java:85)
    at org.apache.cassandra.service.CassandraDaemon$3.run(CassandraDaemon.java:694)
ERROR [StorageServiceShutdownHook] 2020-10-26 19:11:12,504  StorageService.java:5374 - Attempting to continue draining after pre-shutdown hooks returned exception
java.lang.AssertionError: null
    at org.apache.cassandra.gms.Gossiper.updateLocalApplicationState(Gossiper.java:1930)
    at com.datastax.bdp.gms.DseState.updateApplicationState(DseState.java:471)
    at com.datastax.bdp.gms.DseState.setApplicationState(DseState.java:446)
    at com.datastax.bdp.gms.DseState.setBooleanApplicationState(DseState.java:434)
    at com.datastax.bdp.gms.DseState.setActiveStatusSync(DseState.java:219)
    at com.datastax.bdp.server.DseDaemon.preShutdownHook(DseDaemon.java:884)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:162)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:153)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:147)
    at org.apache.cassandra.service.StorageService.drain(StorageService.java:5372)
    at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:767)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:88)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
gossip
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

The reason your nodes can't communicate with each other is that you've misconfigured your cluster.

Background

The listen_address is used for internode communication. A simpler way to explain it is that it is used by nodes to talk to each other privately so this is always set to the private IP of the server. Gossip port 7000 will be bound to this address.

If you have a multi-region cloud setup where nodes cannot communicate on their private IP across regions, set the broadcast_address to the public IP so intra-region communications go out on the WAN. Otherwise, do not set this property.

Applications/clients connect to the nodes on the native_transport_address. This needs to be set to the public IP so clients can talk to the cluster. CQL client port 9042 as well as all other public-facing ports will be bound to this address.

For a list of public and private ports used by DataStax Enterprise, see Securing DataStax Enterprise ports.

Seed nodes are used to bootstrap (kickstart) the gossip process so by definition, the seeds list needs to be configured with the private IPs of the seed nodes.

Correct configuration

Following on from my explanation above, you need to set the following properties in cassandra.yaml:

listen_address: private_ip
native_transport_address: public_ip
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "private_ip_1, private_ip_2"

Additional info

The Thrift RPC service and native transport server were bound to the rpc_address. Thrift was deprecated in Apache Cassandra 2.2 (CASSANDRA-9319) and completely removed in Cassandra 4.0 (CASSANDRA-11115).

In anticipation of the full removal of Thrift, rpc_address was deprecated in DSE 6.0 and replaced with native_transport_address (DB-1130). The list of deprecated properties are:

Deprecated property New from DSE 6.0
rpc_address native_transport_address
rpc_interface native_transport_interface
rpc_interface_prefer_ipv6 native_transport_interface_prefer_ipv6
rpc_port native_transport_port
broadcast_rpc_address native_transport_broadcast_address
rpc_keepalive native_transport_keepalive

It's important to note that the properties have not been replaced yet in open-source Apache Cassandra (CASSANDRA-11107). Cheers!

18 comments Share
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

ranjeet_ranjee avatar image ranjeet_ranjee ·

Hi Erick,


I have updated the cassandra.yaml.

listen_address: private_ip
pnative_transport_address: public_ip


After updating, DSE getting shut down.


Just for information, we have only 2 nodes of the cluster.

0 Likes 0 · ·
Erick Ramirez avatar image Erick Ramirez ♦♦ ranjeet_ranjee ·

Are you sure it's not because you added a "p" in front of pnative_transport_address?

0 Likes 0 · ·
ranjeet_ranjee avatar image ranjeet_ranjee Erick Ramirez ♦♦ ·

I am sorry for typo but yes I am sure its 

  
                      
    
                       
  1. listen_address: private_ip
    2. 
                       
  2. native_transport_address: public_ip
    3.


0 Likes 0 · ·
Show more comments
Erick Ramirez avatar image Erick Ramirez ♦♦ ranjeet_ranjee ·

You need to provide info on the full error and full stack trace so we can assist. Cheers!

0 Likes 0 · ·
ranjeet_ranjee avatar image ranjeet_ranjee Erick Ramirez ♦♦ ·

I have updated my question with a full stack trace after 'some more error'.

0 Likes 0 · ·
Show more comments
Erick Ramirez avatar image Erick Ramirez ♦♦ ·

@ranjeet_ranjee not sure where the misunderstanding is but in my answer, there were only three properties that needed to be configured:

  1. the listen address
  2. the native transport address
  3. the seeds list

Nothing else. I recommend you start over with a new and fresh configuration file and only configure the properties I indicated. Cheers!

0 Likes 0 · ·
ranjeet_ranjee avatar image ranjeet_ranjee Erick Ramirez ♦♦ ·

Do you mean to say I should remove it?

broadcast_address: Public IP
listen_on_broadcast_address: true
native_transport_allow_older_protocols: true
native_transport_keepalive: true
native_transport_broadcast_address: public IP
0 Likes 0 · ·
Erick Ramirez avatar image Erick Ramirez ♦♦ ranjeet_ranjee ·

Yes, like I said, you should only configure the three properties I specified. Cheers!

1 Like 1 · ·
Show more comments