Unable to connect to GOSSIP, reason: connection timed out

Question

This question was closed Oct 28, 2020 at 03:34 AM by Erick Ramirez for the following reason: The question is answered, right answer was accepted

question

ranjeet_ranjee asked Oct 19, '20 Erick Ramirez commented Oct 28, '20

Unable to connect to GOSSIP, reason: connection timed out

Hi, I am facing a Gossip error. Here you go

Gossip Setting inside cassandra.yaml both the nodes

cluster_name: tracking-1
listen_address: resepective both nodes public ip
broadcast_address: Cassandra-10 Public IP
broadcast_address: Cassandra-9 Public IP
listen_on_broadcast_address: true
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "Cassandra-9 Private IP,Cassandra-9-Public IP,Cassandra-10 Public IP,Cassandra-10-private IP"

DEBUG [NonPeriodicTasks:1] 2020-10-19 18:13:24,500  PullRequestScheduler.java:213 - Cancelling pull request to /Cassandra-10 for local schema version 092ca5d4-2b03-3804-adb2-112abb3ae18b.
DEBUG [NonPeriodicTasks:1] 2020-10-19 18:13:24,648  PullRequestScheduler.java:213 - Cancelling pull request to /Cassandra-10 for local schema version 092ca5d4-2b03-3804-adb2-112abb3ae18b.
DEBUG [CoreThread-1] 2020-10-19 18:13:24,819  OutboundMessagingConnection.java:461 - Unable to connect to GOSSIP with id 6 from /cassandra-9 to /Cassandra-10-private-ip via (/cassandra-9,/Cassandra-10-private-ip:7000), reason: connection timed out: /Cassandra-10-private-ip:7000
DEBUG [ScheduledFastTasks:1] 2020-10-19 18:13:25,020  OutboundMessagingConnection.java:396 - Connection attempt to GOSSIP with id 6 from /cassandra-9 to /Cassandra-10-private-ip via (/cassandra-9,/Cassandra-10-private-ip:7000)
DEBUG [ScheduledFastTasks:1] 2020-10-19 18:13:25,020  NettyFactory.java:231 - Creating outbound bootstrap to peer (/cassandra-9,/Cassandra-10-private-ip:7000), compression: false, encryption: disabled, coalesce: DISABLED

[UPDATE] Here you error after making changes to

broadcast_address: Public IP
listen_address: private_ip
native_transport_address: public_ip
listen_on_broadcast_address: true
native_transport_allow_older_protocols: true
native_transport_keepalive: true
native_transport_broadcast_address: public IP
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "private_ip_1, private_ip_2"

ERROR [DSE main thread] 2020-10-26 19:11:12,481  CassandraDaemon.java:901 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1772)
    at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:653)
    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:1035)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:796)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:730)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:405)
    at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:541)
    at org.apache.cassandra.service.CassandraDaemon.activate0(CassandraDaemon.java:733)
    at org.apache.cassandra.service.CassandraDaemon.access$100(CassandraDaemon.java:85)
    at org.apache.cassandra.service.CassandraDaemon$3.run(CassandraDaemon.java:694)
ERROR [StorageServiceShutdownHook] 2020-10-26 19:11:12,504  StorageService.java:5374 - Attempting to continue draining after pre-shutdown hooks returned exception
java.lang.AssertionError: null
    at org.apache.cassandra.gms.Gossiper.updateLocalApplicationState(Gossiper.java:1930)
    at com.datastax.bdp.gms.DseState.updateApplicationState(DseState.java:471)
    at com.datastax.bdp.gms.DseState.setApplicationState(DseState.java:446)
    at com.datastax.bdp.gms.DseState.setBooleanApplicationState(DseState.java:434)
    at com.datastax.bdp.gms.DseState.setActiveStatusSync(DseState.java:219)
    at com.datastax.bdp.server.DseDaemon.preShutdownHook(DseDaemon.java:884)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:162)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:153)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:147)
    at org.apache.cassandra.service.StorageService.drain(StorageService.java:5372)
    at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:767)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:88)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)

gossip

10 |1000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

DataStax Enterprise is powered by the best distribution of Apache Cassandra ™

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Privacy Policy Terms of Use

Answer 1 · 2020-10-21T00:09:09Z

Erick Ramirez answered Oct 21, '20 Erick Ramirez commented Oct 28, '20

The reason your nodes can't communicate with each other is that you've misconfigured your cluster.

Background

The listen_address is used for internode communication. A simpler way to explain it is that it is used by nodes to talk to each other privately so this is always set to the private IP of the server. Gossip port 7000 will be bound to this address.

If you have a multi-region cloud setup where nodes cannot communicate on their private IP across regions, set the broadcast_address to the public IP so intra-region communications go out on the WAN. Otherwise, do not set this property.

Applications/clients connect to the nodes on the native_transport_address. This needs to be set to the public IP so clients can talk to the cluster. CQL client port 9042 as well as all other public-facing ports will be bound to this address.

For a list of public and private ports used by DataStax Enterprise, see Securing DataStax Enterprise ports.

Seed nodes are used to bootstrap (kickstart) the gossip process so by definition, the seeds list needs to be configured with the private IPs of the seed nodes.

Correct configuration

Following on from my explanation above, you need to set the following properties in cassandra.yaml:

listen_address: private_ip
native_transport_address: public_ip

seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "private_ip_1, private_ip_2"

Additional info

The Thrift RPC service and native transport server were bound to the rpc_address. Thrift was deprecated in Apache Cassandra 2.2 (CASSANDRA-9319) and completely removed in Cassandra 4.0 (CASSANDRA-11115).

In anticipation of the full removal of Thrift, rpc_address was deprecated in DSE 6.0 and replaced with native_transport_address (DB-1130). The list of deprecated properties are:

Deprecated property	New from DSE 6.0
rpc_address	native_transport_address
rpc_interface	native_transport_interface
rpc_interface_prefer_ipv6	native_transport_interface_prefer_ipv6
rpc_port	native_transport_port
broadcast_rpc_address	native_transport_broadcast_address
rpc_keepalive	native_transport_keepalive

It's important to note that the properties have not been replaced yet in open-source Apache Cassandra (CASSANDRA-11107). Cheers!

18 Share

10 |1000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

ranjeet_ranjee · Oct 21, 2020 at 02:44 PM

Hi Erick,

I have updated the cassandra.yaml.

listen_address: private_ip
pnative_transport_address: public_ip

After updating, DSE getting shut down.

Just for information, we have only 2 nodes of the cluster.

0 ·

Erick Ramirez ♦♦ ranjeet_ranjee · Oct 22, 2020 at 01:07 AM

Are you sure it's not because you added a "p" in front of pnative_transport_address?

0 ·

ranjeet_ranjee Erick Ramirez ♦♦ · Oct 22, 2020 at 11:17 AM

I am sorry for typo but yes I am sure its

  
                      
                       listen_address: private_ip
                       native_transport_address: public_ip

0 ·

Show more comments

Erick Ramirez ♦♦ ranjeet_ranjee · Oct 26, 2020 at 05:19 AM

You need to provide info on the full error and full stack trace so we can assist. Cheers!

0 ·

ranjeet_ranjee Erick Ramirez ♦♦ · Oct 26, 2020 at 07:47 AM

I have updated my question with a full stack trace after 'some more error'.

0 ·

Show more comments

Erick Ramirez ♦♦ · Oct 28, 2020 at 02:22 AM

@ranjeet_ranjee not sure where the misunderstanding is but in my answer, there were only three properties that needed to be configured:

the listen address
the native transport address
the seeds list

Nothing else. I recommend you start over with a new and fresh configuration file and only configure the properties I indicated. Cheers!

0 ·

ranjeet_ranjee Erick Ramirez ♦♦ · Oct 28, 2020 at 02:34 AM

Do you mean to say I should remove it?

broadcast_address: Public IP
listen_on_broadcast_address: true
native_transport_allow_older_protocols: true
native_transport_keepalive: true
native_transport_broadcast_address: public IP

0 ·

Erick Ramirez ♦♦ ranjeet_ranjee · Oct 28, 2020 at 02:54 AM

Yes, like I said, you should only configure the three properties I specified. Cheers!

1 ·

Show more comments

question

Unable to connect to GOSSIP, reason: connection timed out

1 Answer

Background

Correct configuration

Additional info

question details

OTHER RESOURCES

question

Unable to connect to GOSSIP, reason: connection timed out

1 Answer

Background

Correct configuration

Additional info

question details

Related Questions

OTHER RESOURCES