This article discusses an issue where nodes in a cluster lose connectivity with each other during low traffic periods.
This article discusses an issue where nodes in a cluster lose connectivity with each other during low traffic periods.
In most cases, nodes in a cluster fail to communicate with each other because a firewall is closing the socket when it detects that the connection between 2 nodes is idle. By default, most firewalls are configured with a timeout period of 5 minutes.
We recommend setting TCP keepalive to 60 seconds with 3 probes every 10 seconds on every node in the cluster:
$ sudo sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10
These settings will detect dead TCP connections after 90 seconds (wait 60 seconds + send 3 probes every 10 seconds). The probes don't contain data so the additional traffic on the network is insignificant.
Note that you will need to consult the relevant documentation for your Linux distribution on how to persist these changes across reboots. Cheers!
5 People are following this question.
Unable to access web UI of OpsCenter running on GCP from a browser
How do I configure Cassandra to use public IP addresses?
Why can't nodes join the cluster when rpc_address and broadcast_address is set to the public IP?
How do I verify that the native transport port 9042 is open?
What is the minimum bandwidth required to operate a multi-DC cluster in different regions?
DataStax Enterprise is powered by the best distribution of Apache Cassandra ™
© 2023 DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.
Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.
Privacy Policy Terms of Use