I am trying to restart cluster on the same machines used for one deployment.
Process I am following is as follows:
1. I am loading data in the cluster of x nodes. For example, 4 nodes
2. I let the replication & streaming complete between the nodes as I am using RF=3
3. Once the nodes are equally balanced, I run `nodetool drain` and kill the cassandra process running on the node starting from the 4th node, keeping seed node as a last.
4. Now I am trying to restart the cluster, starting with seed node. Seed node just starts fine but the other nodes.
Errors on every other nodes are as follows:
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /x.x.x.x:7000 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) at io.netty.channel.unix.Socket.finishConnect(Socket.java:251) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:673) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any peers java.lang.RuntimeException: Unable to gossip with any peers at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805) at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648) at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:784) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:729) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887) ERROR [main] 2021-11-14 13:50:33,900 CassandraDaemon.java:909 - Exception encountered during startup java.lang.RuntimeException: Unable to gossip with any peers at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805) at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648) at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:784) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:729) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
And at the end of log, it says like this:
INFO [MemtableFlushWriter:4] 2021-11-14 13:50:38,374 LogTransaction.java:240 - Unfinished transaction log, deleting /home/ubuntu/cassandra_dc1_rack1_node2_data/system_schema/functions-96489b7980be3e14a70166a0b9159450/nb_txn_flush_d9600960-4551-11ec-ba00-a776ea0a5340.log INFO [StorageServiceShutdownHook] 2021-11-14 13:50:38,379 HintsService.java:220 - Paused hints dispatch
I am using `GossipingPropertyFileSnitch` and have removed `cassandra-topology.properties` file.
I tried several times starting from scratch, fresh but every time when I try to restart, it fails. My main goal is to restart cluster in normal, balanced & query ready state.
Any help would be greatly appreciated. Thank you.