Hi all,
We are facing some problems in the scaling of our nodes, previously we are having 7 nodes in our cluster. we have planned to add 7 nodes to our cluster, we are adding one by one node and successfully added 4 nodes and while adding the 5th node the bootstrap process started and after 36+ hrs the node was not part of a cluster, there is no connectivity for that node and nodetool commands are not working and checked storage and it was using 1.5 TB out of 3 TB like other nodes.
So what are steps need to follow to get back that node?
below type of logs are there in system.log
2021-08-23 04:48:36:309*[ERROR]*STREAM-OUT-/10.24.13.107:7000*o.a.c.s.StreamSession*logError*[Stream #07aaa190-0376-11ec-9159-45ce30811155] Streaming error occurred on session with peer 10.24.13.107 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_181] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_181] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_181] at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[na:1.8.0_181] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_181] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.doFlush(BufferedDataOutputStreamPlus.java:323) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.flush(BufferedDataOutputStreamPlus.java:331) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:409) [apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:380) [apache-cassandra-3.11.3.jar:3.11.3] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]