We are in the process of scaling the cluster from 30 nodes to about 40.
Configuration for bootstrap:
auto_bootstrap: true (Default)-Dcassandra.consistent.rangemovement=false (Default)
The bootstrap seems to be complete with all the nodes streaming data to this new node. Yet, the node seems to stay in JOINING state and bootstrapping eventually is timing out (after 3 hrs; streaming_socket_timeout_in_ms). This is an inconsistent state with the new node being stuck in UJ state forever.I tried nodetool bootstrap resume which also hangs indefinitely. I checked nodetool netstats and none of the nodes are streaming data to the new node.
Now, since I know that this node has all the data that belongs to it, I try to add auto_bootstrap: false in the cassandra.yaml and restart cassandra process. My expectation was that adding auto_bootstrap: false will not bother about streaming data from other nodes, but seems like I am missing something here.
The node seems to receive data from other nodes and the bootstrapping is starting all over again.
I went another step ahead and tried by adding the -Dcassandra.consistent.rangemovement=false along with the auto_bootstrap: false (I did this as I intermittently got RuntimeException suggesting A node required to move the data consistently is down although all the nodes were UN).
I still see that the node tries to stream data from other nodes. Am I missing something here.Would really appreciate if someone could help me out here.
