Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

satvantsingh_190085 avatar image
satvantsingh_190085 asked Erick Ramirez edited

Why did a removed node not bootstrap when I added it back in to the cluster?

I was simulating a scenario in which any particular node corrupted and we need to remove that particular node .


Step 1 : I shutdown one node(104.42.177.52) from my 2 node cluster .(Assume it got corrupted)

Step 2 : Run "nodetool assassinate" command from active node -

cassandra@mongodbPoc2:~/node1/bin$ ./nodetool assassinate 104.42.177.52

Step 3: Check the status from "nodetool status" and node has been removed from my cluster.

" Till now everything fine , I started my node again which I have removed and its join the cluster back"


Questions - How a removed node(removed forcefully) join back again without bootstrapping ?

Will my data be in corrupted ?

cassandrabootstrapassassinatedecommision
1591525647375.png (29.4 KiB)
1591525707275.png (15.2 KiB)
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

There are 2 scenarios where a node will not bootstrap when added to a cluster:

  1. the node is configured as a seed node in cassandra.yaml
  2. the node's data directory is not empty and the system.local table is already populated with data

Based on the screenshots you provided, the node had 174MB of data both before it was assassinated and after it was added back to the cluster so it indicates that you did not delete the contents of the data directory before you added it back in to the cluster.

When a node joins a cluster, Cassandra checks the bootstrapped column in the system.local table. What takes place next depends on the value of the bootstrapped column:

  • COMPLETED - node will not bootstrap and startup as normal
  • DECOMMISSIONED - node will not bootstrap (unless it was overriden with cassandra.override_decommission=true) and not startup
  • empty (because the table doesn't exist) - node is new and will bootstrap

If you want a node to bootstrap after you have removed it from a cluster, you need to delete the contents of the following directories:

  • data_file_directories
  • hints_directory
  • commitlog_directory
  • cdc_raw_directory
  • saved_caches_directory

As a side note, assassinating a node is not the recommended way of removing nodes from a cluster. Use these procedures instead where possible:

  • decommission the node is it is running and operational
  • remove the node if it is down and cannot be brought back online
  • assassinate a node only if attempts to remove it failed

For more info, see the documented procedure for removing a node from a cluster. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.