Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

Tri avatar image
Tri asked Erick Ramirez edited

What is the difference between nodetool rebuild vs simply bootstrapping a new node in new datacenter?

Let's assume I would like to add a 2nd datacenter to my cluster which is currently single-DC. I following the instructions outlined in Adding a datacenter to a cluster

There is a step:

Run nodetool rebuild on each node in the new datacenter, specifying the datacenter to rebuild from. This step replicates the data to the new datacenter in the cluster.

QUESTION: would a normal bootstrapping process, by simply starting the new node (in the new DC), achieve the same goal as the recommended nodetool rebuild ?

By the definition of nodetool rebuild I am wondering if it is a kind of bootstrap which allows the new DC to limit the scope of the source DCs (which send data) while a normal bootstrapping would get all existing DCs involved.

bootstraprebuild
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

2 Answers

bettina.swynnerton avatar image
bettina.swynnerton answered Erick Ramirez edited

Hi @Tri,

the new node in the new datacenter will need to have joined the cluster by the time you run rebuild.

When a new node is bootstrapped into the cluster, it contacts the seed nodes about gossip state and schema, calculates the tokens and streams the data it is responsible for.

In the case of joining as a new node in a new datacenter, It is bootstrapped in without any responsibility for user data, as the replication strategy in the cluster schema has not yet been changed to take the new topology of the cluster into account. So it is bootstrapping alright, but it is only receiving the system tables and information that is cluster wide.

The nodetool rebuild is then the operation that ensures that the data is streamed from the source datacenter to the new node in the new datacenter. Again, this takes the topology and tokens of the new datacenter into account.

So, bootstrap will happen first (as the node is joining the cluster), but the rebuild is required to replicate the data as per replication strategy.

Here for example for a non-seed node joining the cluster as part of a new DC, before the schema is changed to take the new datacenter into account. 

INFO  [main] 2020-07-14 16:04:32,946 StorageService.java:1478 - JOINING: waiting for schema information to complete
INFO  [main] 2020-07-14 16:04:32,946 StorageService.java:1478 - JOINING: schema complete, ready to bootstrap
INFO  [main] 2020-07-14 16:04:32,946 StorageService.java:1478 - JOINING: waiting for pending range calculation
INFO  [main] 2020-07-14 16:04:32,947 StorageService.java:1478 - JOINING: calculation complete, ready to bootstrap
INFO  [main] 2020-07-14 16:04:32,947 StorageService.java:1478 - JOINING: getting bootstrap token
INFO  [main] 2020-07-14 16:04:32,949 BootStrapper.java:188 - tokens manually specified as [0]
INFO  [main] 2020-07-14 16:04:32,981 StorageService.java:1478 - JOINING: sleeping 30000 ms for pending range setup
INFO  [main] 2020-07-14 16:05:02,983 StorageService.java:1478 - JOINING: Starting to bootstrap...
INFO  [main] 2020-07-14 16:05:03,007 StreamResultFuture.java:90 - [Stream #c6b3a5b0-c5eb-11ea-843d-09963b9c7a93] Executing streaming plan for Bootstrap
INFO  [StreamConnectionEstablisher:1] 2020-07-14 16:05:03,012 StreamSession.java:267 - [Stream #c6b3a5b0-c5eb-11ea-843d-09963b9c7a93] Starting streaming to /10.101.35.250
INFO  [StreamConnectionEstablisher:1] 2020-07-14 16:05:03,022 StreamCoordinator.java:270 - [Stream #c6b3a5b0-c5eb-11ea-843d-09963b9c7a93, ID#0] Beginning stream session with /10.101.35.250
INFO  [STREAM-IN-/10.101.35.250:7000] 2020-07-14 16:05:03,088 StreamResultFuture.java:187 - [Stream #c6b3a5b0-c5eb-11ea-843d-09963b9c7a93] Session with /10.101.35.250 is complete
INFO  [STREAM-IN-/10.101.35.250:7000] 2020-07-14 16:05:03,108 StreamResultFuture.java:219 - [Stream #c6b3a5b0-c5eb-11ea-843d-09963b9c7a93] All sessions completed
INFO  [main] 2020-07-14 16:05:03,110 StorageService.java:1535 - Bootstrap completed for tokens [0]
INFO  [main] 2020-07-14 16:05:03,121 StorageService.java:1478 - JOINING: Finish joining ring
INFO  [main] 2020-07-14 16:05:03,189 Gossiper.java:1780 - Waiting for gossip to settle...
INFO  [main] 2020-07-14 16:05:11,191 Gossiper.java:1811 - No gossip backlog; proceeding

I hope this helps to see the difference between these two operations.

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Tri avatar image Tri ·

In the case of joining as a new node in a new datacenter, It is bootstrapped in without any responsibility for user data, as the replication strategy in the cluster schema has not yet been changed to take the new topology of the cluster into account.

Oh this is the key difference. I see the difference now. Thanks very much.

[Follow up question posted as #6485]

0 Likes 0 ·
Erick Ramirez avatar image
Erick Ramirez answered

New DC = empty DC

To add to Bettina's answer, there is a crucial point you're missing -- you are adding a new data center. By definition, there is no data in the DC because it is new.

When you use the standard bootstrap for adding new nodes (not adding a new DC), a node in an empty DC will have no data to bootstrap and will automatically join the cluster claiming ownership of token range(s) it doesn't have data for.

As Bettina stated, the idea behind nodetool rebuild is to request the data from a source DC (an existing one) to be streamed to a new node in a new DC.

Parallel builds

One other very important aspect of the process for adding a new DC is that you can add multiple nodes in parallel with auto_bootstrap: false. Once Cassandra has come online on the new nodes, you can build them in parallel (provided the source DC can take the extra load).

If you attempt to bootstrap 2 new nodes at the same time, you will get an exception on startup informing you that another node is already bootstrapping and that you cannot do multiple nodes in parallel. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.