question

Gangadhara M.B avatar image
Gangadhara M.B asked Erick Ramirez commented

What are the conditions for running a cleanup after adding nodes to a DC?

We have multi DC cluster set up ,having Three DC are on-premise and one AWS DC (EC2 I3.2x large Instance) , in total four DC cluster .

1) Now I am considering to add few more nodes only to AW DC, after adding new nodes only to AWS DC , should "nodetool cleanup" run only on AWS DC ? or should "nodetool cleanup" run on all DC including on-prim Three DC .

2) Can "nodetool cleanup" be run in parallel or only one node at a time ?

3) If needs to run "nodetool cleanup" on on-prim DC as well , can we run cleanup in parallel with AWS DC .

4) Any ways to control(parameters) to set before start running cleanup so that it doesn't Impact business as usual and doesn't spike up system resource usage on the node where cleanup runs?

5) Do we also need to run repair(nodetool reapir -pr -j 4) on each node of each DC after cleanup execution completes.

Thanks

Gangadhara

cleanup
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

If you look at the documented and recommended procedure when adding nodes to an existing cluster, the conditions are quite simple:

After all new nodes are running, run nodetool cleanup on each of the previously existing nodes to remove the keys that no longer belong to those nodes. Wait for cleanup to complete on one node before running nodetool cleanup on the next node.

Cleanup can be safely postponed for low-usage hours.

To respond to your questions directly:

  1. Cleanup is only required on rings (DCs) where nodes were added.
  2. As the document states, we recommend cleaning up one node at a time.
  3. It doesn't make sense to cleanup nodes on the on-premise DCs since you didn't add nodes there.
  4. Cleanup one node at a time.
  5. Repairs are part of the normal operation of a cluster and has nothing to do with adding nodes. You should regularly run repairs regardless of when adding nodes or not.

Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Gangadhara M.B avatar image Gangadhara M.B ·

Super quick response , Thanks lot for making my doubts clarified

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Gangadhara M.B ·

Happy to help. Cheers!

0 Likes 0 ·