question

dba.satvant_168667 avatar image
dba.satvant_168667 asked smadhavan commented

What is the best approach for handling a node with a failed data drive?

Hi @Erick Ramirez

What is the best recommendation way to recover node which have a failed drive(Contains data file, commit log file ).Binaries are intact on other drive .

Cluster configuration -

-3 Node cluster

- DSE version 5.8

- Data size ~1.5 TB on each node

- RF is 3 or 2 for application key spaces

-RF is 1 for some default key spaces.

What is the best solution to avoid any data loss ? and what will be the performance impact during recovery?

1) Replace the node with brand new node ?

2) Clean the failed drive and start the Cassandra service on failed drive node and run repair.

3) Replace the drive and start the Cassandra service on failed drive node and run repair.

Thanks

S.Singh

replace nodes
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered smadhavan commented

Replace the failed hardware on the server with a new drive then edit cassandra-env.sh and add the "replace address" flag:

JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=listen_address_of_node

This will force the node to "replace itself" in the ring and bootstrap with the same token ownership(s). Note that you will need to make sure the node doesn't have its own IP listed as a seed in cassandra.yaml.

Start Cassandra on the node and let it bootstrap. Once complete, remove the JVM flag above (restart not required). For details, see the procedure for Replacing a dead node. Cheers!

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

smadhavan avatar image smadhavan ♦ commented ·

@dba.satvant_168667 DSE doesn't have a version called 5.8. I assume, you're referring to either DSE 5.1.8 or DSE 6.8.x here. What Erick mentioned above. Not very different, but the corresponding DSE 5.1.x documentation is here & DSE 6.8.x documentation is here.

0 Likes 0 ·