question

suujana.naga16_165607 avatar image
suujana.naga16_165607 asked Erick Ramirez answered

Is it necessary to drain when the node is down?

I have a cluster of 9 nodes in production running DSE 5.1.15

One node was down in nodetool status and unable to connect to cqlsh.

Getting below error for cqlsh

Connection error: ('Unable to connect to any servers', {'10.31.89.207': error(None, "Tried connecting to [('10.31.89.207', 9042)]. Last error: ('_ssl.c:602: The handshake operation timed out',)")})

So decided restart DSE service like below

nodetool drain
service dse stop
service dse start

It is observed that drain is taking long time , its been 1 and half hour by now and still its running. Load on the node is 1009 GB.

How can i check the logs while running drain to see that its actually doing something? or how much more time it will still take?

Is it necessary to run drain even when the node is down?

dse
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

If the node is unresponsive for whatever reason, it will not be possible to run any nodetool commands since that requires the JVM to be operational.

If nothing is getting logged then it's a good indication that the process is no longer responding. In which case you should just proceed with restarting the node. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.