Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

kranthij29_188881 avatar image
kranthij29_188881 asked ·

nodetool drain taking around 2 to 3 hours

how can we reduce the nodetool drain command run time. it is taking around 3 hours in some of the node. not sure what is going on. and i don't have much data it is around 500 GB per node.


how can we reduce the it.

cassandra
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image
Erick Ramirez answered ·

@kranthij29_188881 If it is the same nodes as the ones you reported in this post, it's like the draining process got stuck trying to shutdown gossip because it's not running.

You should have tried to bring gossip back online with:

$ nodetool enablegossip

In any case, it's strange for gossip to not be running and it indicates that there were other problems with the node and it wasn't exclusive a "drain" issue. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

wdeng avatar image
wdeng answered ·

Did you check your /var/log/cassandra/system.log and see if there is any error/exception thrown during the drain process? Are you able to see some entry like "StorageService.java:nnnn - DRAINED" at the end of the 2-3 hours?

Normally it shouldn't take this long to drain, as it just flushes all memtables and shuts down gossip, native transport and (possibly) RPC services, so even if you have 500GB data, the amount of data in memory (i.e. memtable) is much smaller than that, so flushing should be done much sooner than 2-3 hours.

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.