how can we reduce the nodetool drain command run time. it is taking around 3 hours in some of the node. not sure what is going on. and i don't have much data it is around 500 GB per node.
how can we reduce the it.
Bringing together the Apache Cassandra experts from the community and DataStax.
Want to learn? Have a question? Want to share your expertise? You are in the right place!
Not sure where to begin? Getting Started
You should have tried to bring gossip back online with:
$ nodetool enablegossip
In any case, it's strange for gossip to not be running and it indicates that there were other problems with the node and it wasn't exclusive a "drain" issue. Cheers!
Did you check your /var/log/cassandra/system.log and see if there is any error/exception thrown during the drain process? Are you able to see some entry like "StorageService.java:nnnn - DRAINED" at the end of the 2-3 hours?
Normally it shouldn't take this long to drain, as it just flushes all memtables and shuts down gossip, native transport and (possibly) RPC services, so even if you have 500GB data, the amount of data in memory (i.e. memtable) is much smaller than that, so flushing should be done much sooner than 2-3 hours.
5 People are following this question.