I started with a perfectly token range balanced Cassandra 3.11.8 cluster of 6 nodes (vnodes=4, RF=3) but needed to remove one node from one of the DC's. After a successful "nodetool decommission" process, the cluster now has 5 nodes as expected but if I look at the token ranges that each node is responsible for, one of the remaining 5 nodes now has double the range for each vnode (almost like a single node took all the range from the decommissioned node).
nodetool status (post decom)
nodetool ring (post decom)
Each node is a physical server and defined as a single rack. If you look at the ranges for rack#20 in the "nodetool ring", it has double the range compared to the other nodes. This node#20 is currently suffering many "dropped messages" and now appears overloaded, I suspect due to the extended range it has to service.
Documentation says that rebalancing a vnode based cluster is not required after node removal but is a must for a single token node.
So the question is: Do I need to rebalance my vnode based cluster to get the load percentages and token range distribution back to balanced numbers?
If the answer is yes, then do I use the "nodetool decommission" and node add (with new initial token calculated and configured)? I also have "allocate_tokens_for_keyspace" correctly configured, so would that negate the requirements for the "initial_token" setting?