We have 2 datacenters (DCs). 1 is production, the other is begin migrated to using replication. However we encountered data corruption in one keyspace in the 2nd DC after migration completed.
We tried repair of the table in the keyspace but the row counts are different each time.
We tried scrubbing the table on all the nodes in the 2nd datacenter but the row counts are different each time and there's definitely no writes going on.
If we drop the table and repair it, this would drop the table in the production DC.
So, what is the lowest risk way of rebuilding the table from 1st DC? It seems decommissioning the 2nd DC and re-adding it is the sure way.
How did the table become corrupt? We did a nodetool ---rebuild on each node when creating the 2nd DC but interrupted the rebuild with CTRL-C. It is the only explanation we have.
(We used the instructions here to create the 2nd DC: https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/operations/opsAddDCToCluster.html)
So, how de we *safely* decommission the 2nd DC?
TIA,
Adam