question

azim_91_184236 avatar image
azim_91_184236 asked Erick Ramirez commented

Can the system keyspaces and user keyspaces be replicated to new DC together or system keyspaces first?

Migrating OSS cassandra cluster from OnPrem to Cloud with 'Add datacenter' approach and trying to reconcile the difference between two documents I am following to replicate system keyspaces-

The blog http://www.redshots.com/moving-cassandra-clusters-without-downtime-part-1/ suggests replicating system keyspaces first as we are adding nodes, while the Datastax documentation https://docs.datastax.com/en/dse/6.8/dse-admin/datastax_enterprise/operations/opsAddDCToCluster.html suggests doing all at the same time, in step 9

Question-

1. Is there any reason to replicate the system keyspaces to new DC first with alter keyspace command as suggested in the redshots blog or it doesn't matter that much?

2. If replicating system keyspaces first is recommended, as suggested in redshots blog, are we just replicating new data in those keyspaces, since I didn't see them suggesting to run nodetool rebuild until later which will replicate existing data.

Appreciate your guidance to clarify this.

replicationadd datacenter
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

When you change the replication settings of a keyspace to add replicas in a new DC, two things will happen:

  1. New mutations (INSERT, UPDATE, DELETE) will be sent by coordinators to the new DC.
  2. Nodes in the new DC will accept any non-local reads.

To answer your questions explicitly:

  1. There is no reason to prefer changing the replication settings of one keyspace over another -- system or otherwise.
  2. It isn't. The replication settings for ALL keyspaces (except those with LocalStrategy or EverywhereStrategy) need to be configured before you run a nodetool rebuild on the new nodes.

I need to add that that Step 1 in the Redshots blog post isn't quite right. You should not add the new DC to replication until the new nodes in the new DC are online otherwise the writes in (1) above will fail since the replicas in the new DC are unavailable.

I recommend you follow the steps in the DataStax document Adding a datacenter to a cluster. The steps in this document are tested and confirmed and what all customers use. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

azim_91_184236 avatar image azim_91_184236 commented ·

Thank you @Erick Ramirez for the clarification and guidance, appreciate it!

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ azim_91_184236 commented ·

Not a problem. Cheers!

0 Likes 0 ·