question

KongBaiZZ avatar image
KongBaiZZ asked Erick Ramirez answered

Does changing the replication factor from 1 to 3 affect production operations?

My production environment has one keyspace with a replication factor of 1 , I want to use ALTER KEYSPACE command to make it 3 , I am afraid that this operation will affect the production operation , Please tell me the result

replication
2 comments
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

smadhavan avatar image smadhavan ♦ commented ·

@KongBaiZZ what version of Cassandra are you using? Is it DataStax Enterprise (DSE)® or Open Source Apache Cassandra® or DataStax Astra DB®? Also, could you update your original question here to state what is the read & write consistency level that is being used by your application drivers?

0 Likes 0 ·
smadhavan avatar image smadhavan ♦ smadhavan ♦ commented ·
@KongBaiZZ please see this ^ and respond
0 Likes 0 ·
steve.lacerda avatar image
steve.lacerda answered Erick Ramirez edited

I believe you mean ALTER KEYSPACE. That will actually not have any effect, other than replicating the writes. The impact will come from running a repair after you alter the keyspace replication factor. The repair will create merkle trees which will repair the data and replicate it to the other 2 nodes. That could likely have an impact, but it should be sustainable if you've properly sized your cluster. Repairs are a normal part of Cassandra operation as well, so a properly sized cluster should see negligible impact.

3 comments Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

KongBaiZZ avatar image KongBaiZZ commented ·

@steve.lacerda That is to say, After I use the Aleter Keyspace command, my production environment data can be accessed normally? Can you tell me the specific run command.

//first:
ALTER KEYSPACE XXX WITH replication = {'class':'NetworkTopologyStrategy', 'replication_factor' : 3};

I don't know what command I need to run after Alter keyspace

ll me

0 Likes 0 ·
KongBaiZZ avatar image KongBaiZZ commented ·

thanks for your answer and your team.Thank you

0 Likes 0 ·
steve.lacerda avatar image steve.lacerda ♦ KongBaiZZ commented ·

After you ALTER the keyspace replication factor, cqlsh will say something like you need to run a repair. Which means you will need to run:

nodetool repair -pr <keyspace> 

on all nodes in the cluster.

0 Likes 0 ·
smadhavan avatar image
smadhavan answered KongBaiZZ commented

What @steve.lacerda said. Also, while you're at this activity, I would recommend you to go ahead with leveraging the following so that you don't have to re-do (i.e. change from SimpleStrategy to NetworkTopologyStrategy for the replication class) this when you want to expand your existing cluster to multiple datacenters in future,

REPLICATION = { 
 'class' : 'NetworkTopologyStrategy', 
 'datacenter_name' : N [, 
 'datacenter_name' : N ] 
 }
3 comments Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

KongBaiZZ avatar image KongBaiZZ commented ·

Online apps are using this keyspace data , So I think Alter keyspace will affect the app?? Are you sure it won't have any effect?

If it won't have any effect,Can you tell me how how should I do this.




first step am i running this command ?

  1. ALTER KEYSPACE XXX WITH replication = {'class':'SimpleStrategy', 'replication_factor' : 3};


But I don't know what command I need to run after Alter keyspace;

Can you tell me the detailed command steps?
one:
two:

three:


0 Likes 0 ·
smadhavan avatar image smadhavan ♦ commented ·

@KongBaiZZ see the documentation here for reference. Yes, your [properly right sized] cluster will be operating normally as it was before after issuing the alter command.

0 Likes 0 ·
KongBaiZZ avatar image KongBaiZZ smadhavan ♦ commented ·

thanks for your answer, I will run Alter command in low traffic time.


1 Like 1 ·
Erick Ramirez avatar image
Erick Ramirez answered

Changing the replication factor from 1 to 3 will have a huge impact to production clusters.

With only 1 replica, you would almost certainly be using a consistency level of ONE or LOCAL_ONE for read requests. Immediately after you issue the ALTER KEYSPACE command, there will be an additional 2 replicas which would "think" they own the data and will respond to read requests.

However until you have full repaired the keyspace on all nodes in the ring, those 2 additional replicas will not have the data yet so your application will get inconsistent results.

As a workaround, you should update your application to use a consistency of LOCAL_QUORUM for read requests to reduce the possibility that reads will return no results. Cheers!

Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.