What would happen if multiple app sessions make schema changes at the same time?

Question

question

teriksson asked Nov 30, '20 Erick Ramirez commented Dec 2, '20

What would happen if multiple app sessions make schema changes at the same time?

Reading up on Schema changes / meta data changes / Schema migration

I come accross amongst others this article:

https://docs.datastax.com/en/developer/java-driver/3.7/manual/metadata/

But what if :

All nodes does not reply , what happens then ?
Do we end up rolling back, OR do we end up out of synch ?

I have understood (experienced actually) that we end up out of synch, and that we had to restart all the nodes in the cluster as we did not know which node had the wrong information.

But the larger the cluster becomes where 3 nodes is small, 30 nodes is not big, 300 nodes hmm kind of big.... the more problematic it is to restart all those nodes.

Is there any way to know which node(s) that are not in line, not synched ?

further my understanding is also that the meta data agreement is with ALL, as in ALL DCs, DC1, DC2, DCn, so a rather big synchronisation that needs to happen within 10 seconds.

What if

MORE THAN ONE application tries to modify the same Keyspace at the same time or close to the same time ?

Reading the article this should be ok it seems like, but my experience is not that at all.

So does that mean that all applications/clients that need to change the schema / meta data needs to first coordinate this with e.g. global mutex, zookeeper or something similar to ensure that only one tries to do this at the same time ?

java driver schema

1606726863772.png (17.2 KiB)

10 |1000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

DataStax Enterprise is powered by the best distribution of Apache Cassandra ™

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Privacy Policy Terms of Use

Answer 1 · 2020-12-01T05:39:42Z

Erick Ramirez answered Dec 1, '20 Erick Ramirez commented Dec 2, '20

If not all nodes respond or some timeout then your cluster is mostly likely to end up in a schema disagreement -- there is no rollback mechanism that takes place in this scenario. And you are correct that a rolling restart is required to fix it.

For this reason, programatically changing the schema isn't recommended. The feature is provided by the drivers as a convenience but as you already know, there are a lot of problems that can arise from it. Cheers!

2 Share

10 |1000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

pahlgren commented · Dec 01, 2020 at 08:36 AM

What is the difference when doing a change "manually" using cqlsh and doing it from an application if the application make sure it only do one change at a time?

Is it okay to do changes "manually" to several keyspaces at the same time?

0 ·

Erick Ramirez ♦♦ pahlgren commented · Dec 02, 2020 at 02:53 AM

Making changes through cqlsh means that you are forced to only one DDL statement at a time mostly because you don't get the command prompt back until the previous DDL statement has completed.

And no, you can not perform multiple DDL changes in parallel regardless of which keyspace definition is being altered. Cheers!

0 ·

question