Reading up on Schema changes / meta data changes / Schema migration
I come accross amongst others this article:
But what if :
- All nodes does not reply , what happens then ?
Do we end up rolling back, OR do we end up out of synch ?
I have understood (experienced actually) that we end up out of synch, and that we had to restart all the nodes in the cluster as we did not know which node had the wrong information.
But the larger the cluster becomes where 3 nodes is small, 30 nodes is not big, 300 nodes hmm kind of big.... the more problematic it is to restart all those nodes.
Is there any way to know which node(s) that are not in line, not synched ?
further my understanding is also that the meta data agreement is with ALL, as in ALL DCs, DC1, DC2, DCn, so a rather big synchronisation that needs to happen within 10 seconds.
- MORE THAN ONE application tries to modify the same Keyspace at the same time or close to the same time ?
Reading the article this should be ok it seems like, but my experience is not that at all.
So does that mean that all applications/clients that need to change the schema / meta data needs to first coordinate this with e.g. global mutex, zookeeper or something similar to ensure that only one tries to do this at the same time ?