System: Single noded Cassandra with python driver. We only use ObjectMapper(ORM) and don't write queries directly. We also take advantage of UDTS. A table for us is a combination of primitives data types and udts.
We have a kube cluster with Cassandra and 2 services:
- Service A: handles only schema parts of cassandra like sync_tables
- Its duty is to bootstrap cassandra with our app logic. It will take care of table creation and populate some values
- Service B: consumes Cassandra: (running queries/inserts etc)
- REST endpoint
It seems to be an issue when Service B starts before Service A.
Timeline:
T0: Cassandra starts
T1: Service B start first and creates a connection to cassandra but no queries were executed.
T2: Service A starts and goes creating tables, udts etc.... It also adds a row in a table
T3: Service B receives a request and needs to update a column in the row created by Service A @T2.
The issue is that service B fails to update the row. We tried using the save() and also with update(coumn: value) apis but we receive the same error. One thing that we observed is that select queries seem to work.
If we restart service B, everything works and we are able to do saves/updates. This lead us thinking that the Service B connection didn't had the latest schema .
Our assumption was that once a connection is created, further schema changes are synced with all client connections.
Is this something that should work or we need to enforce an order to our services with some init containers?
The issue is fixed if we guarantee that service A finishes before B starts but just wanted to double check if this is a known issue or is it something wrong with our config.
