andrei.mircescu_192518 avatar image
andrei.mircescu_192518 asked Erick Ramirez commented

Why do schema changes not propagate to some Python clients running in containers?

System: Single noded Cassandra with python driver. We only use ObjectMapper(ORM) and don't write queries directly. We also take advantage of UDTS. A table for us is a combination of primitives data types and udts.

We have a kube cluster with Cassandra and 2 services:

  • Service A: handles only schema parts of cassandra like sync_tables
    • Its duty is to bootstrap cassandra with our app logic. It will take care of table creation and populate some values
  • Service B: consumes Cassandra: (running queries/inserts etc)
    • REST endpoint

It seems to be an issue when Service B starts before Service A.


T0: Cassandra starts

T1: Service B start first and creates a connection to cassandra but no queries were executed.

T2: Service A starts and goes creating tables, udts etc.... It also adds a row in a table

T3: Service B receives a request and needs to update a column in the row created by Service A @T2.

The issue is that service B fails to update the row. We tried using the save() and also with update(coumn: value) apis but we receive the same error. One thing that we observed is that select queries seem to work.

Error - <Error from server: code=2000 [Syntax error in CQL query] message="line 1:108 mismatched character '1' expecting '-'">

If we restart service B, everything works and we are able to do saves/updates. This lead us thinking that the Service B connection didn't had the latest schema .

Our assumption was that once a connection is created, further schema changes are synced with all client connections.

Is this something that should work or we need to enforce an order to our services with some init containers?

The issue is fixed if we guarantee that service A finishes before B starts but just wanted to double check if this is a known issue or is it something wrong with our config.

python driver
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

I'm not personally aware of this scenario happening but that's not to say others haven't run into the problem before. As you stated, the expectation is that schema changes are propagated across the cluster.

But it sounds more like your services have other dependencies which need to be satisfied so you need to put in place rules that require service A to be fully operational before service B is started.

If you're possibly able to provide some minimal code samples, it would help us understand what your services are doing and it may yield clues. There's a limit to how much you can post so you might consider uploading them to a site like and posting the URL here.

In the meantime, I'm going to socialise your post internally with the Drivers team at DataStax for ideas.

2 comments Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

andrei.mircescu_192518 avatar image andrei.mircescu_192518 commented ·

Hi Erick,

We will try to reproduce the issue and give you some minimal examples.

Initially, both services were executing sync_tables on the same model. At that point we were hitting errors regarding schema changes.

The datastax docs on sync_table recommends to not use it concurrently with other clients

This function should be used with caution, especially in production environments. Take care to execute schema modifications in a single context (i.e. not concurrently with other clients).

A question regarding this doc paragraph. When it says "not concurrently with other clients", it refers to "not concurrently with other clients at all "? or "don't do sync tables concurrently" ?

The change we did is that only one service will do sync_tables but they will still start concurrently.

After this change, we ended up with the error that we are discussing right now.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ andrei.mircescu_192518 commented ·

I think the problem is both services are making schema changes and it's problematic. I'm getting clarification from the Drivers team. Cheers!

0 Likes 0 ·