Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

andrei.mircescu_192518 avatar image
andrei.mircescu_192518 asked ·

Why do schema changes not propagate to some Python clients running in containers?

System: Single noded Cassandra with python driver. We only use ObjectMapper(ORM) and don't write queries directly. We also take advantage of UDTS. A table for us is a combination of primitives data types and udts.

We have a kube cluster with Cassandra and 2 services:

  • Service A: handles only schema parts of cassandra like sync_tables
    • Its duty is to bootstrap cassandra with our app logic. It will take care of table creation and populate some values
  • Service B: consumes Cassandra: (running queries/inserts etc)
    • REST endpoint

It seems to be an issue when Service B starts before Service A.

Timeline:

T0: Cassandra starts

T1: Service B start first and creates a connection to cassandra but no queries were executed.

T2: Service A starts and goes creating tables, udts etc.... It also adds a row in a table

T3: Service B receives a request and needs to update a column in the row created by Service A @T2.

The issue is that service B fails to update the row. We tried using the save() and also with update(coumn: value) apis but we receive the same error. One thing that we observed is that select queries seem to work.

Error - <Error from server: code=2000 [Syntax error in CQL query] message="line 1:108 mismatched character '1' expecting '-'">


If we restart service B, everything works and we are able to do saves/updates. This lead us thinking that the Service B connection didn't had the latest schema .

Our assumption was that once a connection is created, further schema changes are synced with all client connections.

Is this something that should work or we need to enforce an order to our services with some init containers?

The issue is fixed if we guarantee that service A finishes before B starts but just wanted to double check if this is a known issue or is it something wrong with our config.

python driver
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

I'm not personally aware of this scenario happening but that's not to say others haven't run into the problem before. As you stated, the expectation is that schema changes are propagated across the cluster.

But it sounds more like your services have other dependencies which need to be satisfied so you need to put in place rules that require service A to be fully operational before service B is started.

If you're possibly able to provide some minimal code samples, it would help us understand what your services are doing and it may yield clues. There's a limit to how much you can post so you might consider uploading them to a site like https://gist.github.com and posting the URL here.

In the meantime, I'm going to socialise your post internally with the Drivers team at DataStax for ideas.

2 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Hi Erick,

We will try to reproduce the issue and give you some minimal examples.


Initially, both services were executing sync_tables on the same model. At that point we were hitting errors regarding schema changes.

The datastax docs on sync_table recommends to not use it concurrently with other clients

This function should be used with caution, especially in production environments. Take care to execute schema modifications in a single context (i.e. not concurrently with other clients).

A question regarding this doc paragraph. When it says "not concurrently with other clients", it refers to "not concurrently with other clients at all "? or "don't do sync tables concurrently" ?

The change we did is that only one service will do sync_tables but they will still start concurrently.

After this change, we ended up with the error that we are discussing right now.

0 Likes 0 · ·
Erick Ramirez avatar image Erick Ramirez ♦♦ andrei.mircescu_192518 ·

I think the problem is both services are making schema changes and it's problematic. I'm getting clarification from the Drivers team. Cheers!

0 Likes 0 · ·