Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

dmngaya avatar image
dmngaya asked ·

Why does the driver use consistency level of ALL when all queries are configure with LOCAL_QUORUM?

Last week we deployed code to force a consistency level of LOCAL_QUORUM on all queries. We are seeing errors (see ‘sample-app-log-errors.txt’ file) pop up in production talking about read failures at consistency level of ALL. Our problem is that the CL of ALL is not defined anywhere in our codebase. I ran our application opening TRACE logging for com.datastax.driver.core.Connection to get us query logging. These queries can be seen in the ‘sample-dse-java-driver-logging.txt’ file. Throughout this whole file, the consistency levels defined for each query are LOCAL_ONE or LOCAL_QUORUM, no ALLs. We do use prepared statements so not all of the full queries are displayed, only the ID from the server side. I have pasted 3 “full cycles” (see below paragraph for explanation on app process) which shows the consistency levels for a full app cycle multiple times. So based on this, the following question: I thought the statement consistency level set in the driver should be the CL used, so where is ‘ALL’ coming from in the error message?

App process:
We store data files from a data system. Each ‘row’ has a corresponding hash in the eaa.keyshash table with it’s measurementId. This table is used for us to tell if we need to merge rows. If we find a row hash that matches a previous row, we pull this ID then that gets us all of our partition keys to do an eaa.measurement read query to pull a row. We then merge parameter data for our new row into our old row, then update that row. It is possible, in a large data file, for multiple rows to need to update previously inserted rows, so a row stored previously from this file may be getting updated. These files can contain hundred of thousands of rows, each performing those three steps above. The keyshash query is a LWT, if the row gets inserted then there is no update to perform and we straight insert the measurement row. If that LWT fails, it gives us our measurementId that we use for the update. All of the statements use LOCAL_QUORUM consistency. We do use some LOCAL_ONE consistency levels for determining what files to run, but that query doesn’t run as frequently as the others that use LOCAL_QUORUM.

In the application, we do have a retry utility. This retry is very simple, if the query fails it just waits for X amount of time and retries on the same consistency level, and only retries 3 times. We see the above error after a file has failed 3 times. What is confusing is that no where in our application do we define a consistency level of ALL, only LOCAL_QUORUM.

Best regards

java driver
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image
Erick Ramirez answered ·

@dmngaya The most likely scenario in your case is that nodes are overloaded for writes. When you do reads and there is a digest mismatch among replicas, a blocking read-repair kicks in with a consistency level of ALL. This is specially evident for conditional updates (LWTs) because of the guarantees that the writes are consistent across replicas.

The CL of ALL "leaks out" in the logs particularly when the blocking RR times out, again pointing to the fact that the nodes are overloaded and the synchonising writes were not acknowledged by all the replicas. The CL in the timeout errors you're seeing does not indicate that the driver ignored the CL you've set for the query. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

alexandre.dutra avatar image
alexandre.dutra answered ·

I cannot access the log files you linked. That said, Cassandra sometimes issues background blocking queries at consistency ALL, e.g. you have a digest mismatch from replicas and the coordinator issues a repair with consistency ALL and waits for it to finish. If the timeout kicks in at that moment, the client may get back an error saying "write/read timeout at CL ALL". See https://issues.apache.org/jira/browse/CASSANDRA-15395. This is normal and doesn't mean that your CL was ignored.

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.