Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

denis.koblov avatar image
denis.koblov asked ·

Why are we getting unexpected digest mismatch exceptions?

Hey, guys. I need your help))

I use the Apache Cassandra 3.11.10 version.

There are a lot of mismatch exceptions in the cassandra logs:

[ReadRepairStage:2285] o.a.cassandra.service.ReadCallback - [] Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(1314781106783051769, 6576656e7447726f75703d4d6573736167655570646174652c20657874656e73696f6e3d273137363939313230313027) (09d49371609ab4cd4a7874fcbf3a31c1 vs 318c5f818eb033fda7c2d9a14d997485)
	at org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:95)
	at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:235)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
	at java.lang.Thread.run(Thread.java:748)

After the investigation, I have found the following behavior:

1) We have the following table:

CREATE TABLE IF NOT EXISTS tmp.my_table (
  unique_key                    text,
  id                            text,
  data                          blob,
  location                      text,
  PRIMARY KEY ((unique_key), id)
  ) WITH
    caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'} AND
    compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'};
		

2) In our application, we execute the following CQL with consistency level LOCAL_QUORUM:

select data from tmp.my_table where unique_key in :unique_keys and id=:id

3) The mismatch exception is thrown after executing this CQL.

4) The data for the suspicious row in different nodes is equal.

5) We have found (by debugging), that during the digest calculation, the value of the location column is null for specific nodes. In particular:

  • if node1 is the coordinator, then when calculating the digest, the location value for node1 is null, and for node2 it is filled.
  • if node2 is coordinator, then when calculating the digest, the location value for node1 is filled, but for node2 it is null.

It happens, because the location column marks as skippable in the BTreeRow for coordinator node:

 // We include the cell unless it is 1) shadowed, 2) for a dropped column or 3) skippable.
// And a cell is skippable if it is for a column that is not queried by the user and its timestamp
// is lower than the row timestamp (see #10657 or SerializationHelper.includes() for details).

boolean isSkippable = !queriedByUserTester.test(column);

Comment from the org.apache.cassandra.db.rows.SerializationHelper.includes():

// During queries, some columns are included even though they are not queried by the user because
// we always need to distinguish between having a row (with potentially only null values) and not
// having a row at all (see #CASSANDRA-7085 for background). In the case where the column is not
// actually requested by the user however (canSkipValue), we can skip the full cell if the cell
// timestamp is lower than the row one, because in that case, the row timestamp is enough proof
// of the liveness of the row. Otherwise, we'll only be able to skip the values of those cells.

But this column isn't skippable for the other nodes. Why?

As soon as we changed the CQL from

select data from tmp.my_table where unique_key in :unique_keys and id=:id

to

select data, location from tmp.my_table where unique_key in :unique_keys and id=:id

the mismatch errors disappeared.

Is this bug in the Cassandra? Or we make something wrong?

read-repair
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

It feels like you got caught in the weeds a bit. I don't think there's anything sinister in the exceptions you're seeing though I'm happy to be corrected.

In my experience, the digest mismatch occur because replicas are out-of-sync -- it's as simple as that. If your cluster is overloaded such that it can't keep up with writes, then you'll see nodes reporting mutations getting dropped and replicas end up being inconsistent.

You said that the errors disappeared when you changed the query though I don't believe that's the reason the errors stopped. When you read the data with the first query and a digest mismatch is detected, a read-repair operation triggers an update of the data. With the replicas synchronised, a successive read using the second query does not result in a digest mismatch.

If you really feel that there's a bug, my suggestion is that you enable tracing and run the first query on IDs you haven't read before. The trace output will show you if there's a digest mismatch and the read-repair getting triggered.

But I don't think this is a problem if the nodes are not dropping mutations and you regularly repair your cluster. Cheers!

2 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

No, Erick, it isn't a replication problem.

I have a particular record in the 3 nodes.

I execute the request in the node1 and I observed digest mismatch exception.

I execute the request again in the node1, and I observed digest mismatch exception again.

etc.

I changed request (added location column). The digest mismatch error for my suspicious entry disappears.

If I revert the changes, and again execute request without location columt, the digest mismatch exception appears again.

0 Likes 0 ·

Did you enable tracing as I suggested? Cheers!

0 Likes 0 ·