Hey, guys. I need your help))
I use the Apache Cassandra 3.11.10 version.
There are a lot of mismatch exceptions in the cassandra logs:
[ReadRepairStage:2285] o.a.cassandra.service.ReadCallback - [] Digest mismatch: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(1314781106783051769, 6576656e7447726f75703d4d6573736167655570646174652c20657874656e73696f6e3d273137363939313230313027) (09d49371609ab4cd4a7874fcbf3a31c1 vs 318c5f818eb033fda7c2d9a14d997485) at org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:95) at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:235) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) at java.lang.Thread.run(Thread.java:748)
After the investigation, I have found the following behavior:
1) We have the following table:
CREATE TABLE IF NOT EXISTS tmp.my_table (
unique_key text,
id text,
data blob,
location text,
PRIMARY KEY ((unique_key), id)
) WITH
caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'} AND
compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'};
2) In our application, we execute the following CQL with consistency level LOCAL_QUORUM:
select data from tmp.my_table where unique_key in :unique_keys and id=:id
3) The mismatch exception is thrown after executing this CQL.
4) The data for the suspicious row in different nodes is equal.
5) We have found (by debugging), that during the digest calculation, the value of the location column is null for specific nodes. In particular:
- if node1 is the coordinator, then when calculating the digest, the location value for node1 is null, and for node2 it is filled.
- if node2 is coordinator, then when calculating the digest, the location value for node1 is filled, but for node2 it is null.
It happens, because the location column marks as skippable in the BTreeRow for coordinator node:
// We include the cell unless it is 1) shadowed, 2) for a dropped column or 3) skippable. // And a cell is skippable if it is for a column that is not queried by the user and its timestamp // is lower than the row timestamp (see #10657 or SerializationHelper.includes() for details). boolean isSkippable = !queriedByUserTester.test(column);
Comment from the org.apache.cassandra.db.rows.SerializationHelper.includes():
// During queries, some columns are included even though they are not queried by the user because // we always need to distinguish between having a row (with potentially only null values) and not // having a row at all (see #CASSANDRA-7085 for background). In the case where the column is not // actually requested by the user however (canSkipValue), we can skip the full cell if the cell // timestamp is lower than the row one, because in that case, the row timestamp is enough proof // of the liveness of the row. Otherwise, we'll only be able to skip the values of those cells.
But this column isn't skippable for the other nodes. Why?
As soon as we changed the CQL from
select data from tmp.my_table where unique_key in :unique_keys and id=:id
to
select data, location from tmp.my_table where unique_key in :unique_keys and id=:id
the mismatch errors disappeared.
Is this bug in the Cassandra? Or we make something wrong?
