question

Tri avatar image
Tri asked Sakiv commented

How does Cassandra guarantee data integrity?

This might be paranoid but how can we be sure that Cassandra writes exactly what it is given? Let take a simplistic example. A 1 node cluster. And table with only 2 columns

CREATE TABLE users (userId UUID, username TEXT, PRIMARY KEY (userId));

INSERT INTO users (userId, username) VALUES (UUID(), 'Issac Newton');

What is the underlying mechanism that guarantees that Cassandra did write the given value 'Issac Newton' and not something else? Like ' 'Issac N' ?

In a general case, Cassandra may receive a data which could be different than the original data for example the row partially truncated/mutated while in-flight. Or even when Cassandra did receive the correct row data, maybe some random things still happen so Cassandra writes B instead of A.

cassandra
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Sakiv avatar image Sakiv commented ·

@Erick Ramirez Apologies, if I understand the context here, this is for integrity of the Query in flight?

In that case, I have a different question. How does Cassandra ensure integrity of data in Commit Log, SSTables? How can tampering of the same be identified?

Under compression we have crc_check_chance.. it defaults to 1, does that ensure integrity of data is checked when data is read?

Digest.crc32 .. are the contents computed per row.. per SSTable and stored here? Would appreciate if you could elaborate Data Integrity of Data Stored on Disk as well.

Many thanks in advance!

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

There are no guarantees in life :) but the likelihood of the scenario you described is pretty unlikely. Let me explain.

The CQL native protocol is frame-based and looks like this:

Header validation

The 9-byte header is composed of:

  • protocol version, e.g. v4 (1 byte)
  • flags: compression, tracing, custom payload, warning (1 byte)
  • stream ID (2 bytes)
  • operation code, e.g. STARTUP, AUTHENTICATE, QUERY, PREPARE, AUTH_CHALLENGE (1 byte)
  • length of the body of the frame up to a maximum of 256MB (4 bytes)

The body of the message is variable in size but it's length is defined in the header so if the body's length is off for whatever reason, it gets picked up immediately.

Message validation

Within the body itself for messages of operation code QUERY, the contents look like:

<query_string><query_parameters>

Going in deeper where the message is a request, the format of the query parameters section contains the values for bound variables:

  <consistency><flag><value_1><value_2>...<value_n>[<timestamp>]

Since the query string enumerates the bound variables, a value missing would invalidate the message.

Serialisation format validation

Furthermore, the serialisation formats of the CQL data types define how the drivers encode the values. Values are represented as bytes and the format includes an integer prefix that denotes the length of the value. It's another layer of validation built into the protocol.

Replicas

On the write path, the mutation will get sent to multiple replicas (based on keyspace replication).

For a replication factor of 3, the chances of the value getting corrupted/truncated by the time it is written on 3 nodes is nearly impossible unless all 3 replicas had a hardware failure because of Cassandra's nothing-shared architecture. Data on disk is immutable so the only way it would get corrupted is through a bad disk.

For more details, see the CQL Binary Protocol v4 Specification document. Cheers!


community-7876-cql-protocol-v4-frames.png (6.4 KiB)
2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Tri avatar image Tri commented ·

Thanks very much @Erick Ramirez. This answers perfectly the question. Thanks for the efforts making a concise and readable answer while explaining a complex under-the-hood details.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Tri commented ·

Nice pickup! I spent over an hour trying to find the right words, labouring over which details to include and which details to leave out.

I didn't want readers to be overwhelmed with the technical implementation of the protocol but I also didn't want to over-simplify it.

I found it really difficult finding the sweet spot so thanks for acknowledging it. Cheers!

1 Like 1 ·