question

oliver avatar image
oliver asked Erick Ramirez commented

Can you confirm my understanding of consistency levels?

Hello, i try to understand RF+CL in an multi-DC example with one replica down (in DC3):

DC1
 6 nodes: 1 to 6
 RF=3
DC2
 4 nodes: 7 to 10
 RF=2
DC3
 3 nodes: 11 to 13
 RF=1

Suppose 1)
A Client writes with consistency LOCAL_ONE to a coordinator node in DC1 who replicates the data to three local nodes and forwards it to some remote coordinators in DC2 and DC3.
Suppose the only one replica in DC3 is down and the coordinator in DC3 stores a hint (in this case: Coordinator node is not the Replica). DC2 should be fine, so two replicas receive the data.

Assumption 1) (write consistency levels in strongest-to-weakest order [1]):

EACH_QUORUM fails, no ACK from DC3
QUORUM succeeds (4 nodes could ACK)
LOCAL_QUORUM succeeds, 2 nodes in DC1 could ACK
ONE fails: DC3 could not ACK
LOCAL_ONE succeeds, DC1 ACK
ANY succeeds

(With CL=ANY would although succeed on DC3, but all other CL would fail on DC3)

I'm confused because going from strongest-to-weakest changes/switches between failing and succeeding. Is there any mistake in my assumptions in Step 1)?

Suppose 2)
Situation after step 1) would be:
DC1: All 6 nodes up, 3 nodes stores the requested data
DC2: All 4 nodes up, 2 nodes stores the requested data
DC3: Only 2 of 3 nodes up. The former remote coordinator from step 1) still stores the Hint for the down-replica.

Now i am looking to a read to some coordinator node in DC3 with respect to different CL.

Assumption 2) (read consistency levels in strongest-to-weakest order [1])

QUORUM succeeds since 4 nodes could serve
LOCAL_QUORUM fails, no replica in DC3 has the data
ONE fails since no replica in DC3 holds the data
LOCAL_ONE fails since no replica in DC3 holds the data
SERIAL succeeds (*)
LOCAL_SERIAL succeeds (*)

(*) Not sure if i understand SERIAL and LOCAL_SERIAL right...

Here i am confused in the same way. Are my assumptions wrong? Did i understood something completely wrong?

[1] https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml/dmlConfigConsistency.html

consistency level
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

Yes, there are several points you misunderstood.

... and forwards it to some remote coordinators in DC2 and DC3.

... and the coordinator in DC3 stores a hint (in this case: Coordinator node is not the Replica).

There is only ever one coordinator for all requests and the coordinator will always be in the local DC, never in the remote DCs.

From "Assumption 1":

ONE fails: DC3 could not ACK

DC3 is not relevant for consistency ONE since all requests are coordinated by the local DC1 DC so requests with consistency ONE will succeed.

The former remote coordinator from step 1) still stores the Hint for the down-replica.

Now i am looking to a read to some coordinator node in DC3 with respect to different CL.

As I stated above, there is no remote coordinator. All requests will be coordinated by a node in the local DC1 DC.

Not sure if i understand SERIAL and LOCAL_SERIAL right...

Serial consistency levels only apply to the read phase of the read-before-write for lightweight transactions (LWT), also known as compare-and-set (CAS) operations.

You cannot use serial consistency levels for regular reads. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

oliver avatar image oliver commented ·

Thanks for your answer!

I am still a little bit confused. From "Cassandra: The Definitive Guide: Distributed Data at Web Scale" ("The Cassandra Write Path" (Page 187)):

"If the cluster spans multiple data centers, the local coordinator node selects a
remote coordinator in each of the other data centers to forward the write to the replicas
in that data center. Each of the remote replicas acknowledges the write directly to the
original coordinator node."

Since the Usage-Column in dmlConfigConsistency states about LOCAL_ONE in the Write-CLs "...but cross-DC traffic is not. LOCAL_ONE accomplishes this."
i thought that "ONE" does the cross-DC traffic. And since the Description of "ONE" states
"A write must be written to the commit log and memtable of at least one replica node." (which is not possible in my example for DC3) i thought ONE would fail.


So i am still seem to misinterpret it...

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ commented ·

... the local coordinator node selects a remote coordinator in each of the other data centers to forward the write to the replicas in that data center.

This statement just reinforces my point. I think you're confusing "remote coordinator" with a "coordinator in a remote DC" -- they're 2 different things.

Requests are coordinated by a node in the local DC instead of a node in a remote DC.

i thought that "ONE" does the cross-DC traffic...

This is correct but only in special circumstances. CL ONE will allow replicas in a remote DC only if you explicitly configure the driver to allow remote nodes to be contacted. Allowing remote nodes is not recommended because it means that an application instance in the local DC can query nodes in a remote DC and will affect the performance of your application.

For remote nodes to be contacted, it means that all nodes in the local DC are down. Something is really wrong for the driver to get to this stage. Cheers!

0 Likes 0 ·