Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

Jason avatar image
Jason asked Erick Ramirez commented

How can I check for the cause of inconsistencies between DC1 and DC2?

I have two Datacenter, dc1 and dc2 on Openstack, each datacenter have 3 nodes. use the first node of both datacenter as seeds node.

keyspace "userElement" setting is 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 3};

My application use LOCAL_QUORUM to write data to "user" table inside "userElement" keyspace. normally, the replication working fine, all data write into dc1 will replicate to dc2. But sometimes, one of the records missing on dc2. For instance when I check record count(*).

dc1:

cassandra@cqlsh> select count(*) from user.userElement;

count
-------
16

dc2:

cassandra@cqlsh> select count(*) from user.userElement;

count
-------
15

How could I check or any log could show what happened?

replication
1619923150133.png (3.4 KiB)
2 comments
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

David Jones-Gilardi avatar image David Jones-Gilardi ♦ ·

What happens if instead of using a count(*) you query specifically for the data you are looking for? Counts in C* might not always give you the number you expect because of how data is stored so just using a count to verify your data might be throwing you a red herring.

0 Likes 0 ·
Jason avatar image Jason David Jones-Gilardi ♦ ·

I had tried to export all the data out use CQL copy from both sites, and compare the result, it's missing one record in DC2.

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

The symptoms you described indicate to me that the nodes in DC2 were either unresponsive at some point or were down which is why they missed the write.

You will need to review the logs for indications of (a) dropped mutations, or (b) GC pauses which could account for the node being unresponsive. Cheers!

4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Jason avatar image Jason ·

Thank you Erick for your answer, however, my solution is active-Standby Geo Redundancy. The DC2 is a standby site that not take any traffic, so it's not a busy site that could busy for CUDR, or high memory usage need a full GC.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Jason ·

From a write perspective, it doesn't matter that DC2 is a "standby" site -- ALL mutations get sent to ALL replicas in ALL DCs.

For a replica in any DC to be inconsistent, it missed a write (a dropped mutation) for whatever reason. Cheers!

0 Likes 0 ·
Jason avatar image Jason Erick Ramirez ♦♦ ·

Got your point. How could I troubleshoot this issue to find the exact reason of missed a write? From Cassandra Trace level log? How to avoid it happened?


0 Likes 0 ·
Show more comments