Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

Jason avatar image
Jason asked ·

How can I check for the cause of inconsistencies between DC1 and DC2?

I have two Datacenter, dc1 and dc2 on Openstack, each datacenter have 3 nodes. use the first node of both datacenter as seeds node.

keyspace "userElement" setting is 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 3};

My application use LOCAL_QUORUM to write data to "user" table inside "userElement" keyspace. normally, the replication working fine, all data write into dc1 will replicate to dc2. But sometimes, one of the records missing on dc2. For instance when I check record count(*).

dc1:

cassandra@cqlsh> select count(*) from user.userElement;

count
-------
16

dc2:

cassandra@cqlsh> select count(*) from user.userElement;

count
-------
15

How could I check or any log could show what happened?

replication
1619923150133.png (3.4 KiB)
2 comments
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

What happens if instead of using a count(*) you query specifically for the data you are looking for? Counts in C* might not always give you the number you expect because of how data is stored so just using a count to verify your data might be throwing you a red herring.

0 Likes 0 ·
Jason avatar image Jason David Jones-Gilardi ♦ ·

I had tried to export all the data out use CQL copy from both sites, and compare the result, it's missing one record in DC2.

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

The symptoms you described indicate to me that the nodes in DC2 were either unresponsive at some point or were down which is why they missed the write.

You will need to review the logs for indications of (a) dropped mutations, or (b) GC pauses which could account for the node being unresponsive. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.