question

nw avatar image
nw asked nw commented

How can I use SERIAL consistency for LWT for two active-active DC?

I have setup two data center with 5 nodes in each DC and each DC have connected two client application. I am using LWT with SERIAL consistency for some operation to get unique value from both DC.
2 DC, 1 Cluster,5 Nodes in each DC, NetworkTopologyStrategy, Replication factor : 3 when creating keyspace, Apache cassandra 3.11.4

I have following questions.

1. If one DC goes down then LWT query fail with error Not enough replicas available for query at consistency SERIAL. How can I resolve this scenario?

2. Option one works when I use LOCAL_SERAIL and set property 'allowRemoteDCsForLocalConsistencyLevel' at driver level. But in this case I am missing unique value from DC. How can I fix this?

Thanks in advance

lightweight transactions
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered nw commented

The challenge in your situation is that you have client apps connected to each DC. In this scenario, the result of the compare phase (of the compare-and-set) must be universal across the whole cluster because you can't have a client in DC1 thinking a partition doesn't exist and insert it while another in DC2 potentially does the same thing.

There's a tiny window where the 2 clients are writing the exact same partition at the exact same time. In practice this is almost non-existent unless these following two conditions are true:

  1. the 2 DCs are geographically very distant that there is high latency so there is a lag with data getting replicated between them, and
  2. it is possible for your apps to create the same partition in the 2 different DCs at exactly the same time.

If these two conditions exist in your use case then you have no choice but to use SERIAL consistency which means your app cannot tolerate a full DC outage. However, I would argue that condition (2) only exists in limited cases and if this doesn't apply to your case then LOCAL_SERIAL is acceptable. Cheers!

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

nw avatar image nw commented ·

Thanks for quick reply Erick. I need to implement DC failover or some nodes goes down so I will use LOCAL_SERIAL and if there are some data discrepancy that I will handle in application level. I observed error like NoHostAvailableException, OperationTimedOutException when some nodes goes down. Do I need to implement retry policy to handle this error?

0 Likes 0 ·