Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

guzelcihad avatar image
guzelcihad asked Erick Ramirez answered

Does the Java driver randomly route requests to nodes in all DCs?

Hi,

There are 3 dc available in our infra. I saw that queries are configured consistency level of ONE for write and read queries. In this case, I am wondering how the cassandra behaves.

I am sure cassandra client sends request to one dc (because properties file in spring includes all the node ip's in dc1), but I am wondering how coordinator node decides to route this request to other nodes?

Does it route request randomly include all the nodes available in other dc?

Here is the keyspace definition:

CREATE KEYSPACE <keyspace> WITH replication = {'class': 'NetworkTopologyStrategy', 'backup_dc1': '1', 'backup_dc2': '1', 'dc1': '3', 'dc2': '3', 'dc3': '3'}  AND durable_writes = true;
java driverconsistency level
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

starlord avatar image
starlord answered guzelcihad commented

Using a CL of ONE has the ability to hit any DC.

The driver will learn the cluster's topology from the contact point(s) you provide, but when you use ONE the coordinator will choose the replica that is determined to be the closest or best performing by the snitch. The snitch can route around nodes with higher latencies, so you have the ability to hit any DC.

If you desire to hit only the DC you are connecting to, you should use a CL of LOCAL_ONE instead.

1 comment Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Hi @starlord, I saw conflict in your words. You first said that the coordinator will choose the closest replica. Then you said it is possible to route the node with higher latencies. Can you clarify me? Thank you
0 Likes 0 ·
steve.lacerda avatar image
steve.lacerda answered steve.lacerda commented

With a consistency level of ONE, your queries can be routed to any node in any DC. The contact points that you provide in your properties file are only for the initial contact, from there the driver grabs the system.peers info and then uses any node within that peers table. That means the driver can connect to any node in the cluster at this point. However, you can control the load balancing using a DCInferring or DCAwareRoundRobin, or something along those lines. If you do specify a load balancing policy that chooses the local dc only, then the coordinator should always be in the local dc.

From the coordinator perspective, the coordinator decides the replica based on the latencies. If a node in dc2 is faster than the local, the coordinator will choose the node in dc2. The idea is that if you bombard one dc, then the other dc's may actually be faster, so the coordinator will use the other dc.

Also, please keep in mind that for writes all nodes receive the write, but only ONE is required to ack for the write to be considered successful.

If you want to avoid this contact to the non-local dc's, please use LOCAL_ONE.

2 comments Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Can we say that the coordinator always choose the low latency path in the cluster without concerning about the any dc?

0 Likes 0 ·

No, you can't say that. The coordinator follows the consistency level, so if the consistency is LOCAL_ONE, LOCAL_QUORUM, et cetera, then the coordinator will only choose from the local dc. If the CL is not local, then yes, that's true. The coordinator will choose whichever nodes have the lowest latency in order to fulfill the request.

0 Likes 0 ·
Erick Ramirez avatar image
Erick Ramirez answered

The Java driver generates a query plan for each request -- a list of cluster nodes to contact, tried in order one at a time until contact is established. In an optimal situation where all nodes are fully operational, the first node in the query plan is used to execute a query.

The query plan is (1) different for each query as a means of (2) balancing the load across nodes in the cluster. The query plan (3) only contains nodes which are available to process requests and does not include (4) nodes which are ignored (not in the "local" DC) or (5) unavailable (down/unresponsive).

Unless you're running with a very old version of Spring, the Java driver uses a default load-balancing policy that will only ever include local nodes in the query plan which means that requests will never get routed to remote DCs regardless of whether the request is configured with a local consistency or not.

You will need to configure the driver with the name of the DC which is "local" to it. For example:

datastax-java-driver.basic.load-balancing-policy {
  local-datacenter = DC1
}
CqlSession session = CqlSession.builder()
    .withLocalDatacenter("DC1")
    .build();

Requests only get routed to the local DC to prevent cross-DC latency. You need to explicitly configure the driver to allow remote nodes in the query plan. Even if you allow remote nodes to be used, the driver will not include them for local consistency levels unless you explicitly configure the driver to allow it.

To be clear, we do NOT recommend configuring DC-failover or allowing remote nodes at the app/client level -- you should perform the failover at the infrastructure layer, meaning you should divert traffic to app instances in another DC while you investigate/resolve the issues in the local DC.

For more details, see the Java driver Load balancing document. Cheers!

Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.