question

Using a CL of ONE has the ability to hit any DC.

The driver will learn the cluster's topology from the contact point(s) you provide, but when you use ONE the coordinator will choose the replica that is determined to be the closest or best performing by the snitch. The snitch can route around nodes with higher latencies, so you have the ability to hit any DC.

If you desire to hit only the DC you are connecting to, you should use a CL of LOCAL_ONE instead.

With a consistency level of ONE, your queries can be routed to any node in any DC. The contact points that you provide in your properties file are only for the initial contact, from there the driver grabs the system.peers info and then uses any node within that peers table. That means the driver can connect to any node in the cluster at this point. However, you can control the load balancing using a DCInferring or DCAwareRoundRobin, or something along those lines. If you do specify a load balancing policy that chooses the local dc only, then the coordinator should always be in the local dc.

From the coordinator perspective, the coordinator decides the replica based on the latencies. If a node in dc2 is faster than the local, the coordinator will choose the node in dc2. The idea is that if you bombard one dc, then the other dc's may actually be faster, so the coordinator will use the other dc.

Also, please keep in mind that for writes all nodes receive the write, but only ONE is required to ack for the write to be considered successful.

If you want to avoid this contact to the non-local dc's, please use LOCAL_ONE.

The Java driver generates a query plan for each request -- a list of cluster nodes to contact, tried in order one at a time until contact is established. In an optimal situation where all nodes are fully operational, the first node in the query plan is used to execute a query.

The query plan is (1) different for each query as a means of (2) balancing the load across nodes in the cluster. The query plan (3) only contains nodes which are available to process requests and does not include (4) nodes which are ignored (not in the "local" DC) or (5) unavailable (down/unresponsive).

Unless you're running with a very old version of Spring, the Java driver uses a default load-balancing policy that will only ever include local nodes in the query plan which means that requests will never get routed to remote DCs regardless of whether the request is configured with a local consistency or not.

You will need to configure the driver with the name of the DC which is "local" to it. For example:

datastax-java-driver.basic.load-balancing-policy {
  local-datacenter = DC1
}

CqlSession session = CqlSession.builder()
    .withLocalDatacenter("DC1")
    .build();

Requests only get routed to the local DC to prevent cross-DC latency. You need to explicitly configure the driver to allow remote nodes in the query plan. Even if you allow remote nodes to be used, the driver will not include them for local consistency levels unless you explicitly configure the driver to allow it.

To be clear, we do NOT recommend configuring DC-failover or allowing remote nodes at the app/client level -- you should perform the failover at the infrastructure layer, meaning you should divert traffic to app instances in another DC while you investigate/resolve the issues in the local DC.

For more details, see the Java driver Load balancing document. Cheers!