question

irisha_mur_186571 avatar image
irisha_mur_186571 asked Erick Ramirez edited

Getting lots of NoNodeAvailableException after upgrading to Java driver 4.9.0

We recently migrated on new driver to connect to Cassandra - 4.9.0. We started to see a lot of exceptions like

"No node was available to execute the query" and "Query timed out after PT2S "


We tried to improve a lot of properties without any luck. Do you have suggestions how to investigate these issues?

datastax-java-driver {
 
  basic.request {
    timeout = 500 seconds
  }
 
  basic.load-balancing-policy {
    local-datacenter = "us-east"
  }
 
  advanced.connection {
    pool {
      local {
        size = 32
      }
      remote {
        size = 32
      }
    }
    max-requests-per-connection = 2048
    max-orphan-requests = 800
  }
 
  advanced.reconnect-on-init = true
 
  advanced.reconnection-policy {
    class = com.assetcontrol.acx.persistence.cassandra.ConfigurableRetryPolicy
  }
 
  advanced.heartbeat {
    interval = 30 seconds
  }
 
  advanced.continuous-paging {
    timeout {
      first-page = 60 seconds
      other-pages = 30 seconds
    }
  }
 }
java driver
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

This exception message:

No node was available to execute the query

happens when no coordinators were contacted because the driver thinks all nodes are down and unavailable.

It is possible that you are saturating the connections to the nodes in the cluster and the driver eventually marks them as unavailable. It gets to a point where all the nodes are marked as "DOWN" and there are none left so you end up with the NoNodeAvailableException.

Maybe consider throttling the requests using the advanced.throttler configuration. Here's an example where the number of concurrent requests are limited to 10K and additional requests get queued:

  advanced.throttler {
    class = ConcurrencyLimitingRequestThrottle
    max-concurrent-requests = 10000
    max-queue-size = 100000
  }

For other throttling options and details, see Request throttling. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.