question

rajib76 avatar image
rajib76 asked Erick Ramirez answered

Getting NoNodeAvailableException when inserting loads of data

We are getting the below exception sometimes while inserting loads of data into Cassandra

Exception occurred while reading Inventory Location java.lang.RuntimeException: com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the query","origin":"rep","source_instance":"0","source_type":"APP/PROC/WEB","timestamp":1644006755885638164}

Is there a way to find out if this is happening because we saturated all the connections to the nodes in the cluster. Where do we setup the maximum connections that a node can take.

In this case, is it recommended to setup throttling and connection pooling. If we setup throttling, is there a scenario where we may lose some inserts to the database or will the requests be queued and later executed once the load comes down

dsejava driver
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

The NoNodeAvailableException gets thrown by the driver when there are no nodes available to contact as request coordinators. This happens when the driver has marked all nodes as "down" because they are down (Cassandra is stopped/shutdown) or unresponsive (in a long GC pause for example).

Another cause is when the available streams per connection (32K max) has run out and the connected node cannot accept anymore requests.

In almost all cases, increasing the maximum requests per connection doesn't yield any benefits. The connections get maxed out because the nodes are busy processing requests and cannot keep up with the traffic. Tuning the connection pool will make no difference if the nodes are already overloaded.

There are two solutions to this problem:

  1. Do not overload your cluster. Instead, limit the traffic from the application with Request throttling.
  2. Increase the capacity of your cluster by adding more nodes.

For more information, see the Tuning section of the Java driver Connection pooling document. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.