question

dharm.parakh_188244 avatar image
dharm.parakh_188244 asked dharm.parakh_188244 commented

Async Graph Execution Failing Intermittently with NoNodeAvailableException

I have a batch job used to publish data on DSE Graph. The job keeps Cassandra session cached through its lifecycle and keep creating "GraphTraversal" objects as needed. In my test run of few hundred mutation traversals it worked fine, when I did execution of around 50k traversals, I see few of the traversals failing intermittently with following error:

com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the query
        at com.datastax.oss.driver.api.core.AllNodesFailedException.fromErrors(AllNodesFailedException.java:53)
        at com.datastax.dse.driver.internal.core.cql.continuous.ContinuousRequestHandlerBase.sendRequest(ContinuousRequestHandlerBase.java:366)
        at com.datastax.dse.driver.internal.core.cql.continuous.ContinuousRequestHandlerBase.onThrottleReady(ContinuousRequestHandlerBase.java:276)
        at com.datastax.oss.driver.internal.core.session.throttling.PassThroughRequestThrottler.register(PassThroughRequestThrottler.java:52)
        at com.datastax.dse.driver.internal.core.graph.ContinuousGraphRequestHandler.<init>(ContinuousGraphRequestHandler.java:90)
        at com.datastax.dse.driver.internal.core.graph.GraphRequestAsyncProcessor.process(GraphRequestAsyncProcessor.java:83)
        at com.datastax.dse.driver.internal.core.graph.GraphRequestAsyncProcessor.process(GraphRequestAsyncProcessor.java:38)
        at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
        at com.datastax.dse.driver.api.core.graph.GraphSession.executeAsync(GraphSession.java:82)

We are building the traversals using Fluent APIs and then running it explicit using async execution of DSE driver.

com.datastax.dse.driver.api.core.graph.GraphSession#executeAsync

code snippet:

CompletionStage<AsyncGraphResultSet> res =
            cassandraConnector.getCqlSession().executeAsync(FluentGraphStatement.newInstance(traversal));
    res.whenComplete((r, t) -> {
        if (t != null) {
            logger.error("error occured while executing traversals.", t);
            reportError(t);
        }
    });


Here is my driver configuration:

datastax-java-driver {
    basic {
        contact-points=["myhost:9042"]
        load-balancing-policy.local-datacenter = GraphDC1
        session-name = cfx-adv
        request.page-size = 50000
        config-reload-interval = 0
    }
    basic.graph{
        name=iag_test
        traversal-source="g"
        timeout = 60 seconds
    }
    advanced {
        protocol.compression = lz4
        connection {
          max-requests-per-connection = 1024
          pool {
            local.size = 1
            remote.size = 1
          }
        }
      }#end of advanced
}


DSE Version: 6.8.1, Driver : 4.8

Can somebody help me identify why driver is not able to find the node while service is up and running. Is it something related to driver configuration or thread pool ?


dsejava drivergraph
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered dharm.parakh_188244 commented

I have noted that you're using localhost as a contact point for the driver:

datastax-java-driver {
    basic {
        contact-points=["localhost:9042"]
        ...
    }
    ...
}

If I were to guess, you are connecting to nodes in a cluster that is not listening for client connections on localhost.

You need to configure contact points using the IP address configured on native_transport_address. An easy way to verify which IP is bound to the CQL client port 9042 is by running the following command on the DSE nodes:

$ sudo lsof -nPi -sTCP:LISTEN

Cheers!

[UPDATE] The NoNodeAvailableException is thrown for 2 reasons:

  • all nodes are down (or unreachable)
  • all the contact points provided are invalid

I'm inclined to think that it's the latter because the driver also reported AllNodesFailedException. Again, you need to use the correct IP address that is configured on native_transport_address.

7 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

dharm.parakh_188244 avatar image dharm.parakh_188244 commented ·

@Erick Ramirez Sorry for causing confusion, I have correct hostname ip in my app configuration, so it is not related to that.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ dharm.parakh_188244 commented ·

It is still related. Let me update my answer with the details.

0 Likes 0 ·
dharm.parakh_188244 avatar image dharm.parakh_188244 Erick Ramirez ♦♦ commented ·

Thank for the update, as I mentioned we are updating 50k + records in sequence and only few of them are failing. So how this can be related ?

0 Likes 0 ·
Show more comments