Async Graph Execution Failing Intermittently with NoNodeAvailableException

Question

question

dharm.parakh_188244 asked Aug 17, '20 dharm.parakh_188244 commented Aug 20, '20

Async Graph Execution Failing Intermittently with NoNodeAvailableException

I have a batch job used to publish data on DSE Graph. The job keeps Cassandra session cached through its lifecycle and keep creating "GraphTraversal" objects as needed. In my test run of few hundred mutation traversals it worked fine, when I did execution of around 50k traversals, I see few of the traversals failing intermittently with following error:

com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the query
        at com.datastax.oss.driver.api.core.AllNodesFailedException.fromErrors(AllNodesFailedException.java:53)
        at com.datastax.dse.driver.internal.core.cql.continuous.ContinuousRequestHandlerBase.sendRequest(ContinuousRequestHandlerBase.java:366)
        at com.datastax.dse.driver.internal.core.cql.continuous.ContinuousRequestHandlerBase.onThrottleReady(ContinuousRequestHandlerBase.java:276)
        at com.datastax.oss.driver.internal.core.session.throttling.PassThroughRequestThrottler.register(PassThroughRequestThrottler.java:52)
        at com.datastax.dse.driver.internal.core.graph.ContinuousGraphRequestHandler.<init>(ContinuousGraphRequestHandler.java:90)
        at com.datastax.dse.driver.internal.core.graph.GraphRequestAsyncProcessor.process(GraphRequestAsyncProcessor.java:83)
        at com.datastax.dse.driver.internal.core.graph.GraphRequestAsyncProcessor.process(GraphRequestAsyncProcessor.java:38)
        at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
        at com.datastax.dse.driver.api.core.graph.GraphSession.executeAsync(GraphSession.java:82)

We are building the traversals using Fluent APIs and then running it explicit using async execution of DSE driver.

com.datastax.dse.driver.api.core.graph.GraphSession#executeAsync

code snippet:

CompletionStage<AsyncGraphResultSet> res =
            cassandraConnector.getCqlSession().executeAsync(FluentGraphStatement.newInstance(traversal));
    res.whenComplete((r, t) -> {
        if (t != null) {
            logger.error("error occured while executing traversals.", t);
            reportError(t);
        }
    });

Here is my driver configuration:

datastax-java-driver {
    basic {
        contact-points=["myhost:9042"]
        load-balancing-policy.local-datacenter = GraphDC1
        session-name = cfx-adv
        request.page-size = 50000
        config-reload-interval = 0
    }
    basic.graph{
        name=iag_test
        traversal-source="g"
        timeout = 60 seconds
    }
    advanced {
        protocol.compression = lz4
        connection {
          max-requests-per-connection = 1024
          pool {
            local.size = 1
            remote.size = 1
          }
        }
      }#end of advanced
}

DSE Version: 6.8.1, Driver : 4.8

Can somebody help me identify why driver is not able to find the node while service is up and running. Is it something related to driver configuration or thread pool ?

dse graph java driver

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-08-18T04:04:22Z

Erick Ramirez answered Aug 18, '20 dharm.parakh_188244 commented Aug 20, '20

I have noted that you're using localhost as a contact point for the driver:

datastax-java-driver {
    basic {
        contact-points=["localhost:9042"]
        ...
    }
    ...
}

If I were to guess, you are connecting to nodes in a cluster that is not listening for client connections on localhost.

You need to configure contact points using the IP address configured on native_transport_address. An easy way to verify which IP is bound to the CQL client port 9042 is by running the following command on the DSE nodes:

$ sudo lsof -nPi -sTCP:LISTEN

Cheers!

[UPDATE] The NoNodeAvailableException is thrown for 2 reasons:

all nodes are down (or unreachable)
all the contact points provided are invalid

I'm inclined to think that it's the latter because the driver also reported AllNodesFailedException. Again, you need to use the correct IP address that is configured on native_transport_address.

7 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

dharm.parakh_188244 · Aug 18, 2020 at 04:51 AM

@Erick Ramirez Sorry for causing confusion, I have correct hostname ip in my app configuration, so it is not related to that.

0 ·

Erick Ramirez ♦♦ dharm.parakh_188244 · Aug 18, 2020 at 05:00 AM

It is still related. Let me update my answer with the details.

0 ·

dharm.parakh_188244 Erick Ramirez ♦♦ · Aug 18, 2020 at 05:22 AM

Thank for the update, as I mentioned we are updating 50k + records in sequence and only few of them are failing. So how this can be related ?

0 ·

Show more comments

question