Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

sean.fennell_168661 avatar image
sean.fennell_168661 asked ·

DSE Graph 6.0.6 on k8s cannot find hosts if underlying IP changes

We are hosting a Java Spring Boot service which connects to DSE graph 6.0.6 via k8. The nodes are exposed via a dns host name.

When kubernetes shuffles and realocates a node to a new ip address, the driver cannot find the cluster nodes anymore and the service stops working. Is the driver connection caching underlying host ips? How can we be resilient to k8 rebalancing?

dsegraphkubernetes
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

jim.dickinson_187342 avatar image
jim.dickinson_187342 answered ·

Are you running both DSE and the client app inside Kubernetes? If so, can you make a ClusterIP service per node? That would create static IPs within the Kubernetes cluster that you could pass to the client app.

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image
Erick Ramirez answered ·

@sean.fennell_168661 Depending on which driver you're using, it is possible to get around the kubernetes behaviour of shuffling nodes around and assigning IPs. If you were using the Java driver for example, in the OSS 4.x versions and DataStax 2.x versions you can make the driver resolve the hostnames every time it opens a new connection with:

advanced.resolve-contact-points = false

This won't guarantee there won't be interruptions to your application particularly when a high number of nodes have changed IPs. But at least the driver will be able to establish new connections to the new IPs.

Credit goes to @olivier.michallat_30685 for tips on this issue. For more information, see the Java driver Reference Configuration document. Cheers!

EDITED: Changed to "false". Thanks to @alexandre.dutra for spotting the mistake. :)

4 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Since you are using Spring Boot, you might want to upgrade it to 2.3.0.M3 in order to be able to use the latest Java driver (4.5.0):

https://github.com/spring-projects/spring-boot/releases/tag/v2.3.0.M3

2 Likes 2 · ·

Also, with k8s you actually want that property to be false:

advanced.resolve-contact-points = false
0 Likes 0 · ·

which maven repositories do you recommend I investigate?

I'm currently using:

        <dependency>
            <groupId>com.datastax.dse</groupId>
            <artifactId>dse-java-driver-core</artifactId>
            <version>1.6.7</version>
        </dependency>
        <dependency>
            <groupId>com.datastax.dse</groupId>
            <artifactId>dse-java-driver-graph</artifactId>
            <version>1.6.7</version>
        </dependency>
0 Likes 0 · ·

The above coordinates are for the legacy DSE Java driver 1.x. You won't be able to benefit from the fix explained by @Erick Ramirez unless you upgrade to DSE Java driver 2.0+ or OSS Java driver 4.5+.

Once/if you upgrade, the driver coordinates should look like the one below:

<dependency>
    <groupId>com.datastax.oss</groupId>
    <artifactId>java-driver-core</artifactId>
    <version>4.5.1</version>
</dependency>

If upgrading is not an option for you, I suggest that you try @jim.dickinson_187342 suggestion instead. Good luck!

0 Likes 0 · ·