We are hosting a Java Spring Boot service which connects to DSE graph 6.0.6 via k8. The nodes are exposed via a dns host name.
When kubernetes shuffles and realocates a node to a new ip address, the driver cannot find the cluster nodes anymore and the service stops working. Is the driver connection caching underlying host ips? How can we be resilient to k8 rebalancing?
@sean.fennell_168661 Depending on which driver you're using, it is possible to get around the kubernetes behaviour of shuffling nodes around and assigning IPs. If you were using the Java driver for example, in the OSS 4.x versions and DataStax 2.x versions you can make the driver resolve the hostnames every time it opens a new connection with:
advanced.resolve-contact-points = false
This won't guarantee there won't be interruptions to your application particularly when a high number of nodes have changed IPs. But at least the driver will be able to establish new connections to the new IPs.
Credit goes to @olivier.michallat_30685 for tips on this issue. For more information, see the Java driver Reference Configuration document. Cheers!
EDITED: Changed to "false". Thanks to @alexandre.dutra for spotting the mistake. :)
Since you are using Spring Boot, you might want to upgrade it to 2.3.0.M3 in order to be able to use the latest Java driver (4.5.0):
https://github.com/spring-projects/spring-boot/releases/tag/v2.3.0.M3
Also, with k8s you actually want that property to be false:
advanced.resolve-contact-points = false
which maven repositories do you recommend I investigate?
I'm currently using:
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-java-driver-core</artifactId>
<version>1.6.7</version>
</dependency>
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-java-driver-graph</artifactId>
<version>1.6.7</version>
</dependency>
The above coordinates are for the legacy DSE Java driver 1.x. You won't be able to benefit from the fix explained by @Erick Ramirez unless you upgrade to DSE Java driver 2.0+ or OSS Java driver 4.5+.
Once/if you upgrade, the driver coordinates should look like the one below:
<dependency>
<groupId>com.datastax.oss</groupId>
<artifactId>java-driver-core</artifactId>
<version>4.5.1</version>
</dependency>
If upgrading is not an option for you, I suggest that you try @jim.dickinson_187342 suggestion instead. Good luck!
Are you running both DSE and the client app inside Kubernetes? If so, can you make a ClusterIP service per node? That would create static IPs within the Kubernetes cluster that you could pass to the client app.
6 People are following this question.
