spring-boot app keeps connecting to replaced node

Question

question

wederbrand asked Jan 29, '22 Erick Ramirez commented Mar 15, '22

spring-boot app keeps connecting to replaced node

We recently had an outage on one our nodes and it was replaced with a new one.

We run version 3.11.10 and followed the guide here with

 -Dcassandra.replace_address=address_of_dead_node

As far as I can tell, from the clusters point of view, all is fine. nodetool status reports the new node taking over, querying system.peers correctly shows the new ip and not the old ip.

However, our spring-boot apps, running spring-boot version 2.5.3, spring-data version 2021.0.3 and datastax cassandra drivers version 4.11.2, doesn't work as I would have hoped.

So I've recreated the problem locally with fo4r cassandra nodes running in docker. After 3 of them has been started and the app connected to them I kill one and start the fourth with replace_address of the one I killed.

As it did for us in the real cluster, everything on the cluster looks fine.

When the node was lost the app started to try to reconnect periodically, as expected, but when the new node was added these log lines (among others) could be found.

[s1] Updating token map but some nodes/tokens have changed, full rebuild

[s1] Unexpected error while refreshing token map, keeping previous version
(and in the exception on that line, extracted for readability)
Multiple entries with same key: 
   Murmur3Token(-1075386851749511590)=Node(endPoint=/192.168.14.14:9042, hostId=401f6950-a1bb-468a-9494-1bf71353bf0d, hashCode=603a11aa) 
   and 
   Murmur3Token(-1075386851749511590)=Node(endPoint=/192.168.14.12:9042, hostId=2ce63970-e286-412a-b2b4-9ed24fa1469f, hashCode=6213b695)

[s1] Node(endPoint=/192.168.14.14:9042, hostId=401f6950-a1bb-468a-9494-1bf71353bf0d, hashCode=603a11aa) was IGNORED, changing to LOCAL

It then goes on to prepare all the prepared statements, as expected, but it never queries .14 (the new node) and obviously not .12 (the old node). And it keeps trying to reconnect to .12 even though it has been replaced by now.

Do we need to do something different? Setup our apps differently? Is there an option to periodically forget all the tokens and fetch all fresh?

Thanks.

Edit 2022-02-03. Both when the app starts and when the node is replaced similar logs appear (and looks correct)

[s1|default] Evaluator did not assign a distance to node Node(endPoint=/192.168.14.13:9042, hostId=c3b8c4f9-7bbe-498f-8f15-b05f8a8dad07, hashCode=3e4efa)
[s1] com.datastax.oss.driver.internal.core.loadbalancing.DefaultLoadBalancingPolicy@1fbd0955 suggested Node(endPoint=/192.168.14.13:9042, hostId=c3b8c4f9-7bbe-498f-8f15-b05f8a8dad07, hashCode=3e4efa) to LOCAL, checking what other policies said
[s1] Shortest distance across all policies is LOCAL
[s1] Node(endPoint=/192.168.14.13:9042, hostId=c3b8c4f9-7bbe-498f-8f15-b05f8a8dad07, hashCode=3e4efa) was IGNORED, changing to LOCAL

[s1|default] Evaluator did not assign a distance to node Node(endPoint=/192.168.14.14:9042, hostId=d1889735-5587-4cfb-ab96-59415e8895f6, hashCode=20217314)
[s1] com.datastax.oss.driver.internal.core.loadbalancing.DefaultLoadBalancingPolicy@1fbd0955 suggested Node(endPoint=/192.168.14.14:9042, hostId=d1889735-5587-4cfb-ab96-59415e8895f6, hashCode=20217314) to LOCAL, checking what other policies said
[s1] Shortest distance across all policies is LOCAL
[s1] Node(endPoint=/192.168.14.14:9042, hostId=d1889735-5587-4cfb-ab96-59415e8895f6, hashCode=20217314) was IGNORED, changing to LOCAL

java driver spring-boot

10 |1000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

DataStax Enterprise is powered by the best distribution of Apache Cassandra ™

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Privacy Policy Terms of Use

Answer 1 · 2022-01-31T06:54:11Z

Erick Ramirez answered Jan 31, '22 Erick Ramirez commented Mar 15, '22

The output you posted doesn't appear to agree with your description of the issue. In this output, the nodes have different host IDs:

Murmur3Token(-1075386851749511590)=Node(endPoint=/192.168.14.14:9042, hostId=401f6950-a1bb-468a-9494-1bf71353bf0d, hashCode=603a11aa) 
Murmur3Token(-1075386851749511590)=Node(endPoint=/192.168.14.12:9042, hostId=2ce63970-e286-412a-b2b4-9ed24fa1469f, hashCode=6213b695)

On the other hand, I haven't come across any reports of this issue before. I'm not saying that it doesn't happen, just that I haven't heard of it.

Are you running with the open-source version of Apache Cassandra? I wanted to confirm that your not running with some other proprietary distribution that's isn't tested with the drivers. Cheers!

[UPDATE] After re-reading your post, I've had a bit of change of mind because of this line:

[s1] Node(endPoint=/192.168.14.14:9042, hostId=401f6950-a1bb-468a-9494-1bf71353bf0d, hashCode=603a11aa) was IGNORED, changing to LOCAL

This indicates to me that the new node doesn't belong to the same DC as the replaced node so the driver is ignoring it, preferring only nodes in the local DC. Cheers!

17 Share

10 |1000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez ♦♦ commented · Mar 15 at 02:22 AM

@wederbrand I've logged this as a bug in JAVA-3002. Cheers!

1 ·

wederbrand commented · Jan 31 at 07:29 AM

Hi!

We run standard open-source version of Cassandra and in my local docker based tests I'm running the official docker image with version 3.11.10

All the nodes belong the the standard test DC "datacenter1" which is also what I've set the property spring.data.cassandra.local-datacenter to.

Do I need to re-use the hostId? I find nothing regarding that in the "Replacing a dead node"-documentation.

I think the reason it IGNOREs the new node is because the token collisions.

0 ·

Erick Ramirez ♦♦ wederbrand commented · Feb 02 at 02:20 AM

It's actually the load balancing policy which generates the log entry about the node getting IGNORED. See the driver's LoadBalancingPolicyWrapper.java class here.

Since you already have debugging enabled, you should be able to see the node's computed distance and the shortest distance in the logs. It might give us an insight on why the node is being ignored. Cheers!

0 ·

wederbrand Erick Ramirez ♦♦ commented · Feb 02 at 08:17 AM

I did the debugging but nothing weird stands out. The new node also transitions from IGNORED -> LOCAL (I've updated the question with this).
I still think the problem is that the old broken node never get's removed.

I would expect some event to be fired saying that the old node was removed.
Debugging during the replace-phase I get a call to addNode() but never any calls to removeNode() which I would have expected.

This in turn leads to that the old node still looks to be holding the tokens and the collision happens.

After this all calls are sent to the two original nodes that are still up, none are sent to the new replacement node.

0 ·

Show more comments

question