question

colleen.velo_21840 avatar image
colleen.velo_21840 asked Erick Ramirez commented

Node completing bootstrapping process without streaming any data

We are seeing a strange issue on our multiple DC `v2.2.13` Apache Cassandra ring cluster that is running on AWS.

Our DCs are in the us-east-1/eu-west-1/ap-northeast-1 regions and we are using IPv6 for Cassandra communication.

We have been able to successfully add nodes to the us-east-1 and eu-west-1 DCs, but we are hitting a strange intermittent issue when trying to add additional nodes in the ap-northeast-1 region. The bootstrapping process will start and then immediately complete without streaming any data. (Note: we have triple-checked that the new nodes are _not_ seed nodes). Again, this appears to be intermittent.

Has anyone seen this before?


cassandrabootstrapping
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image Erick Ramirez ♦♦ commented ·

Can you post a copy of the system.log so we could review it? It sounds like there was nothing to stream to the node but the log will give us a better idea of what's going on. Cheers!

0 Likes 0 ·

3 Answers

ben.krug_85176 avatar image
ben.krug_85176 answered colleen.velo_21840 commented

You mentioned that you verified, it's not configured as a seed, and auto_bootstrap is not set to false. Also (separately) you said it joined the proper DC, and jumped to 'UN' state.

Did it show as 'UN' from other nodes, or did you just check on the node itself? In bootstrap, the node will say it's 'UN' but other nodes will say it's 'DN'.

How long did you give the process? Before streaming, other nodes have to do anticompaction to gather the data to stream, and in some cases this can take time, during which there will be no streaming. Did it show 'UN' from all nodes, and have no data?

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

colleen.velo_21840 avatar image colleen.velo_21840 commented ·

@ben.krug_85176 — excellent question — yes, the newly added node (w/o data) was seen as "UN" by the other nodes in the cluster as well.

0 Likes 0 ·
colleen.velo_21840 avatar image colleen.velo_21840 commented ·

@ben.krug_85176 —

Also, regarding the bootstrapping process, I noticed this oddity in the Cassandra system.log file — the "JOINING: Starting to bootstrap" and "Bootstrap completed! for the tokens" messages occur within the same minute:

65057:INFO [main] 2019-06-12 23:22:10,586 StorageService.java:1142 - JOINING: Starting to bootstrap...

65059:INFO [main] 2019-06-12 23:22:10,597 StorageService.java:1199 - Bootstrap completed! for the tokens [3572182645598824729, 33164969472.......]

0 Likes 0 ·
@brasco --Donnie Roberson-- avatar image
@brasco --Donnie Roberson-- answered colleen.velo_21840 commented

It looks like there has been a recently filed Jira about bootstrapping completing without streaming data on 2.2.13 with very similar configuration. Is this your Jira? If not are you seeing anything similar in your logs that are reported in https://issues.apache.org/jira/browse/CASSANDRA-15155

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

colleen.velo_21840 avatar image colleen.velo_21840 commented ·

@@brasco --Donnie Roberson-- — that is the Jira ticket that our company filed.

0 Likes 0 ·
colleen.velo_21840 avatar image
colleen.velo_21840 answered

Here is a detailed code walkthrough that was posted to the Cassandra-Dev group mailing list:

* https://lists.apache.org/thread.html/456bd3b7eae65bf8b528e5e2ec5e25be8928f374a2ba5dbd0d1274ca@%3Cdev.cassandra.apache.org%3E

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.