DataStax Academy FAQ

DataStax Academy migrated to a new learning management system (LMS) in July 2020. We are also moving to a new Cassandra Certification process so there are changes to exam bookings, voucher system and issuing of certificates.

Check out the Academy FAQ pages for answers to your questions:


question

colleen.velo_21840 avatar image
colleen.velo_21840 asked ·

Node completing bootstrapping process without streaming any data

We are seeing a strange issue on our multiple DC `v2.2.13` Apache Cassandra ring cluster that is running on AWS.

Our DCs are in the us-east-1/eu-west-1/ap-northeast-1 regions and we are using IPv6 for Cassandra communication.

We have been able to successfully add nodes to the us-east-1 and eu-west-1 DCs, but we are hitting a strange intermittent issue when trying to add additional nodes in the ap-northeast-1 region. The bootstrapping process will start and then immediately complete without streaming any data. (Note: we have triple-checked that the new nodes are _not_ seed nodes). Again, this appears to be intermittent.

Has anyone seen this before?


cassandrabootstrapping
1 comment
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Can you post a copy of the system.log so we could review it? It sounds like there was nothing to stream to the node but the log will give us a better idea of what's going on. Cheers!

0 Likes 0 · ·
colleen.velo_21840 avatar image
colleen.velo_21840 answered ·

Here is a detailed code walkthrough that was posted to the Cassandra-Dev group mailing list:

* https://lists.apache.org/thread.html/456bd3b7eae65bf8b528e5e2ec5e25be8928f374a2ba5dbd0d1274ca@%3Cdev.cassandra.apache.org%3E

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@brasco --Donnie Roberson-- avatar image
@brasco --Donnie Roberson-- answered ·

It looks like there has been a recently filed Jira about bootstrapping completing without streaming data on 2.2.13 with very similar configuration. Is this your Jira? If not are you seeing anything similar in your logs that are reported in https://issues.apache.org/jira/browse/CASSANDRA-15155

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@@brasco --Donnie Roberson-- — that is the Jira ticket that our company filed.

0 Likes 0 · ·
ben.krug_85176 avatar image
ben.krug_85176 answered ·

You mentioned that you verified, it's not configured as a seed, and auto_bootstrap is not set to false. Also (separately) you said it joined the proper DC, and jumped to 'UN' state.

Did it show as 'UN' from other nodes, or did you just check on the node itself? In bootstrap, the node will say it's 'UN' but other nodes will say it's 'DN'.

How long did you give the process? Before streaming, other nodes have to do anticompaction to gather the data to stream, and in some cases this can take time, during which there will be no streaming. Did it show 'UN' from all nodes, and have no data?

2 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@ben.krug_85176 — excellent question — yes, the newly added node (w/o data) was seen as "UN" by the other nodes in the cluster as well.

0 Likes 0 · ·

@ben.krug_85176

Also, regarding the bootstrapping process, I noticed this oddity in the Cassandra system.log file — the "JOINING: Starting to bootstrap" and "Bootstrap completed! for the tokens" messages occur within the same minute:

65057:INFO [main] 2019-06-12 23:22:10,586 StorageService.java:1142 - JOINING: Starting to bootstrap...

65059:INFO [main] 2019-06-12 23:22:10,597 StorageService.java:1199 - Bootstrap completed! for the tokens [3572182645598824729, 33164969472.......]

0 Likes 0 · ·