question

utkarsh-devops avatar image
utkarsh-devops asked Erick Ramirez commented

Rebuild returns "Unable to find sufficient sources for streaming range"

Team,

I have a Cassandra cluster of 6 nodes and I've added a new datacenter to the existing setup of 6 nodes. I’ve followed all the steps but getting the below error when I run nodetool rebuild on the new dc’s nodes

$ nodetool rebuild -- datacenter1
nodetool: Unable to find sufficient sources for streaming range (389537640310009811,390720100939923035] in keyspace system_distributed
See 'nodetool help' or 'nodetool help <command>'.

Nodetool status

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address        Load     Tokens Owns (effective) Host ID                   Rack
UN 172.21.201.205 1.75 GiB 256    16.5% ff0accd4-c33a-4984-967f-3ec763fe5414 rack1
UN 172.21.201.45  1.55 GiB 256    17.0% d3ac5afa-d561-43ee-89e2-db1d20c59b38 rack1
UN 172.21.201.44  2.37 GiB 256    17.0% 73d8e6c6-0aa3-4a91-80fc-8c7068c78a64 rack1
UN 172.21.201.207 1020.15 MiB 256 16.0% 5751ea7d-b339-43b3-bcfe-89fcbc60dea0 rack1
UN 172.21.201.46  1.64 GiB 256    17.0% 1c1afbfc-6a4b-40f0-a4c3-1eaa543eb2d5 rack1
UN 172.21.201.206 1.13 GiB 256    17.2% b11bfef9-e708-45cc-9ab3-e52983834096 rack1
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address     Load       Tokens Owns (effective) Host ID                   Rack
UN 10.41.6.155 983.91 KiB 256    17.3% bf7244bb-70dc-4d91-8131-cbe4886f09e7 rack1
UN 10.41.6.157 946.36 KiB 256    15.5% 5499e7cc-db23-4163-8f0c-8f437f61bd6f rack1
UN 10.41.6.156 1.14 MiB   256    15.3% f27e94a6-7e1c-4177-9f88-36d821a7808d rack1
UN 10.41.6.159 659.3 KiB  256    17.3% 453e97df-5b83-4798-9e5e-a13bbb33acee rack1
UN 10.41.6.158 909.51 KiB 256    18.2% a4bc046a-e2ef-4fd4-9ab7-18be642a4d5a rack1
UN 10.41.6.160 1.08 MiB   256    15.5% 267cf9d0-cd55-4186-a737-998443125b19 rack1

node tool describe cluster status

# nodetool describecluster
Cluster Information:
        Name: Recommendation
        Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
        DynamicEndPointSnitch: enabled
        Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
        Schema versions:
                ea63e099-37c5-3d7b-9ace-32f4c833653d: [10.41.6.155, 10.41.6.157, 10.41.6.156, 10.41.6.159, 10.41.6.158, 10.41.6.160]
                fd507b64-3070-3ffd-8217-f45f2f188dfc: [172.21.201.205, 172.21.201.45, 172.21.201.44, 172.21.201.207, 172.21.201.206, 172.21.201.46]

Can someone quickly help me fix this issue?

rebuild
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

utkarsh-devops avatar image utkarsh-devops commented ·

Found the issue, Both the DC had different Cassandra version due to which we've observed schema mismatch.

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

The most likely cause of this error is incorrect replication settings on the keyspace:

Unable to find sufficient sources for streaming range (389537640310009811,390720100939923035] in keyspace system_distributed

For example if you misspelled the DC names in the keyspace replication, Cassandra won't be able to find replicas for particular ranges since the DC wouldn't exist.

As I previously stated in question #5024, I recommend changing the replication strategy of system keyspaces configured with SimpleStrategy to NetworkTopologyStrategy with 3 replicas in each DC. Specifically these keyspaces:

CREATE KEYSPACE system_auth WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}  AND durable_writes = true;
CREATE KEYSPACE system_distributed WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;
CREATE KEYSPACE system_traces WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = true;

WARNING: I do not recommend you alter keyspaces which have LocalStrategy. Specifically these keyspaces:

CREATE KEYSPACE system_schema WITH replication = {'class': 'LocalStrategy'}  AND durable_writes = true;
CREATE KEYSPACE system WITH replication = {'class': 'LocalStrategy'}  AND durable_writes = true;

There is a side issue that there is schema disagreement between the two DCs. You will need to resolve that issue before you can proceed with the rebuild.

Note that if the new nodes in dc1 DC were previously part of another cluster, you cannot just add them to a new cluster. You need to completely wipe the contents of the data directory including saved_caches and commitlog before you can add them to a new cluster. Otherwise, they will have a version of the schema which will clash with the schema of the existing cluster. Cheers!

[UPDATE] In the discussion on ASF Slack, the underlying problem with the schema disagreement was the mixed-versions between the two DCs.

As I stated in my original answer, you needed to resolve the schema disagreement first. As you discovered, it isn't possible to make schema changes when the cluster is running with mixed-versions.

4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

utkarsh-devops avatar image utkarsh-devops commented ·

The fix was to keep the Cassandra version same on both the DC's and it worked well.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ utkarsh-devops commented ·

I've noted the discussion on ASF Slack. Cheers!

0 Likes 0 ·
jetj0 avatar image jetj0 Erick Ramirez ♦♦ commented ·

good afternoon! i have the similar problem,and i cannot post the question on this site, how can i communicate with you?

0 Likes 0 ·
Show more comments