Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

kelly.summerlin_151704 avatar image
kelly.summerlin_151704 asked ·

What to do when cloning a cluster with OpsCenter results in a timeout when loading snapshots on S3?

I am attempting to following the documentation from this page:

Cloning cluster data from a defined other location

But when I attempt to Restore from Other Location and specify the S3 bucket where the backups are stored, I get an error:

Error loading snapshots: The request to /DC1/backups timed out after 60 seconds.


The original DC1 cluster has 4 nodes and the new DC1 cluster has 4 nodes. The datacenter name on both is the same as well, DC1.


What else should I be looking for to get this to work?

restores3opscenter-backup
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

kelly.summerlin_151704 avatar image
kelly.summerlin_151704 answered ·

What finally worked for me was to move or copy the S3 bucket contents so that the <s3_bucket>/snapshots folder was laid out with folder names that matched the Host IDs of the nodes in my new cluster. I guess that is how OpsCenter backup works is that it copies the data into a snapshots folder with subfolders for each of the Host IDs.

Once I copied the old HostID folder names to match the new HostIDs in my new cluster, I could use the Sync Data feature under the Actions button on the Backup Service view. Then I was able to use the Restore Backup feature and select one of the previous backups. Restore complained about not being able to verify the new topology with the old topology, so I had to use the SSTableLoader restore option (much slower).

Honestly this doesn't seem to be any better than the old option of just copying the /cassandra/data/* folders from one machine to another.

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@kelly.summerlin_151704 the steps you described indicate that you were "restoring" snapshots instead of using OpsCenter cloning feature. You don't need to manually mess around with the snapshot folders in S3 to clone to a new cluster. Cheers!

0 Likes 0 · ·
Erick Ramirez avatar image
Erick Ramirez answered ·

@kelly.summerlin_151704 the error you are seeing occurs when OpsCenter tries to scan the S3 bucket for a list of stored snapshots but the S3 lookup times out.

You will need to check the OpsCenter log for clues on why OpsCenter is having issues connecting to S3 and why S3 is unresponsive. Troubleshooting this issue is very involved so if your company has a current DSE subscription, please log a Support ticket so one of our engineers can assist you. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.