Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

Mike_201 avatar image
Mike_201 asked ·

OpsCenter 6.1.6 backup to S3 on new DSE 4.8.16 nodes fails with "Sync Snapshot Failed"

I doubled the size of my cluster adding new nodes on ubuntu 18.04. Installed DSE 4.8.16 from the downloadable binary. Installed java version "1.8.0_211" on the new nodes. Updated the datastax-agents from OpCenter and expected my scheduled backup to s3 to "just work" but it doesn't. Fails on all 6 of the new nodes as shown in the screenshot. I don't exactly know where to begin, whether this is an opscenter config problem or something else. Any help welcome!

opscenterbackups3
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

@Mike_201 It isn't possible to diagnose the issue from the limited information you've provided. You'll need to review the OpsCenter logs to get the exact error as it will most likely give you a clue on the root cause of the failure.

Given that the failure is localised to the 6 new nodes in the cluster, it is most likely environmental and would be something misconfigured on the servers themselves. Have a look at the permissions on the backup staging directory on the nodes. Hint: You've configured this in the backup schedule in the OpsCenter Backup Service UI.

You will also need to review the agent logs for each of the nodes to work out why the snapshot sync to S3 failed. I imagine you will have difficulty with this because the backup is very old (Feb 24) and you probably no longer have the logs.

In any case, it will be difficult to troubleshoot this with you in the Q&A format of the Community so I would recommend that you log a ticket on the DataStax Support portal and attach a Diagnostics Report so one of our engineers can analyse it for you. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.