I doubled the size of my cluster adding new nodes on ubuntu 18.04. Installed DSE 4.8.16 from the downloadable binary. Installed java version "1.8.0_211" on the new nodes. Updated the datastax-agents from OpCenter and expected my scheduled backup to s3 to "just work" but it doesn't. Fails on all 6 of the new nodes as shown in the screenshot. I don't exactly know where to begin, whether this is an opscenter config problem or something else. Any help welcome!
@Mike_201 It isn't possible to diagnose the issue from the limited information you've provided. You'll need to review the OpsCenter logs to get the exact error as it will most likely give you a clue on the root cause of the failure.
Given that the failure is localised to the 6 new nodes in the cluster, it is most likely environmental and would be something misconfigured on the servers themselves. Have a look at the permissions on the backup staging directory on the nodes. Hint: You've configured this in the backup schedule in the OpsCenter Backup Service UI.
You will also need to review the agent logs for each of the nodes to work out why the snapshot sync to S3 failed. I imagine you will have difficulty with this because the backup is very old (Feb 24) and you probably no longer have the logs.
In any case, it will be difficult to troubleshoot this with you in the Q&A format of the Community so I would recommend that you log a ticket on the DataStax Support portal and attach a Diagnostics Report so one of our engineers can analyse it for you. Cheers!
5 People are following this question.
