vkrot avatar image
vkrot asked Erick Ramirez answered

Getting "Detected unreadable sstables" when restoring OpsCenter backup on S3 to local machine

Hi all,

I'm new to cassandra and DSE and have problems restoring daily backup from DSE to local machine.

There is a DSE cluster managed by OpsCenter, version 6.8.13. OpsCenter reports version 6.8.13 and cqlsh, when connected, reports version 6.8.13

I want to restore entire or partial CF from daily backup to local machine, but without any luck. DSE is configured to take daily backups to s3. What I did:

  • download and run exactly the same dse 6.8.13 locally
  • locally create a CF with exactly the same structure (DDL taken from OpsCenter)
  • find backups.json file in s3 - I take it from one of the nodes. Afterwards I find all rows related to my CF and download corresponding files from s3://buckets/{node-id}/sstables/{filename} to local dir

Now I tried a couple of ways to import those files locally:

  1. copy downloaded files to /var/lib/cassandra/data/database/table/ and run nodetool refresh - no luck
  2. copy copy downloaded files to /var/lib/cassandra/data/imported-data and run nodetool import database tablename
  3. use sstableloader

the result is always the same - empty output from the tools and nothing interesting in logs. After I restart cassandra - it doesn't start with error "Detected unreadable sstables ... imported files"

What am I doing wrong? How can I import sstables from a backup?

[UPDATE] example files I copied:

{'format': 'bti-', 'generation': '20519', 'name': '94a95de27224e63d289662bc34e2c39bc64181407d67f17f7b942370a60fd86b-bb-20519-bti-CompressionInfo.db', 'type': 'CompressionInfo', 'keyspace': 'abc', 'size': 27567, 'uniquifier': '94a95de27224e63d289662bc34e2c39bc64181407d67f17f7b942370a60fd86b', 'compressed': None, 'cf': 'kpi_monthly', 'version': 'bb'}
{'format': 'bti-', 'generation': '20519', 'name': '94a95de27224e63d289662bc34e2c39bc64181407d67f17f7b942370a60fd86b-bb-20519-bti-Filter.db', 'type': 'Filter', 'keyspace': 'abc', 'size': 83120, 'uniquifier': '94a95de27224e63d289662bc34e2c39bc64181407d67f17f7b942370a60fd86b', 'compressed': None, 'cf': 'kpi_monthly', 'version': 'bb'}
{'format': 'bti-', 'generation': '20519', 'name': '94a95de27224e63d289662bc34e2c39bc64181407d67f17f7b942370a60fd86b-bb-20519-bti-Data.db', 'type': 'Data', 'keyspace': 'abc', 'size': 86855574, 'uniquifier': '94a95de27224e63d289662bc34e2c39bc64181407d67f17f7b942370a60fd86b', 'compressed': None, 'cf': 'kpi_monthly', 'version': 'bb'}
{'format': 'bti-', 'generation': '20519', 'name': '94a95de27224e63d289662bc34e2c39bc64181407d67f17f7b942370a60fd86b-bb-20519-bti-Partitions.db', 'type': 'Partitions', 'keyspace': 'abc', 'size': 1122076, 'uniquifier': '94a95de27224e63d289662bc34e2c39bc64181407d67f17f7b942370a60fd86b', 'compressed': None, 'cf': 'kpi_monthly', 'version': 'bb'}
{'format': 'bti-', 'generation': '20519', 'name': '94a95de27224e63d289662bc34e2c39bc64181407d67f17f7b942370a60fd86b-bb-20519-bti-Statistics.db', 'type': 'Statistics', 'keyspace': 'abc', 'size': 11710, 'uniquifier': '94a95de27224e63d289662bc34e2c39bc64181407d67f17f7b942370a60fd86b', 'compressed': None, 'cf': 'kpi_monthly', 'version': 'bb'}
1 comment
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

vkrot avatar image vkrot commented ·

[Update posted in original question]

[Post converted to comment since it's not an "answer"]

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

The error you described indicates that the node contains invalid files that is causing weird issues on your machine.

The likely issue is that you haven't got a clean installation and your machine contains data files from an older version of Cassandra and/or DSE.

You will need to review the contents of the data directories on the machine and correlate them with the log entries on startup for clues. If you need assistance doing this, please log a ticket with DataStax Support so one of our engineers can assist you. Cheers!

10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.