DataStax Academy FAQ

DataStax Academy migrated to a new learning management system (LMS) in July 2020. We are also moving to a new Cassandra Certification process so there are changes to exam bookings, voucher system and issuing of certificates.

Check out the Academy FAQ pages for answers to your questions:


question

satvantsandhu2_189909 avatar image
satvantsandhu2_189909 asked ·

Is deleting SSTables then running a repair the correct action for dealing with compaction errors?

Hi..

I was seeing an error in my running cluster below -

Unable to seek to position 18502695515461889 in /var/lib/cassandra/data/keyspace1/standard1-79c18de09be711eab72de9991ca46fde/ba-749-bti-Rows.db (0 bytes)

So I shutdown my node and delete particular sstable and start the node after that I run the nodetool repair and issue got resolved.


So now I wanted to know is that process is right ? If yes , then can we do same process for sstable having size in GB ?

cassandra
1 comment
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@satvantsandhu2_189909, can you update your original question with the Apache Cassandra™ or DataStax Enterprise (DSE) version and also attach (not paste) the entire system.log capturing the time period when this error happened for additional context here? My initial feeling is you are hitting symptoms referred in CASSANDRA-4687, but it's too early to conclude without knowing additional details. Thanks!

0 Likes 0 · ·
Erick Ramirez avatar image
Erick Ramirez answered ·

Deleting SSTables should not be your default action when you run into issues. It is dangerous as it is prone to human error and can lead to data loss in some circumstances, particularly when repairs are not current or don't run regularly.

Instead, you should investigate and try to determine the cause. In these situations, the full error message, full stack trace as well as Cassandra version is really useful as they provide a clue for why the error was generated.

Depending on what the issue was, it is usually possible to recover the data by running tools such as sstablescrub. Or if it is related to a known issue, the SSTable might be recoverable by performing a binary update to a minor version of C*. A recent example of this is an issue where the serialisation header for UDTs were corrupted after upgrading from DSE 5.0 that prevented SSTables from being compacted. This was easily fixed by upgrading to a newer version of DSE (DB-2954, CASSANDRA-15035). Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

saravanan.chinnachamy_185977 avatar image
saravanan.chinnachamy_185977 answered ·

@satvantsandhu2_189909 If you update the question with more details like cassandra version, any error message from logs (system.log etc) it will help to provide a better resolution.

Ideally we will like to understand the source of the error and fix the cause.

Having said that, as for as repairs are concerned, generally run repairs in the following cases as detailed here When to run anti-entropy repair

  • Routinely to maintain node health.
  • When recovering a node after a failure while bringing it back into the cluster.
  • To update data on a node containing infrequently read data, and subsequently does not get read repair.
  • To update data on a downed node.
  • When recovering missing data or corrupted SSTables. You must run non-incremental repair.

Thera are several several types of repairs like Partitioner range(-pr), Local (-local) vs datacenter (-dc) vs cluster-wide repair, Endpoint range vs Subrange repair(-st, -et),Full repair vs incremental repair (-full vs -inc),Parallel vs Sequential repair (default, -seq, -dc-par).

Each scenario requires a particular type of repair. Please refer our documentation to understand the types of repairs and when to run them Manual repair: Anti-entropy repair


Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.