Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started



satvantsandhu2_189909 avatar image
satvantsandhu2_189909 asked Erick Ramirez answered

Is deleting SSTables then running a repair the correct action for dealing with compaction errors?


I was seeing an error in my running cluster below -

Unable to seek to position 18502695515461889 in /var/lib/cassandra/data/keyspace1/standard1-79c18de09be711eab72de9991ca46fde/ba-749-bti-Rows.db (0 bytes)

So I shutdown my node and delete particular sstable and start the node after that I run the nodetool repair and issue got resolved.

So now I wanted to know is that process is right ? If yes , then can we do same process for sstable having size in GB ?

1 comment
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@satvantsandhu2_189909, can you update your original question with the Apache Cassandra™ or DataStax Enterprise (DSE) version and also attach (not paste) the entire system.log capturing the time period when this error happened for additional context here? My initial feeling is you are hitting symptoms referred in CASSANDRA-4687, but it's too early to conclude without knowing additional details. Thanks!

0 Likes 0 ·
saravanan.chinnachamy_185977 avatar image
saravanan.chinnachamy_185977 answered

@satvantsandhu2_189909 If you update the question with more details like cassandra version, any error message from logs (system.log etc) it will help to provide a better resolution.

Ideally we will like to understand the source of the error and fix the cause.

Having said that, as for as repairs are concerned, generally run repairs in the following cases as detailed here When to run anti-entropy repair

  • Routinely to maintain node health.
  • When recovering a node after a failure while bringing it back into the cluster.
  • To update data on a node containing infrequently read data, and subsequently does not get read repair.
  • To update data on a downed node.
  • When recovering missing data or corrupted SSTables. You must run non-incremental repair.

Thera are several several types of repairs like Partitioner range(-pr), Local (-local) vs datacenter (-dc) vs cluster-wide repair, Endpoint range vs Subrange repair(-st, -et),Full repair vs incremental repair (-full vs -inc),Parallel vs Sequential repair (default, -seq, -dc-par).

Each scenario requires a particular type of repair. Please refer our documentation to understand the types of repairs and when to run them Manual repair: Anti-entropy repair

10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image
Erick Ramirez answered

Deleting SSTables should not be your default action when you run into issues. It is dangerous as it is prone to human error and can lead to data loss in some circumstances, particularly when repairs are not current or don't run regularly.

Instead, you should investigate and try to determine the cause. In these situations, the full error message, full stack trace as well as Cassandra version is really useful as they provide a clue for why the error was generated.

Depending on what the issue was, it is usually possible to recover the data by running tools such as sstablescrub. Or if it is related to a known issue, the SSTable might be recoverable by performing a binary update to a minor version of C*. A recent example of this is an issue where the serialisation header for UDTs were corrupted after upgrading from DSE 5.0 that prevented SSTables from being compacted. This was easily fixed by upgrading to a newer version of DSE (DB-2954, CASSANDRA-15035). Cheers!

10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.