Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started



igor.rmarinho_185445 avatar image
igor.rmarinho_185445 asked ·

Why are rows still missing from reads after I've restored from snapshots?


I'm having issues when I'm restoring the snapshot from a tables,

I create a table tab1 with 5 rows on my node 3
Took the snapshot
Truncate the table tab1
Stopped cassandra
Deleted all commitlogs
Deleted the file inside the /data/keyspace/table_name/
Copied the file from the snapshot to  /data/keyspace/table_name/
Started cassandra
Ran nodetool repair keyspace_name
[cqlsh 6.8.0 | DSE 6.8.0 | CQL spec 3.4.5 | DSE protocol v2]
Use HELP for help.
Current consistency level is ONE.


nodetool clearsnapshot --all

cqlsh 10.*.*.3  -u cassandra -p cassandra -e 'DESC SCHEMA;' > /backups/snapshots/schema_filename.cql
nodetool ring > /backups/snapshots/token_range_filename
nodetool snapshot -t snapshot_backup_n3_t
find /cassandra/ -path */snapshots/snapshot_backup_n3_t/* -type f >> "/backups/snapshots/backup_file_listing"

/usr/bin/nice -10 /bin/tar -pzcvf /backups/snapshots/backup_snapshot_node3_$(date +%F).tar --files-from=/backups/snapshots/backup_file_listing 

tbs1, before I take the snapshot.

(Some rows were deleted and inserted before the snapshot was taken and the keyspace was repaired, to simulate an usual prod situation)

 name  | solr_query
 test7 |       null
 test3 |       null
 test1 |       null
 test9 |       null
 test5 |       null

after restore the snapshot

 name  | solr_query
 test7 |       null
 test3 |       null
 test1 |       null

Sometimes I tried the same process I had all the 5 rows other only 3 rows. What could be the issue?


1 comment
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

@igor.rmarinho_185445 Restoring from snapshots is a little more nuanced than its equivalent operation in a relational database.

When data is written, the mutation includes a timestamp of when it was written. Note that a mutation is any INSERT, UPDATE or DELETE. The write time (metadata) is stored in the SSTable together with the data itself.

Let me illustrate with a scenario. Consider this:

  • partition X was inserted into the database at 16:25
  • at some point, partition X was flushed with other partitions to SSTable 456
  • a snapshot was taken at 16:45
  • partition X was deleted at 17:23
  • at 18:56, SSTable 456 was restored from snapshot taken at 16:45

User tries to read partition X:

SELECT ... FROM sometable WHERE pk = X

but no rows are returned. This is expected because SSTable 456 only contained the INSERT for partition X at 16:25. However, the DELETE at 17:23 is still valid. As far as C* is concerned, the partition was deleted so it won't get returned in the results.

You can check for this if you enable tracing in cqlsh where you'll see that rows/cells get skipped because of the existence of tombstones. Cheers!

2 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Hi Erick.

So the only way to skip this issue would be set my gc_grace to 1 hour, if I take a snapshot every hour? Or even set the gc_grace to zero in the table before start the recovering and after change It back to the original value?


0 Likes 0 ·

@igor.rmarinho_185445 It depends on what you want to achieve. If you're looking to start over, you should TRUNCATE the table before you perform a restore. Also, make sure that you restore data on all nodes in the cluster (if you weren't already). Cheers!

0 Likes 0 ·