question

igor.rmarinho_185445 avatar image
igor.rmarinho_185445 asked yashwanth.kondeti@verizon.com commented

Why are rows still missing from reads after I've restored from snapshots?

Hi,

I'm having issues when I'm restoring the snapshot from a tables,

I create a table tab1 with 5 rows on my node 3
Took the snapshot
Truncate the table tab1
Stopped cassandra
Deleted all commitlogs
Deleted the file inside the /data/keyspace/table_name/
Copied the file from the snapshot to  /data/keyspace/table_name/
Started cassandra
Ran nodetool repair keyspace_name
[cqlsh 6.8.0 | DSE 6.8.0 | CQL spec 3.4.5 | DSE protocol v2]
Use HELP for help.
cqlsh> CONSISTENCY;
Current consistency level is ONE.
cqlsh>

Script

nodetool clearsnapshot --all

cqlsh 10.*.*.3  -u cassandra -p cassandra -e 'DESC SCHEMA;' > /backups/snapshots/schema_filename.cql
nodetool ring > /backups/snapshots/token_range_filename
nodetool snapshot -t snapshot_backup_n3_t
find /cassandra/ -path */snapshots/snapshot_backup_n3_t/* -type f >> "/backups/snapshots/backup_file_listing"

/usr/bin/nice -10 /bin/tar -pzcvf /backups/snapshots/backup_snapshot_node3_$(date +%F).tar --files-from=/backups/snapshots/backup_file_listing

tbs1, before I take the snapshot.

(Some rows were deleted and inserted before the snapshot was taken and the keyspace was repaired, to simulate an usual prod situation)

 name  | solr_query
-------+------------
 test7 |       null
 test3 |       null
 test1 |       null
 test9 |       null
 test5 |       null

after restore the snapshot

 name  | solr_query
-------+------------
 test7 |       null
 test3 |       null
 test1 |       null

Sometimes I tried the same process I had all the 5 rows other only 3 rows. What could be the issue?

Thanks

cassandrarestore
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

smadhavan avatar image smadhavan ♦ commented ·

Similar to the earlier question, https://community.datastax.com/questions/3206/incremental-backup-questions.html

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered igor.rmarinho_185445 commented

@igor.rmarinho_185445 Restoring from snapshots is a little more nuanced than its equivalent operation in a relational database.

When data is written, the mutation includes a timestamp of when it was written. Note that a mutation is any INSERT, UPDATE or DELETE. The write time (metadata) is stored in the SSTable together with the data itself.

Let me illustrate with a scenario. Consider this:

  • partition X was inserted into the database at 16:25
  • at some point, partition X was flushed with other partitions to SSTable 456
  • a snapshot was taken at 16:45
  • partition X was deleted at 17:23
  • at 18:56, SSTable 456 was restored from snapshot taken at 16:45

User tries to read partition X:

SELECT ... FROM sometable WHERE pk = X

but no rows are returned. This is expected because SSTable 456 only contained the INSERT for partition X at 16:25. However, the DELETE at 17:23 is still valid. As far as C* is concerned, the partition was deleted so it won't get returned in the results.

You can check for this if you enable tracing in cqlsh where you'll see that rows/cells get skipped because of the existence of tombstones. Cheers!

3 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

igor.rmarinho_185445 avatar image igor.rmarinho_185445 commented ·

Hi Erick.

So the only way to skip this issue would be set my gc_grace to 1 hour, if I take a snapshot every hour? Or even set the gc_grace to zero in the table before start the recovering and after change It back to the original value?

Thanks

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ igor.rmarinho_185445 commented ·

@igor.rmarinho_185445 It depends on what you want to achieve. If you're looking to start over, you should TRUNCATE the table before you perform a restore. Also, make sure that you restore data on all nodes in the cluster (if you weren't already). Cheers!

0 Likes 0 ·
yashwanth.kondeti@verizon.com avatar image yashwanth.kondeti@verizon.com commented ·

@Erick Ramirez In the situation that you mentioned, it looks like Cassandra will not restore the particular set of data because the delete operation has the latest timestamp. Then how do we we fix an accidental deletion in cassandra?

For example, I took a snapshot at 2pm and I accidentally deleted 100 rows at 3pm. I want to restore the data back to how it was at 2pm using the snapshot from 2pm. When I restore the snapshot, would I get the 100 rows data back or would the delete timestamp overrule the restore for those 100 records?

What other steps do I need to do to restore those 100 records?


All always, really appreciate your inputs!!!


Thanks,

Yashwanth.


0 Likes 0 ·