question

whpw avatar image
whpw asked whpw commented

nodetool garbagecollect does not delete tombstones after TTL has expired

I tried to delete tombstones on a table, but it is not possible.

I did following:

/appl/cassandra/bin/nodetool -u "user" -pw 'pass' -h $(hostname -i) garbagecollect -j 4 keyspaceA tableA

and also with -g CELL

/appl/cassandra/bin/nodetool -u "user" -pw 'pass' -h $(hostname -i) garbagecollect -g CELL -j 4 keyspaceA tableA

but a lot of tombstones still exists.

e.g.

"rows" : [
{
"type" : "row",
"position" : 30,
"clustering" : [ "2019-09-21 02:00+0200", "042c650a-6b42-42c8-813d-e2c91a4831c9" ],
"liveness_info" : { "tstamp" : "2019-09-22T00:38:19.287798Z", "ttl" : 34186698, "expires_at" : "2020-10-21T16:56:37Z", "expired" : true },
"cells" : [

I can not find a reason for not deleting expired cells.

gc_grace_time ist 10days (default)

Thanks

tombstonesnodetool
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered whpw commented

There are some issues with the garbagecollect command. Range tombstones and row tombstones won't necessarily get purged.

There are plans to improve the reliability of the command in future versions. Cheers!

3 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

whpw avatar image whpw commented ·

thanks for the quick answer. Are there any other options to purge those tombstones? compaction, repairs....because I see lots of log entries like this

Read 4778 live rows and 19757 tombstone cells for query.....

I believe the tombstones are responsible for timeouts and high read latency for a special table.

[Post converted to comment since it is not an "answer"]

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ whpw commented ·

You can temporarily workaround the issue by forcing a major compaction. But I can't stress enough that this is a band-aid solution and you need to fix the underlying data model problem.

Have a look at my post Why forcing a major compaction on a table is not ideal. Cheers!

0 Likes 0 ·
whpw avatar image whpw Erick Ramirez ♦♦ commented ·

I did a compaction --split-output on the affected table - no chance - the tombstones still exist.

Backup and Restore - maybe a possible solution ? It's a small table


0 Likes 0 ·