question

gpatcham_37382 avatar image
gpatcham_37382 asked Erick Ramirez commented

What is best way to use Timewindow compaction strategy for old data

We have a brand new table with TWCS strategy enabled. with 7 days bucket window and there is need to backfill some old data. So, when we backfill old data with "USING TIMESTAMP" will data be compacted into appropriate buckets ? Or what's the best practice ?. Also during testing I'm not seeing data not getting compacted into 1 sstable for a given window bucket. Here are the options used


compaction = {'class': 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 'compaction_window_size': '7', 'compaction_window_unit': 'DAYS', 'max_threshold': '4', 'min_threshold': '4', 'unchecked_tombstone_compaction': 'true'}

I'm checking count of sstable usgin nodetool tablestats.


As part of my testing I used below options

compaction = {'class': 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 'compaction_window_size': '1', 'compaction_window_unit': 'HOURS', 'max_threshold': '4', 'min_threshold': '4', 'unchecked_tombstone_compaction': 'true'}


compactiontimewindowcompaction strategy
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

@gpatcham_37382 there is no way of loading "old" data. TWCS works on the basis of when data was inserted. Whatever was loaded in the last 7 days would go into 1 bucket regardless of whether some of the data is 14 days or 1 month old. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

gpatcham_37382 avatar image gpatcham_37382 commented ·

@Erick Ramirez Thanks for your response.

If the bucket data is not getting compacted to 1 SStable , what options should I look into?

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ gpatcham_37382 commented ·

@gpatcham_37382 it would be great if you could ask a new question so we can keep this post just about loading old data and TWCS. Cheers!

0 Likes 0 ·