Background: Cassandra is being used to save millions of claims records in order to generate aggregated data using the claims available in Cassandra DB 3.11.5. Only the data for last 90 days is considered while generating the aggregated data and hence the older data needs to be deleted to free up the DB space. The aggregated is generated for every 3 hours in a day using Batch runs that are configured in the system.
Approach: App team has found a path to leverage TimeToLive functionality of Cassandra which indicates a value for the record to be valid i.e. TimeToLive is calculated for each of the record while saving into Database. But right after this change the read operations as part of the batch have been failing with ReadTimeOut Exception. The default value of gc_grace_seconds vale was 10 days on the table and we tried by changing it to 24 hours. Even after the change the application is still experiencing ReadTimeOut Exception. The next step is to bring down the value of gc_grace_seconds to further down to see if that helps.
Sample Error Message:
ERROR [ReadStage-1] 2020-08-19 19:41:47,169 StorageProxy.java:1896 - Scanned over 100001 tombstones during query 'SELECT * FROM data.claimcassandraa_5abf9e5d784895ec2acd499f0c766b38 WHERE token(claimcoreintakedatestr) > -4887371003292131092 AND token(claimcoreintakedatestr) <= -4842270319348757322 LIMIT 1000' (last scanned row partion key was ((20200520), 1, 20E528509300, CSP, UHG-Quest-AVRO-Claim, 2020-05-20 00:00-0500, top_)); query aborted
Question: When the TimeToLive value on a record expires in Cassandra, it is being converted as Tombstone and hence these records are not considered for any read operation.
o How does Cassandra takes care of deleting Tombstone records ?
o Is there any compaction process that gets triggered periodically to delete these records or an explicit Node Tool repair command needs to be run to take care of this issue.
Please let me know if you need any further details.
