I am developing a background process that deletes rows from a Cassandra 3.11.2 table with a given partition key, then immediately adds rows to the same table using the same partition key -- in effect, a refresh process for portions of the table. My current environment has only a single node, so the complexities of propagating deletes and tombstones across a multi-node cluster are not yet present. The gc_grace_seconds for the table is the default value 864000, and no time-to-live limits have been set.
Immediately after running the process it works as expected, with the correct data refreshed. However, after restarting the Cassandra node a few minutes later, queries against the table return no data at all. It seems that the contents of the table added by the refresh have vanished. I thought at first there was simply a problem in my delete logic, but with further testing have ruled this out. I am using the Java driver version 4.12 to process the deletes.
In particular, I tried adding the WITH TIMESTAMP phrase to the prepared DELETE statement to ensure that only rows with timestamps prior to the current time were deleted, but this change had no effect.
I now think this behavior is a result of Cassandra tombstones and/or compaction processing, but still don't understand why it should happen. (I have a general understanding of tombstone processing, but little hands-on experience with them in live environments.) When I look at the contents of Cassandra's data directory in the file system, I can see previous Cassandra sstables still on the disk, but a newer empty subdirectory has been added that seems to be superceding it. I have been searching the web and docs, but haven't found any discussion of this scenario.
Can anyone explain why this is happening and how to prevent it?