We are using prometheus exporter and now have a graph like this
The metrics in Grafana that I am using :
sum by (table) (increase(cassandra_table_tombstonescannedhistogram{ datacenter="$datacenter", keyspace="$keyspace", table != ""}[$__interval] ) )
There is ONE table in particular that stands out 'resource_relations'
It is very unfancy and looks like this:
CREATE TABLE resource_inventory.resource_relations (
id text PRIMARY KEY,
brandid text
meta_last_updated_at timestamp,
role text,
sourceid text,
sourcetype text,
targetid text,
targettype text,
validfromdate timestamp,
validtodate timestamp
)
So there are no complex types, no lists,sets or maps, that could infer a Tombstone
When I run sstabledump I come up with nothing e.g
{
"partition" : {
"key" : [ "149b8878-988b-d2de-0d9f-edd24d37dade" ],
"position" : 1088684
},
"rows" : [
[
{
"partition" : {
"key" : [ "38d1074b-238c-35ea-2a5e-c9bf9a9d289d" ],
"position" : 0
},
"rows" : [
{
"type" : "row",
"position" : 157,
"liveness_info" : { "tstamp" : "2021-01-28T03:51:14.972Z" },
"cells" : [
{ "name" : "meta_last_updated_at", "value" : "2021-01-15 04:30:00.000Z" },
{ "name" : "role", "value" : "preactivation" },
{ "name" : "sourceid", "value" : "555111222333" },
{ "name" : "sourcetype", "value" : "msisdns" },
{ "name" : "targetid", "value" : "999999000545288339" },
{ "name" : "targettype", "value" : "sim-cards" },
{ "name" : "validfromdate", "value" : "2021-01-17 04:00:00.000Z" },
{ "name" : "validtodate", "value" : "2021-03-18 04:00:00.000Z" }
]
}
]
}
Looking at the output from nodetool tablestats
bash-4.4$ /opt/cassandra/bin/nodetool --host 123.123.123.123 tablestats resource_inventory.resource_relations Total number of tables: 489 ---------------- Keyspace : resource_inventory Read Count: 1784479130 Read Latency: 0.03874267611692382 ms Write Count: 38278342 Write Latency: 0.02417774894743351 ms Pending Flushes: 0 Table: resource_relations SSTable count: 1 Space used (live): 46695950 Space used (total): 46695950 Space used by snapshots (total): 336774624 Off heap memory used (total): 551576 SSTable Compression Ratio: 0.5290182800227687 Number of partitions (estimate): 334380 Memtable cell count: 48 Memtable data size: 8440 Memtable off heap memory used: 0 Memtable switch count: 600 Local read count: 331718 Local read latency: NaN ms Local write count: 225656 Local write latency: NaN ms Pending flushes: 0 Percent repaired: 0.0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 417928 Bloom filter off heap memory used: 417920 Index summary off heap memory used: 126336 Compression metadata off heap memory used: 7320 Compacted partition minimum bytes: 150 Compacted partition maximum bytes: 215 Compacted partition mean bytes: 204 Average live cells per slice (last five minutes): 1.0 Maximum live cells per slice (last five minutes): 1 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 Dropped Mutations: 0
There are according to nodetool
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
How is this possible ?
Where does the jmx metrics come from ? why is it saying it has scanned millions of Tombstones, when I cannot find any trace of them.
