Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started



teriksson avatar image
teriksson asked Erick Ramirez commented

JMX reports lots of tombstones, but I can't see why

We are using prometheus exporter and now have a graph like this


The metrics in Grafana that I am using :

sum by (table) (increase(cassandra_table_tombstonescannedhistogram{ datacenter="$datacenter", keyspace="$keyspace", table != ""}[$__interval] ) )

There is ONE table in particular that stands out 'resource_relations'

It is very unfancy and looks like this:

CREATE TABLE resource_inventory.resource_relations (
    id text PRIMARY KEY,
    brandid text
    meta_last_updated_at timestamp,
    role text,
    sourceid text,
    sourcetype text,
    targetid text,
    targettype text,
    validfromdate timestamp,
    validtodate timestamp

So there are no complex types, no lists,sets or maps, that could infer a Tombstone

When I run sstabledump I come up with nothing e.g

    "partition" : {
      "key" : [ "149b8878-988b-d2de-0d9f-edd24d37dade" ],
      "position" : 1088684
    "rows" : [
    "partition" : {
      "key" : [ "38d1074b-238c-35ea-2a5e-c9bf9a9d289d" ],
      "position" : 0
    "rows" : [
        "type" : "row",
        "position" : 157,
        "liveness_info" : { "tstamp" : "2021-01-28T03:51:14.972Z" },
        "cells" : [
          { "name" : "meta_last_updated_at", "value" : "2021-01-15 04:30:00.000Z" },
          { "name" : "role", "value" : "preactivation" },
          { "name" : "sourceid", "value" : "555111222333" },
          { "name" : "sourcetype", "value" : "msisdns" },
          { "name" : "targetid", "value" : "999999000545288339" },
          { "name" : "targettype", "value" : "sim-cards" },
          { "name" : "validfromdate", "value" : "2021-01-17 04:00:00.000Z" },
          { "name" : "validtodate", "value" : "2021-03-18 04:00:00.000Z" }

Looking at the output from nodetool tablestats

bash-4.4$ /opt/cassandra/bin/nodetool --host tablestats resource_inventory.resource_relations
Total number of tables: 489
Keyspace : resource_inventory
Read Count: 1784479130
Read Latency: 0.03874267611692382 ms
Write Count: 38278342
Write Latency: 0.02417774894743351 ms
Pending Flushes: 0
Table: resource_relations
SSTable count: 1
Space used (live): 46695950
Space used (total): 46695950
Space used by snapshots (total): 336774624
Off heap memory used (total): 551576
SSTable Compression Ratio: 0.5290182800227687
Number of partitions (estimate): 334380
Memtable cell count: 48
Memtable data size: 8440
Memtable off heap memory used: 0
Memtable switch count: 600
Local read count: 331718
Local read latency: NaN ms
Local write count: 225656
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 417928
Bloom filter off heap memory used: 417920
Index summary off heap memory used: 126336
Compression metadata off heap memory used: 7320
Compacted partition minimum bytes: 150
Compacted partition maximum bytes: 215
Compacted partition mean bytes: 204
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
Dropped Mutations: 0

There are according to nodetool

Average tombstones per slice (last five minutes): 1.0

Maximum tombstones per slice (last five minutes): 1

How is this possible ?

Where does the jmx metrics come from ? why is it saying it has scanned millions of Tombstones, when I cannot find any trace of them.

1613722039833.png (39.6 KiB)
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

The metrics are coming directly from the metrics beans that Cassandra exposes via JMX. If you are using the DataStax Metrics Collector for Apache Cassandra (MCAC), the agent is simply collecting the data via JMX.

You can see it for yourself if you connect to the node via JMX (using jconsole for example) and check the value for that metric.

I'm not a Grafana expert so I'm not sure what your query is doing. Can you explain it to me in plain English? I'm wondering if it is somehow doing an accumulation of the single tombstone in the table. Cheers!

2 comments Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

The cassandra_table_tombstonescannedhistogram

my interpretation is that this is a counter that increases over time

meaning that at 10:00 it was 500, and at 10:05 it was 600

running increase( cassandra_table_tombstonescannedhistogram{}[5m] ) would then give me the rate per second for the over these 5 minutes, i.e. 100 / 5 / 60 = 0.33

The sum by (table) is simply aggregate statistics from all Cassandra instances per table

0 Likes 0 ·

Did you connect via JMX to see the raw value?

0 Likes 0 ·