Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

Gangadhara M.B avatar image
Gangadhara M.B asked ·

Why are hints files not purged?

We have multi DC Cassandra cluster , with hints_directory: /var/lib/cassandra/hints , the hints are not getting purged/deleted due to which mount point is consuming more space than real data , should hints be deleted/managed by Cassandra itself than manual delete ?

[root@prd-cassandra-data-150 cassandra]# du -sh *
28G commitlog
563G data
661G hints
16K lost+found
104M saved_caches
[root@prd-cassandra-data-15 cassandra]# grep -i /var/lib/cassandr* /etc/dse/cassandra/cassandra.yaml
/etc/dse/cassandra/cassandra.yaml:hints_directory: /var/lib/cassandra/hints
/etc/dse/cassandra/cassandra.yaml: - /var/lib/cassandra/data
/etc/dse/cassandra/cassandra.yaml: - /var/lib/cassandra1/data
/etc/dse/cassandra/cassandra.yaml:commitlog_directory: /var/lib/cassandra/commitlog
/etc/dse/cassandra/cassandra.yaml:saved_caches_directory: /var/lib/cassandra/saved_caches
[root@prd-cassandra-data-150 cassandra]# nodetool info
ID : 7d23ec79-61e6-47b5-b4e1-64d85125bb34
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 846.47 GiB
Generation No : 1607972475
Uptime (seconds) : 5684034
Heap Memory (MB) : 26377.55 / 31744.00
Off Heap Memory (MB) : 4696.87
Data Center : AWSRUSW2MAC
Rack : usw2b
Exceptions : 14
hinted handoff
2 comments
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

DSE Cassandra 5.1.11 running on AWS EC2 nodes .

If we look at attached file , hints files are getting deleted but looks not all hints files deleted/purged . attached file also has "nodetool tpstats" outputhints_delete.txt

0 Likes 0 ·
hints-delete.txt (54.5 KiB)

I do have same an issue when removing the hints, site. anyone can help to do this

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

Hints files get deleted when the hints they contain are handed off to the replicas they belong to. But if the target replica is down or unresponsive, the coordinator node cannot deliver the hints (missed mutations) so it can't purge the hints files.

Hints files building up is just a symptom. The problem you need to focus on is your nodes going down or becoming unresponsive. The most common reason for that is your cluster being overloaded.

If you need assistance investigating the issue, please log a ticket with DataStax Support so one of our engineers can assist you directly. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.