question

xiaowan2_157616 avatar image
xiaowan2_157616 asked Erick Ramirez edited

Issue with truncating hints

Hi DataStax export

one DC has big sized hints(467G) and will use up the whole directory sooner or later and we need to delete it. I ran nodetool truncatehints but it's not working. I see from nodetool tpstats there is 2 active hintedhandoff and 40 pending as follows:

HintedHandoff 2 40 0 0 0

it seems like there is an issue with hintedhandoff which is hung and caused nodetool truncatehints not working.

please advise!

Thanks

hinted handoff
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

@xiaowan2_157616 The pending HintedHandoff tasks are a symptom of a bigger problem -- there are nodes which are either down or unreachable. This is the reason for lots of hints being stored by the coordinator nodes. You need to investigate why other nodes are unreachable and resolve it. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

xiaowan2_157616 avatar image xiaowan2_157616 commented ·

it looks like currently we don't have any down nodes, but the hints is still existed. If they get online, hints supposed to be applied back to nodes but I still didn't see it. we delete hints to claim space & restart. so I know we need repair.But regular repair is too much time costing, please advise which. since we couldn't detect which tables so we just use -pr w/o KS and CF

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ xiaowan2_157616 commented ·

You'll need to run nodetool repair -pr one node at a time to recover the missing mutations from other replicas. It's important that you run it on ALL nodes in ALL DCs. Cheers!

0 Likes 0 ·