question

adityajain22_141041 avatar image
adityajain22_141041 asked adityajain22_141041 commented

Getting huge hints count after adding a DC to a cluster

Version : apache cassandra 3.11.3

Setup : 2 DC, 5 nodes each

After adding DC2 to the cluster, start getting huge hints count in the hints folder, and hints are being saved for across DC.

>all the nodes are UP

>No network drop amd ping latency is 15 ms across DC

>TPS: 800tps

Sample log for multiple nodes

DC 2


INFO [HintsDispatcher:134] 2020-12-07 16:22:37,249 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 74c05489-4374-4013-8ece-fd408b672d3d-1607227934695-1.hints to endpoint /****DC-1 ip3****: 74c05489-4374-4013-8ece-fd408b672d3d, partially
INFO [HintsDispatcher:126] 2020-12-07 16:22:45,066 HintsStore.java:133 - Deleted hint file 5054087d-32f8-4d96-af96-fbc93e2b6cf9-1607338346642-1.hints
INFO [HintsDispatcher:126] 2020-12-07 16:22:45,067 HintsDispatchExecutor.java:282 - Finished hinted handoff of file 5054087d-32f8-4d96-af96-fbc93e2b6cf9-1607338346642-1.hints to endpoint /****DC-1 ip1****: 5054087d-32f8-4d96-af96-fbc93e2b6cf9
INFO [HintsDispatcher:135] 2020-12-07 16:22:45,090 HintsStore.java:133 - Deleted hint file 6b3f348d-62fd-42dd-9f93-5ce0f1cc6b0b-1607338346641-1.hints
INFO [HintsDispatcher:135] 2020-12-07 16:22:45,090 HintsDispatchExecutor.java:282 - Finished hinted handoff of file 6b3f348d-62fd-42dd-9f93-5ce0f1cc6b0b-1607338346641-1.hints to endpoint /****DC-1 ip4****: 6b3f348d-62fd-42dd-9f93-5ce0f1cc6b0b
INFO [HintsDispatcher:128] 2020-12-07 16:22:45,091 HintsStore.java:133 - Deleted hint file e5715590-f083-4dab-9806-c89b31f4bacf-1607338346642-1.hints
INFO [HintsDispatcher:128] 2020-12-07 16:22:45,092 HintsDispatchExecutor.java:282 - Finished hinted handoff of file e5715590-f083-4dab-9806-c89b31f4bacf-1607338346642-1.hints to endpoint /****DC-1 ip2****: e5715590-f083-4dab-9806-c89b31f4bacf

DC1


INFO [HintsDispatcher:15471] 2020-12-07 15:49:06,862 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a-1607310632977-1.hints to endpoint /****DC-2 ip5****: 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a, partially
INFO [HintsDispatcher:15472] 2020-12-07 15:49:16,863 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a-1607310632977-1.hints to endpoint /****DC-1 ip2****: 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a, partially
INFO [HintsDispatcher:15473] 2020-12-07 15:49:26,862 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a-1607310632977-1.hints to endpoint /****DC-1 ip2****: 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a, partially
INFO [HintsDispatcher:15474] 2020-12-07 15:49:36,863 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a-1607310632977-1.hints to endpoint /****DC-1 ip2****: 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a, partially
INFO [HintsDispatcher:15475] 2020-12-07 15:49:46,863 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a-1607310632977-1.hints to endpoint /****DC-1 ip2****:


Need to know, why hints are being applied partially as seen in logs and why the hints physical files are not being deleted. I can see 3 days old hints files in system.

hinted handoff
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered adityajain22_141041 commented

The log entries you posted are the symptoms after the event, not during the event.

Hints are stored by request coordinators when replicas do not acknowledge write requests, usually because the problematic replica is unresponsive or unreachable from the coordinator. This is when some event takes place including (but not limited to) network interruptions, nodes being overloaded, etc.

Hints are handed off to the respective replicas when they come back online -- meaning after the event. When you see hinted handoffs in the logs, it indicates that there was previously a problem with nodes earlier but they are back online so coordinators are sending the mutations to the replicas.

If you're checking the status of the cluster after you see hinted handoffs in the logs, it is expected that the nodes are operational because hints can't be delivered unless the problematic event has passed. Cheers!

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

adityajain22_141041 avatar image adityajain22_141041 commented ·

Thanks @Erick Ramirez


Need to know, why hints are being applied partially as seen in logs and why the hints physical files are not being deleted. I can see 3 days old hints files in system.

0 Likes 0 ·