Version : apache cassandra 3.11.3
Setup : 2 DC, 5 nodes each
After adding DC2 to the cluster, start getting huge hints count in the hints folder, and hints are being saved for across DC.
>all the nodes are UP
>No network drop amd ping latency is 15 ms across DC
>TPS: 800tps
Sample log for multiple nodes
DC 2
INFO [HintsDispatcher:134] 2020-12-07 16:22:37,249 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 74c05489-4374-4013-8ece-fd408b672d3d-1607227934695-1.hints to endpoint /****DC-1 ip3****: 74c05489-4374-4013-8ece-fd408b672d3d, partially INFO [HintsDispatcher:126] 2020-12-07 16:22:45,066 HintsStore.java:133 - Deleted hint file 5054087d-32f8-4d96-af96-fbc93e2b6cf9-1607338346642-1.hints INFO [HintsDispatcher:126] 2020-12-07 16:22:45,067 HintsDispatchExecutor.java:282 - Finished hinted handoff of file 5054087d-32f8-4d96-af96-fbc93e2b6cf9-1607338346642-1.hints to endpoint /****DC-1 ip1****: 5054087d-32f8-4d96-af96-fbc93e2b6cf9 INFO [HintsDispatcher:135] 2020-12-07 16:22:45,090 HintsStore.java:133 - Deleted hint file 6b3f348d-62fd-42dd-9f93-5ce0f1cc6b0b-1607338346641-1.hints INFO [HintsDispatcher:135] 2020-12-07 16:22:45,090 HintsDispatchExecutor.java:282 - Finished hinted handoff of file 6b3f348d-62fd-42dd-9f93-5ce0f1cc6b0b-1607338346641-1.hints to endpoint /****DC-1 ip4****: 6b3f348d-62fd-42dd-9f93-5ce0f1cc6b0b INFO [HintsDispatcher:128] 2020-12-07 16:22:45,091 HintsStore.java:133 - Deleted hint file e5715590-f083-4dab-9806-c89b31f4bacf-1607338346642-1.hints INFO [HintsDispatcher:128] 2020-12-07 16:22:45,092 HintsDispatchExecutor.java:282 - Finished hinted handoff of file e5715590-f083-4dab-9806-c89b31f4bacf-1607338346642-1.hints to endpoint /****DC-1 ip2****: e5715590-f083-4dab-9806-c89b31f4bacf
DC1
INFO [HintsDispatcher:15471] 2020-12-07 15:49:06,862 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a-1607310632977-1.hints to endpoint /****DC-2 ip5****: 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a, partially INFO [HintsDispatcher:15472] 2020-12-07 15:49:16,863 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a-1607310632977-1.hints to endpoint /****DC-1 ip2****: 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a, partially INFO [HintsDispatcher:15473] 2020-12-07 15:49:26,862 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a-1607310632977-1.hints to endpoint /****DC-1 ip2****: 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a, partially INFO [HintsDispatcher:15474] 2020-12-07 15:49:36,863 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a-1607310632977-1.hints to endpoint /****DC-1 ip2****: 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a, partially INFO [HintsDispatcher:15475] 2020-12-07 15:49:46,863 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 4c5194c1-6549-48fd-bbb1-ac67be4b7e3a-1607310632977-1.hints to endpoint /****DC-1 ip2****:
Need to know, why hints are being applied partially as seen in logs and why the hints physical files are not being deleted. I can see 3 days old hints files in system.