question

pranali.khanna101994_189965 avatar image
pranali.khanna101994_189965 asked Erick Ramirez edited

What happens when hinted handoff window expires?

suppose the coordinator have some hints stored for a node to be played when it comes online.

by default the window is 3 hrs. What if this windows expires ?

Firstly , no more hints will be stored right?

also will there be any effect on hints that are already stored ? will they be deleted or ?

hinted handoff
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

The max_hint_window_in_ms is the window of time that hints (missed writes) will be generated (stored) by coordinators for replicas which are down or unresponsive. When a coordinator node detects that the replica comes back online, it effectively "resets" the hint window.

When a replica has been down for longer than the maximum hint window, coordinators will no longer store hints for the replica (see StorageProxy.shouldHint() for details):

boolean hintWindowExpired = Gossiper.instance.getEndpointDowntime(ep) > DatabaseDescriptor.getMaxHintWindow();

However, the stored hints for the replica will be kept by the coordinator nodes until the replica comes back online. Stored hints will be handed off/replayed to replicas even after max_hint_window_in_ms provided:

  • the table the mutation belongs to has not been dropped,
  • the mutation is not expired where a TTL has been set, or
  • the hint is not expired, i.e. it was created/stored less than gc_grace_seconds in the past.

In general terms, if a replica has been down for less than gc_grace_seconds then it should receive the missed mutations. Cheers!

3 comments Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

pranali.khanna101994_189965 avatar image pranali.khanna101994_189965 commented ·

What would happen in the case TTL of the mutation expired after the max hints window expired and the node is still down . That mutation will no longer be stored in the hint right ?

So when the node comes back online how TTL mutation will be played as that has not been stored because it expired after hint window expired?

Does Cassandra checks the timestamp also for the current status while playing mutations in case mutation stored have older timestamp than the current data ?

As per my understanding, when hint window expires the all the further mutations played are lost as they would not be stored

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ pranali.khanna101994_189965 commented ·

It feels like you didn't fully understand my answer so let me respond to your follow up questions briefly.

(1) The mutation is still stored as a hint as I said in my answer.

(2) It is stored as in (1) so this question is moot.

(3) Older timestamp is irrelevant since reads will only return the most current version of the data (with the latest timestamp).

(4) Mutations played? I think you misunderstood -- mutations are writes, and hinted handoff re-plays missed writes. If a node is down longer than the max hint window, you are required to repair it because it would have missed a lot of writes. Cheers!

0 Likes 0 ·
garg1982 avatar image garg1982 commented ·

[Follow up question posted in #9962]

0 Likes 0 ·