question

igor.rmarinho_185445 avatar image
igor.rmarinho_185445 asked Erick Ramirez commented

Unexpected error while recording NodeSync validation

Hi,

I'm getting the use below with DSE 6.8.3 and 6.8.4 and I did'n't find anything in how to fix it. Any ideas on it?

Thank you.

ERROR [InternalResponseStage:14] 2020-10-06 09:45:48,827  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [NodeSync-1] 2020-10-06 09:45:48,830  SystemDistributedKeyspace.java:622 - Unexpected error while recording ongoing NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:10] 2020-10-06 09:45:48,840  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:18] 2020-10-06 09:45:48,842  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:17] 2020-10-06 09:45:48,845  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [NodeSync-1] 2020-10-06 09:45:48,847  SystemDistributedKeyspace.java:622 - Unexpected error while recording ongoing NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:11] 2020-10-06 09:45:48,848  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [NodeSync-1] 2020-10-06 09:45:48,854  SystemDistributedKeyspace.java:622 - Unexpected error while recording ongoing NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:11] 2020-10-06 09:45:48,856  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [NodeSync-1] 2020-10-06 09:45:49,439  SystemDistributedKeyspace.java:622 - Unexpected error while recording ongoing NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:11] 2020-10-06 09:45:49,439  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:11] 2020-10-06 09:45:49,442  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [NodeSync-1] 2020-10-06 09:45:49,443  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
nodesync
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

2 Answers

bettina.swynnerton avatar image
bettina.swynnerton answered igor.rmarinho_185445 commented

Hi @igor.rmarinho_185445,

I can see that these errors happen very frequently and would flood your system.log at this rate.

You can fully disable nodesync as a temporary workaround, and I am checking with the team what could cause this.

Here to disable nodesync ( you will need to run this on every node ):

nodetool nodesyncservice disable

https://docs.datastax.com/en/dse/6.8/dse-dev/datastax_enterprise/tools/nodetool/nodetoolNodesyncserviceDisable.html

Let us know if you have any other exceptions in the system.log after disabling it, perhaps this is a symptom of a misconfiguration.

I'll update this answer with more information when available.

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

igor.rmarinho_185445 avatar image igor.rmarinho_185445 commented ·

Thanks for your answer @bettina.swynnerton. Let me know when you have an update. :)

0 Likes 0 ·
Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

When a segment (a token range for a table) has been validated (the equivalent of "repaired") by the DSE NodeSync service, the details of the validation (including whether it was successful or not, start and end times, token range, etc) are recorded in the NodeSync validations table.

There are two known reasons why the CQL INSERT into the validations table fails:

  1. There are no replicas available (nodes are overloaded/unresponsive, down, unreachable, etc).
  2. The write operation timed out, i.e. zero replicas acknowledged the request.

In your case, the error was neither of the two hence "unexpected".

This is a support issue which is a challenge to address in a Q&A forum format and is not the right place for it. Please log a ticket with DataStax Support and attach a Diagnostics Report from OpsCenter so one of our engineers can investigate and assist you directly. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.