Unexpected error while recording NodeSync validation

Question

question

igor.rmarinho_185445 asked Oct 6, '20 Erick Ramirez commented Oct 16, '20

Unexpected error while recording NodeSync validation

Hi,

I'm getting the use below with DSE 6.8.3 and 6.8.4 and I did'n't find anything in how to fix it. Any ideas on it?

Thank you.

ERROR [InternalResponseStage:14] 2020-10-06 09:45:48,827  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [NodeSync-1] 2020-10-06 09:45:48,830  SystemDistributedKeyspace.java:622 - Unexpected error while recording ongoing NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:10] 2020-10-06 09:45:48,840  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:18] 2020-10-06 09:45:48,842  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:17] 2020-10-06 09:45:48,845  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [NodeSync-1] 2020-10-06 09:45:48,847  SystemDistributedKeyspace.java:622 - Unexpected error while recording ongoing NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:11] 2020-10-06 09:45:48,848  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [NodeSync-1] 2020-10-06 09:45:48,854  SystemDistributedKeyspace.java:622 - Unexpected error while recording ongoing NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:11] 2020-10-06 09:45:48,856  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [NodeSync-1] 2020-10-06 09:45:49,439  SystemDistributedKeyspace.java:622 - Unexpected error while recording ongoing NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:11] 2020-10-06 09:45:49,439  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [InternalResponseStage:11] 2020-10-06 09:45:49,442  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary
ERROR [NodeSync-1] 2020-10-06 09:45:49,443  SystemDistributedKeyspace.java:622 - Unexpected error while recording NodeSync validation: this won't prevent NodeSync but may lead to ranges being validated more often than necessary

nodesync

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-10-06T14:54:00Z

bettina.swynnerton answered Oct 6, '20 igor.rmarinho_185445 commented Oct 6, '20

Hi @igor.rmarinho_185445,

I can see that these errors happen very frequently and would flood your system.log at this rate.

You can fully disable nodesync as a temporary workaround, and I am checking with the team what could cause this.

Here to disable nodesync ( you will need to run this on every node ):

nodetool nodesyncservice disable

https://docs.datastax.com/en/dse/6.8/dse-dev/datastax_enterprise/tools/nodetool/nodetoolNodesyncserviceDisable.html

Let us know if you have any other exceptions in the system.log after disabling it, perhaps this is a symptom of a misconfiguration.

I'll update this answer with more information when available.

1 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

igor.rmarinho_185445 · Oct 06, 2020 at 03:51 PM

Thanks for your answer @bettina.swynnerton. Let me know when you have an update. :)

0 ·

Answer 2 · 2020-10-07T05:52:19Z

Erick Ramirez answered Oct 7, '20 Erick Ramirez commented Oct 16, '20

When a segment (a token range for a table) has been validated (the equivalent of "repaired") by the DSE NodeSync service, the details of the validation (including whether it was successful or not, start and end times, token range, etc) are recorded in the NodeSync validations table.

There are two known reasons why the CQL INSERT into the validations table fails:

There are no replicas available (nodes are overloaded/unresponsive, down, unreachable, etc).
The write operation timed out, i.e. zero replicas acknowledged the request.

In your case, the error was neither of the two hence "unexpected".

This is a support issue which is a challenge to address in a Q&A forum format and is not the right place for it. Please log a ticket with DataStax Support and attach a Diagnostics Report from OpsCenter so one of our engineers can investigate and assist you directly. Cheers!

Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

question