OpsCenter shows 24% of the data was not synced before the NodeSync deadline

Question

question

miguel.oyarzo_185223 asked · Feb 26 at 09:59 PM

OpsCenter shows 24% of the data was not synced before the NodeSync deadline

Through OpsCenter 6.6.7, I've installed DES 6.7.7 in 3 EC2 nodes, same region (single datacenter). All seemed good, no errors, all agents in right version, everything synced in green.

Then, I changed the replication factor and strategy for all my custom keyspaces:

ALTER KEYSPACE <KeySpace> WITH replication = {'class': 'NetworkTopologyStrategy','ap-southeast-2': '3'} AND durable_writes = true;

Also, I changed 3 preinstalled keyspaces system_auth, dse_securit & dse_insights and applied same RF=3 and strategy (NetworkTopologyStrategy).
Finally, I run "nodetool repair -pr" in every node, as suggested

However, Opscenter/NodeSyn option shows: dse_insights is not synced

24% of the data was not synced before the NodeSync deadline. An increase in configured throughput is recommended.

nodetool shows:

$ nodetool status

--  Address      Load       Tokens       Owns    Host ID                               Rack
UN  10.1.18.20   1.8 GiB    8            ?       3dce59c0-0598-4f68-84d0-d3b95bb1e5d7  ap-southeast-2a
UN  10.1.19.20   2.11 GiB   8            ?       10462c39-5ade-4659-b447-50045a510dbf  ap-southeast-2c
UN  10.1.18.148  2.43 GiB   8            ?       d453b12c-c3a9-40b2-9639-08e219b85ca1  ap-southeast-2b

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

$ nodetool repair --full dse_insights
[2020-02-26 21:55:23,501] Skipping anti-entropy repair on tables with NodeSync enabled: [dse_insights.insights_config, dse_insights.tokens].
[2020-02-26 21:55:23,503] Starting repair command #31206 (b0791580-58e2-11ea-ba93-b9f25d1473d7), repairing keyspace dse_insights with repair options (parallelism: parallel, primary range: false, incremental: false, job threads: 1, ColumnFamilies: {}, dataCenters: {}, hosts: {}, previewKind: NONE, # of ranges: 24, pull repair: false, force repair: false)

In OpsCenter I enabled the Repair tool, but no effect.

What's missing or wrong?

cassandra.yaml
- partitioner: Murmur3partitioner
- num_tokens: 8
- allocate_tokens_for_local_replication_factor: 3
- authenticator: DseAuthenticator
- autorizer: DseAutorizer
- Role_manager: DseRoleManager

Miguel

dse nodesync

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-02-27T00:00:11Z

Erick Ramirez answered · Feb 27 at 12:00 AM

@miguel.oyarzo_185223 You can't run a regular repair operation on tables which you have configured to use Nodesync validation.

However, it looks like you're running into a Nodesync reporting bug. OpsCenter reads an internal status table to determine the status of Nodesync validation segments. There are certain situations where the Nodesync status table ends up with stale entries which then causes incorrect status reporting in OpsCenter (internal bug ID OPSC-15823).

The workaround is to temporarily disable the Nodesync service, truncate the Nodesync status table, then re-enable the Nodesync service. For details, see the KB article NodeSync Status tab in OpsCenter reports false urgencies. Cheers!

4 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

miguel.oyarzo_185223 · Feb 27 at 12:42 AM

Hi Erick,
I have disabled Nodesync service on that table (the only one I had enabled), then

cqlsh> truncate table system_distributed.nodesync_status;

and finally I reenabled Nodesync for dse_insights. It now shows "100% of the data was synced within the first 50% of the time period for the NodeSync deadline."
Is this the expected output? thanks you @Erick Ramirez

Also, I have created about 14 keyspaces. Should I enable Nodesync for all of them or no needed?

Cheers

0 · ·

Erick Ramirez ♦♦ miguel.oyarzo_185223 · Feb 27 at 12:50 AM

@miguel.oyarzo_185223 Sorry, I didn't mean "disable Nodesync on the table". I literally meant "disable the Nodesync service". Please see the document I linked for detailed steps for the workaround.

For clarification when you truncate the status table, it applies to all tables with Nodesync validation enabled. Cheers!

0 · ·

miguel.oyarzo_185223 Erick Ramirez ♦♦ · Feb 27 at 02:23 AM

Great answer @Erick Ramirez
All good now!
Cheers

0 · ·

Show more comments

Answer 2 · 2020-02-26T22:01:22Z

sebastian.estevez@datastax.com answered · Feb 26 at 10:01 PM

Here's the doc for tuning nodesync https://docs.datastax.com/en/dse/6.7/dse-dev/datastax_enterprise/config/nodesyncTuning.html

Nodesync and repair should not be run on the same tables. Pick one (new shiny node sync or old ugly inefficient repair)

2 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

miguel.oyarzo_185223 · Feb 26 at 10:13 PM

Hi Sebastian,
Thanks for your prompt answer.

I forgot to mention that I already increased rate_in_kb from 1024 (default) to 2048 in cassandra.yaml and run the installation again, but no effect.
The weird part is this is a new installation and and both tables in that keyspace are empty:

Any idea?
Cheers

0 · ·

Erick Ramirez ♦♦ miguel.oyarzo_185223 · Feb 27 at 12:02 AM

@miguel.oyarzo_185223 Thanks for being part of the DataStax Community. A friendly comment to let you know that I've converted your post to a comment since it is not an "answer". This is to avoid confusion when users view your post in the future. Cheers!

1 · ·

question