question

miguel.oyarzo_185223 avatar image
miguel.oyarzo_185223 asked Erick Ramirez commented

OpsCenter shows 24% of the data was not synced before the NodeSync deadline

Through OpsCenter 6.6.7, I've installed DES 6.7.7 in 3 EC2 nodes, same region (single datacenter). All seemed good, no errors, all agents in right version, everything synced in green.

Then, I changed the replication factor and strategy for all my custom keyspaces:

ALTER KEYSPACE <KeySpace> WITH replication = {'class': 'NetworkTopologyStrategy','ap-southeast-2': '3'} AND durable_writes = true;


Also, I changed 3 preinstalled keyspaces system_auth, dse_securit & dse_insights and applied same RF=3 and strategy (NetworkTopologyStrategy).
Finally, I run "nodetool repair -pr" in every node, as suggested

However, Opscenter/NodeSyn option shows: dse_insights is not synced

24% of the data was not synced before the NodeSync deadline. An increase in configured throughput is recommended.


nodetool shows:

$ nodetool status

--  Address      Load       Tokens       Owns    Host ID                               Rack
UN  10.1.18.20   1.8 GiB    8            ?       3dce59c0-0598-4f68-84d0-d3b95bb1e5d7  ap-southeast-2a
UN  10.1.19.20   2.11 GiB   8            ?       10462c39-5ade-4659-b447-50045a510dbf  ap-southeast-2c
UN  10.1.18.148  2.43 GiB   8            ?       d453b12c-c3a9-40b2-9639-08e219b85ca1  ap-southeast-2b

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

$ nodetool repair --full dse_insights
[2020-02-26 21:55:23,501] Skipping anti-entropy repair on tables with NodeSync enabled: [dse_insights.insights_config, dse_insights.tokens].
[2020-02-26 21:55:23,503] Starting repair command #31206 (b0791580-58e2-11ea-ba93-b9f25d1473d7), repairing keyspace dse_insights with repair options (parallelism: parallel, primary range: false, incremental: false, job threads: 1, ColumnFamilies: {}, dataCenters: {}, hosts: {}, previewKind: NONE, # of ranges: 24, pull repair: false, force repair: false)



In OpsCenter I enabled the Repair tool, but no effect.

What's missing or wrong?

cassandra.yaml
- partitioner: Murmur3partitioner
- num_tokens: 8
- allocate_tokens_for_local_replication_factor: 3
- authenticator: DseAuthenticator
- autorizer: DseAutorizer
- Role_manager: DseRoleManager

Miguel

dsenodesync
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

2 Answers

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

@miguel.oyarzo_185223 You can't run a regular repair operation on tables which you have configured to use Nodesync validation.

However, it looks like you're running into a Nodesync reporting bug. OpsCenter reads an internal status table to determine the status of Nodesync validation segments. There are certain situations where the Nodesync status table ends up with stale entries which then causes incorrect status reporting in OpsCenter (internal bug ID OPSC-15823).

The workaround is to temporarily disable the Nodesync service, truncate the Nodesync status table, then re-enable the Nodesync service. For details, see the KB article NodeSync Status tab in OpsCenter reports false urgencies. Cheers!

4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

miguel.oyarzo_185223 avatar image miguel.oyarzo_185223 commented ·

Hi Erick,
I have disabled Nodesync service on that table (the only one I had enabled), then

cqlsh> truncate table system_distributed.nodesync_status;


and finally I reenabled Nodesync for dse_insights. It now shows "100% of the data was synced within the first 50% of the time period for the NodeSync deadline."
Is this the expected output? thanks you @Erick Ramirez

Also, I have created about 14 keyspaces. Should I enable Nodesync for all of them or no needed?

Cheers

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ miguel.oyarzo_185223 commented ·

@miguel.oyarzo_185223 Sorry, I didn't mean "disable Nodesync on the table". I literally meant "disable the Nodesync service". Please see the document I linked for detailed steps for the workaround.

For clarification when you truncate the status table, it applies to all tables with Nodesync validation enabled. Cheers!

0 Likes 0 ·
miguel.oyarzo_185223 avatar image miguel.oyarzo_185223 Erick Ramirez ♦♦ commented ·

Great answer @Erick Ramirez
All good now!
Cheers

0 Likes 0 ·
Show more comments
sebastian.estevez@datastax.com avatar image
sebastian.estevez@datastax.com answered Erick Ramirez commented

Here's the doc for tuning nodesync https://docs.datastax.com/en/dse/6.7/dse-dev/datastax_enterprise/config/nodesyncTuning.html


Nodesync and repair should not be run on the same tables. Pick one (new shiny node sync or old ugly inefficient repair)

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

miguel.oyarzo_185223 avatar image miguel.oyarzo_185223 commented ·

Hi Sebastian,
Thanks for your prompt answer.

I forgot to mention that I already increased rate_in_kb from 1024 (default) to 2048 in cassandra.yaml and run the installation again, but no effect.
The weird part is this is a new installation and and both tables in that keyspace are empty:

Any idea?
Cheers

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ miguel.oyarzo_185223 commented ·

@miguel.oyarzo_185223 Thanks for being part of the DataStax Community. A friendly comment to let you know that I've converted your post to a comment since it is not an "answer". This is to avoid confusion when users view your post in the future. Cheers!

1 Like 1 ·