question

sunilrpawar7_183464 avatar image
sunilrpawar7_183464 asked Erick Ramirez edited

Why is a full repair causing nodes to go down?

We have 12 nodes Cassandra cluster in production along with 12 nodes for DR.

When we are running a full repair on any one of the nodes that the Cassandra node is going down with status DN and getting out of the cluster.

We can observer high GC pause as well while the repair is being run. We have tried to tune the GC parameters but still no luck.

What are other possibilities that might be causing the node to go down?

For an add on information, when we are running full repair within local dc for each and individual table then it's running without fail.


Please find below information for reference:-

  • amount of memory allocated to the heap:- 32GB
  • total RAM on each server:- 128 GB
  • total number of cores on each server:- 16 / Threads per core:- 1
  • whether the nodes are using G1 GC or CMS:- CMS setting
  • version of Cassandra:- 3.11.2
  • node density (average size of data on each node):- 400 GB
repair
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

sunilrpawar7_183464 avatar image sunilrpawar7_183464 commented ·

For an add on information, when we are running full repair within local dc for each and individual table then it's running without fail.

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

@sunilrpawar7_183464 It sounds like the nodes get overloaded when you run repairs and is the reason they go down with what I suspect are out-of-memory errors.

Contributing factors

(a) If you don't regularly run repairs (or if this is the first time you've run repairs), chances are there's too much discrepancies between replicas and the nodes cannot cope.

(b) There's also the possibility that there is not enough memory allocated to the heap. Please update your original post with the following information:

  • amount of memory allocated to the heap
  • total RAM on each server
  • total number of cores on each server
  • whether the nodes are using G1 GC or CMS
  • version of Cassandra
  • node density (average size of data on each node)

Recommendations

(c) Run a rolling partitioner-range repair, one table at a time and one node at a time:

$ nodetool repair -pr ks_name table_name

(d) Depending on the amount of RAM on each server:

  • if using CMS GC, we recommend allocating at least 16-20GB of memory to the heap (max 24GB)
  • if using G1 GC, allocate at least 20-24GB of memory to the heap (max 31GB)

Make sure to reserve around 8GB of RAM. Cheers!

5 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

sunilrpawar7_183464 avatar image sunilrpawar7_183464 commented ·

We have tried varying the heap memory allocation ranging from 8 GB to 64 GB but still, we couldn't get breakthrough.


As suggested by you in point (C) we have tried running a repair with -pr command for each individual node. It ran successfully for the majority of the table but for one table it's failing always.


The table for which it's failing is contained a maximum load of the total density of the load.

When we are running the repair with start and end token range it's working just fine for entire keyspace but only when we are running for entire table at a time it is creating a problem.

0 Likes 0 ·
vkayanala_42513 avatar image vkayanala_42513 sunilrpawar7_183464 commented ·

@sunilrpawar7_183464 You said, a table failing full repair and also having max load! Is this table distributed evenly across the all nodes in the datacenter?

0 Likes 0 ·
sunilrpawar7_183464 avatar image sunilrpawar7_183464 vkayanala_42513 commented ·

Yes, Table is distributed evenly across all the nodes. The only problem with the table is that GC pause increase whenever we run a repair with -full and -pr option. But with --in-localdc we are not facing any issue.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ sunilrpawar7_183464 commented ·

@sunilrpawar7_183464 In our experience, CMS doesn’t perform well for large heap sizes. You should consider switching to G1 GC with a 31GB heap (not 32GB since it has a significantly lower addressable objects than 31GB). Cheers!

0 Likes 0 ·
sunilrpawar7_183464 avatar image sunilrpawar7_183464 Erick Ramirez ♦♦ commented ·

@Erick Ramirez, We had tried using G 1 GC with different heap size ranging from 8 GB to 64 GB also, but faced the same issue. We will try with keeping heap size 31 Gb instead of 32 GB.

Having said that, the node is getting down while the repair is in progress is due to high GC pauses or there might be chances of other possible scenarios as well which we can check parallelly.

0 Likes 0 ·