question

m7o avatar image
m7o asked m7o commented

Why is TRUNCATE causing nodes to go down?

We have single (1) node Cassandra (v.3.11.2) cluster with heap 1GB.

Case 1:

1. Drop all (500-600 with 1,000 - 200,000 rows for each) tables

2. Create new tables (with new or same structure)

3. Populate the data into Cassandra

Case 2:

1. Truncate all (500-600 with 1,000 - 200,000 rows for each) tables

3. Populate the data into Cassandra

The issue is that when we use truncate case we have OutOfMemory exception after +-200 tables. But for the drop case, it works fine.

Why is a truncate causing nodes to go down? Has anyone encountered such a problem? Perhaps there are guidelines for using truncation?

Best Regards,

Mykhailo

truncate
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered m7o commented

It isn't so much a problem with TRUNCATE itself but it is symptomatic of the underlying issues with your cluster.

The things that are really problematic are:

  1. Single node.
  2. Small heap size.
  3. Too many tables.

As a general rule:

  1. We recommend a minimum of 3 nodes in each DC. Single-node clusters are only suitable for development environments where you mostly do functional testing and low operations per second.
  2. A heap of 1GB is only suitable for very low throughputs. In development environments, we recommend at least 4GB allocated to the heap. For production workloads, 8GB is the minimum for very low traffic but at least 16GB allocated to the heap otherwise.
  3. We recommend a maximum of 200 tables total per cluster across all keyspaces (regardless of the number of keyspaces). Each table uses 1MB of memory to hold metadata about the tables so in your case where 1GB is allocated to the heap, 500-600MB is used just for table metadata with hardly any heap space left for other operations.
4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

m7o avatar image m7o commented · 
1. We recommend a minimum of 3 nodes in each DC. Single-node clusters are only suitable for development environments where you mostly do functional testing and low operations per second.

This general point is not a surprise to me, but I don't understand why it would contribute to the problem we are seeing with TRUNCATE. I would not expect TRUNCATE to require more resources than DROP, and also don't see how adding more nodes would relieve the pressure given that a table might have to be truncated across many nodes.

So, I still have a question why would truncate be worse than drop in our case?

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ m7o commented ·
It seems like you misunderstood my response. If it wasn't clear, I didn't mean that a single-node cluster would cause the issues you're seeing -- just that it's problematic given the low resources you have. Cheers!
0 Likes 0 ·
fwy_187020 avatar image fwy_187020 commented ·

I find this point very interesting, and it is not something I have read in any Cassandra/Datastax documentation.

3. We recommend a maximum of 200 tables total per cluster across all keyspaces (regardless of the number of keyspaces). Each table uses 1MB of memory to hold metadata about the tables so in your case where 1GB is allocated to the heap, 500-600MB is used just for table metadata with hardly any heap space left for other operations.


However, I'm going to ask about this one in a separate thread.

0 Likes 0 ·
m7o avatar image m7o commented ·

After taking a couple of runs, we saw that the memory usage increases all the time when the truncate is used and does not decrease when the garbage collector is called. But for drop and subsequent table creation, this problem is not observed. Could it be a memory leak in the truncate implementation?

Memory heap for the truncate (Note that the memory freezes at 750-800, after which the Cassandra node terminates with OutOfMemory):

refresh-all-domains-truncate.jpg

Memory heap for the drop:

refresh-all-domains-drop.jpgrefresh-all-domains-drop-2.jpg

0 Likes 0 ·
refresh-all-domains-drop.jpg (348.8 KiB)
refresh-all-domains-drop-2.jpg (347.0 KiB)
refresh-all-domains-truncate.jpg (307.1 KiB)