We are using a Cassandra cluster of 10 nodes.each node is allocated a heap size of 64 GB. Number of connections made to a node increases drastically sometimes and it results in increase of heap size(going above 45 GB) this makes all the incoming requests failed. So is there any way to limit the connections.we are using Cassandra V2.0.10.
Bringing together the Apache Cassandra experts from the community and DataStax.
Want to learn? Have a question? Want to share your expertise? You are in the right place!
Not sure where to begin? Getting Started
This question was
closed
by
Erick Ramirez for the following reason: The question is answered, right answer was accepted
Unsupported
This is a really tough question to answer because C* 2.0.10 was released 6 years ago and has reached end-of-service-life which means it hasn't been supported for a number of years now. There's not a lot of people still familiar with C* 2.0 so you're running a huge risk if this is a production cluster.
Max connections
To answer your question, limiting the number of connections depends on whether the nodes are configured with the sync (default) or hsha RPC server.
The synchronous RPC server (sync) uses one thread per Thrift connection. The unlimited rpc_max_threads means that the node will accept as many concurrent requests as memory would allow. Note that each RPC thread uses a minimum stack size of 180KB although your cluster is probably running with the default 256KB (-Xss256k) in cassandra-env.sh.
With hsha (half synchronous, half asynchronous), there is a known issue where unlimited RPC threads can crash Cassandra with an OutOfMemoryError (CASSANDRA-8116). In C* 2.0.15, a configuration exception will get thrown on startup if the max RPC threads is not set.
The workaround is to set rpc_max_threads to a value which makes sense for your use case. You can try 256 or 512 but ultimately, you need to run multiple tests to determine the optimal value for your cluster.
JVM
The ConcurrentMarkSweep garbage collector (CMS) performs well for Cassandra workloads when the heap size is in the 16-20GB range, maybe up to 24GB as the upper limit.
In my experience, you get diminishing returns once you go beyond 24GB because the amount of heap to cleanup (GC) is so large that the stop-the-world pauses get longer.
I imagine you keep increasing the heap size because your nodes are running out of memory. This is counter-productive. From what I've seen in lots of other companies, increasing the heap allocation is the wrong approach. You should really consider resizing your cluster and add more nodes to cope with the load. Cheers!
5 People are following this question.
