Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

sparikh71_28533 avatar image
sparikh71_28533 asked ·

Node crashes with "map failed" error

Hi all, we are seeing this error "java.lang.OutOfMemoryError: Map failed" in the log and nodes are crashing after that. I have verified there is no ULIMIT issue and we are using 64 bit JVM.

We are currently doing heavy data ingestion other than that there is no live traffic. It is a 4 node cluster with 64G ram and 6 7T SSD drives with JBOD setup. Each node has currently has more than 2T of data.

Due to the constant crashing of nodes, compactions (leveled) are taking for ever.Any help or pointers are greatly appreciated. Thanks in advance.

cassandra crashes.
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

2 Answers

Erick Ramirez avatar image
Erick Ramirez answered ·

@sparikh71_28533 The symptoms you describe indicate the nodes are overloaded leading to OOM. Try throttling back the ingestion to a throughput that the cluster can handle. If you provide the version of DSE running on the nodes and amount of memory allocated to the heap, we can give you additional recommendations. Cheers!

Share
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

sparikh71_28533 avatar image
sparikh71_28533 answered ·

Thanks so much @Erick Ramirez for your response. We are running apache cassandra version 3.0.18.

We have allocated 38G to heap and concurrent_compactors at 8. At one time we had it 24 and had to gradually decrease it to 8 because the nodes were crashing so frequently.

Our team's concern is compactions are going at very slow rate. Also the next step we need to is to change the RF from 1 to 3. Until and unless both of these steps are finished we can not make this cluster to go live and accept user traffic.


Do you recommend increasing the RAM followed by heap and will help? Please let me know if I can provide any additional information.

Thanks again.




Share
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.