• Home
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Spaces
  • General forum
  • DataStax Examples
  • DataStax Labs
  • Account help
  • Explore
  • Topics
  • Questions
  • Articles
  • Users
  • Badges
Skip to main content
Login or Sign Up
About
Live Coding
Events
    • General forum
    • DataStax Examples
    • DataStax Labs
    • Account help
    • Topics
    • Questions
    • Articles
    • Users
    • Badges
  • Home
  • General forum

 

Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question
sparikh71_28533 avatar image
sparikh71_28533 asked · Aug 07 at 04:38 PM

Node crashes with "map failed" error

Hi all, we are seeing this error "java.lang.OutOfMemoryError: Map failed" in the log and nodes are crashing after that. I have verified there is no ULIMIT issue and we are using 64 bit JVM.

We are currently doing heavy data ingestion other than that there is no live traffic. It is a 4 node cluster with 64G ram and 6 7T SSD drives with JBOD setup. Each node has currently has more than 2T of data.

Due to the constant crashing of nodes, compactions (leveled) are taking for ever.Any help or pointers are greatly appreciated. Thanks in advance.

cassandra crashes.
Comment
0
10 |400 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

2 Answers
  • Sort by Created Created
  • Sort by Oldest Oldest
  • Sort by Votes Votes
Erick Ramirez avatar image
Erick Ramirez answered · Aug 07 at 11:06 PM

@sparikh71_28533 The symptoms you describe indicate the nodes are overloaded leading to OOM. Try throttling back the ingestion to a throughput that the cluster can handle. If you provide the version of DSE running on the nodes and amount of memory allocated to the heap, we can give you additional recommendations. Cheers!

0 · Share
10 |400 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

sparikh71_28533 avatar image
sparikh71_28533 answered · Aug 09 at 04:23 PM

Thanks so much @Erick Ramirez for your response. We are running apache cassandra version 3.0.18.

We have allocated 38G to heap and concurrent_compactors at 8. At one time we had it 24 and had to gradually decrease it to 8 because the nodes were crashing so frequently.

Our team's concern is compactions are going at very slow rate. Also the next step we need to is to change the RF from 1 to 3. Until and unless both of these steps are finished we can not make this cluster to go live and accept user traffic.


Do you recommend increasing the RAM followed by heap and will help? Please let me know if I can provide any additional information.

Thanks again.




0 · Share
10 |400 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

question details

3 People are following this question.

sparikh71_28533 avatar image rocco.varela_18549 avatar image @brasco --Donnie Roberson-- avatar image
Answers Subscribe to Answers Answers and Comments Subscribe to Comments and Answers

Related Questions

How can I find the root cause of Cassandra dying frequently? 1 Answer

OTHER RESOURCES

DataStax Academy
Documentation
DataStax.com

DataStax Enterprise is powered by the best distribution of Apache Cassandra ™

© 2019 DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Privacy Policy Terms of Use