Node crashes with "map failed" error

Question

question

sparikh71_28533 asked · Aug 07 at 04:38 PM

Node crashes with "map failed" error

Hi all, we are seeing this error "java.lang.OutOfMemoryError: Map failed" in the log and nodes are crashing after that. I have verified there is no ULIMIT issue and we are using 64 bit JVM.

We are currently doing heavy data ingestion other than that there is no live traffic. It is a 4 node cluster with 64G ram and 6 7T SSD drives with JBOD setup. Each node has currently has more than 2T of data.

Due to the constant crashing of nodes, compactions (leveled) are taking for ever.Any help or pointers are greatly appreciated. Thanks in advance.

cassandra crashes.

0

10 |400 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MB each and 10.5 MB total.

Answer 1 · 2019-08-07T23:06:27Z

Erick Ramirez answered · Aug 07 at 11:06 PM

@sparikh71_28533 The symptoms you describe indicate the nodes are overloaded leading to OOM. Try throttling back the ingestion to a throughput that the cluster can handle. If you provide the version of DSE running on the nodes and amount of memory allocated to the heap, we can give you additional recommendations. Cheers!

1 · Share

10 |400 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MB each and 10.5 MB total.

Answer 2 · 2019-08-09T16:23:27Z

sparikh71_28533 answered · Aug 09 at 04:23 PM

Thanks so much @Erick Ramirez for your response. We are running apache cassandra version 3.0.18.

We have allocated 38G to heap and concurrent_compactors at 8. At one time we had it 24 and had to gradually decrease it to 8 because the nodes were crashing so frequently.

Our team's concern is compactions are going at very slow rate. Also the next step we need to is to change the RF from 1 to 3. Until and unless both of these steps are finished we can not make this cluster to go live and accept user traffic.

Do you recommend increasing the RAM followed by heap and will help? Please let me know if I can provide any additional information.

Thanks again.

0 · Share

10 |400 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MB each and 10.5 MB total.

Node crashes with "map failed" error

question details

OTHER RESOURCES