We have a Cassandra cluster of 8 nodes (Apache Cassandra 3.11.11). Due to disk failure on 2 nodes we have removed those nodes from the cluster. While trying to add them back to the cluster the bootstring process is failing with below error.
ERROR [STREAM-IN-/172.29.62.28:7000] 2021-10-10 23:08:03,437 DefaultFSErrorHandler.java:94 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" org.apache.cassandra.io.FSWriteError: java.io.FileNotFoundException: /cassandra/data/keyspace1/data_tbl-b4f243f986c711e8a0bc25f553f28aa6/me-62630-big-Filter.db (Too many open file) at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.flushBf(BigTableWriter.java:496) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.doPrepare(BigTableWriter.java:516) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:364) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:264) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:59) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.finish(RangeAwareSSTableWriter.java:130) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:113) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:672) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:539) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317) ~[apache-cassandra-3.11.11.jar:3.11.11] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102] Caused by: java.io.FileNotFoundException: /cassandra/data/keyspace1/data_tbl-b4f243f986c711e8a0bc25f553f28aa6/me-62630-big-Filter.db (Too many open files) at java.io.FileOutputStream.open0(Native Method) ~[na:1.8.0_102]
This issue is caused by table keyspace1/data_tbl which has too many sstable under its directory. Few sstables are having size less than 1mb.
[cassandra@host data_tbl -b4f243f986c711e8a0bc25f553f28aa6]$ ls -lcrt |wc -l 371455
As temporary fix we have increased open fie limit but that didn't fix the issue and node is again started failing during bootstrap process with above error. Any solution to fix this issue?