Getting "failed to create a child event loop" after making changes to dse.yaml

Question

question

kari.lahdensuo_150258 asked May 5, '20 Erick Ramirez edited Nov 24, '20

Getting "failed to create a child event loop" after making changes to dse.yaml

I am installing a new DSE 6.8 cluster. Installation was successful, but I get error message "failed to create a child event loop" when changing the configuration (dse.yaml).

It similar as this case:

https://stackoverflow.com/questions/49760148/akkadatastax-getting-error-failed-to-create-a-child-event-loop-java-lang-ill

My stack trace:

2020-05-04 11:42:00,931Z [opscenterd] ERROR: Unexpected error processing configure job 08d030ba-e9ce-4ef5-9b47-64b42a24adfb on node name="xx" ssh-management-address="x.x.x.x" node-id="f39e820e-fc0c-48db-9876-2f4b6eba8f50" (async-thread-macro-51)
java.lang.IllegalStateException: failed to create a child event loop
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:50)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:70)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:65)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:56)
at com.datastax.driver.core.NettyUtil.newEventLoopGroupInstance(NettyUtil.java:130)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:86)
at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:990)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1778)
at com.datastax.driver.core.Cluster.init(Cluster.java:220)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:395)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:374)
at com.datastax.driver.core.Cluster.connect(Cluster.java:319)
at com.datastax.driver.core.DelegatingCluster.connect(DelegatingCluster.java:68)
at com.datastax.driver.dse.DseCluster.connect(DseCluster.java:521)
at opscenter_common.driver$validate_credentials_BANG_.invokeStatic(driver.clj:80)
at opscenter_common.driver$validate_credentials_BANG_.invoke(driver.clj:75)
at lcm.dse.auth$get_current_password$fn__8760$fn__8762.invoke(auth.clj:213)
at lcm.dse.auth$get_current_password$fn__8760.invoke(auth.clj:212)
at lcm.dse.auth$get_current_password.invokeStatic(auth.clj:203)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:176)
at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:150)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:103)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:50)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:70)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:65)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:56)
at com.datastax.driver.core.NettyUtil.newEventLoopGroupInstance(NettyUtil.java:130)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:86)
at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:990)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1778)
at com.datastax.driver.core.Cluster.init(Cluster.java:220)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:395)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:374)
at com.datastax.driver.core.Cluster.connect(Cluster.java:319)
at com.datastax.driver.core.DelegatingCluster.connect(DelegatingCluster.java:68)
at com.datastax.driver.dse.DseCluster.connect(DseCluster.java:521)
at opscenter_common.driver$validate_credentials_BANG_.invokeStatic(driver.clj:80)
at opscenter_common.driver$validate_credentials_BANG_.invoke(driver.clj:75)
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.makePipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:174)
at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:150)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:103)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:50)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:70)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:65)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:56)
at com.datastax.driver.core.NettyUtil.newEventLoopGroupInstance(NettyUtil.java:130)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:86)
at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:990)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1778)
at com.datastax.driver.core.Cluster.init(Cluster.java:220)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:395)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:374)
at com.datastax.driver.core.Cluster.connect(Cluster.java:319)
at com.datastax.driver.core.DelegatingCluster.connect(DelegatingCluster.java:68)

I was enabling the authorization_options under dse.yaml

I have

nofile 1048576

lifecycle manager

2

10 |1000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez ♦♦ commented · May 04, 2020 at 09:43 AM

@kari.lahdensuo_150258 We need more info to diagnose the problem. Please edit your original question to include:

the full error message with the stack trace
details of configuration changes you made
the previous configuration values

Cheers!

0 ·

Erick Ramirez ♦♦ Erick Ramirez ♦♦ commented · May 04, 2020 at 12:05 PM

@kari.lahdensuo_150258 I noticed that you posted an answer most likely because your update couldn't fit into a comment which has a maximum of 1000 characters. But your post was not an "answer" so I've rejected it.

A friendly reminder that in my comment, I requested that you edit your original question to include the updates. Cheers!

0 ·

DataStax Enterprise is powered by the best distribution of Apache Cassandra ™

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Privacy Policy Terms of Use

Answer 1 · 2020-05-04T12:03:03Z

Erick Ramirez answered May 4, '20 kari.lahdensuo_150258 commented Nov 24, '20

@kari.lahdensuo_150258 Do you know what the maximum number of open files is configured for the nodes? These sections of the stack trace show that one of the nodes has reached the limit:

Caused by: io.netty.channel.ChannelException: failed to open a new selector
        at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:176)
        at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:150)
        ...
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        ...

Please check the nodes are configured with the recommended settings. In particular, you will need to increase the maximum open files to:

nofile 1048576

For details, see Recommended production settings. Cheers!

UPDATE - If your nodes have more than 1M open files, you'll need to look into why that is the case (e.g. is compaction falling because the node or disk is overloaded). To get around this, temporarily increase the limit. Cheers!

3 Share

10 |1000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

kari.lahdensuo_150258 commented · May 04, 2020 at 11:48 AM

[Update posted in original question]

0 ·

kari.lahdensuo_150258 commented · May 07, 2020 at 06:25 AM

I am setting up the DSE and there is nothing in the Cassandra yet... only DSE is loading this system and it has mainly the default settings. I wonder why there would be so many open files.

I have 3 nodes totally and actually there is another problem after rebooting the other node. It is similar as this and I need to investigate it https://stackoverflow.com/questions/38178719/meld-error-with-datastax-enterprise

0 ·

Erick Ramirez ♦♦ kari.lahdensuo_150258 commented · May 07, 2020 at 06:53 AM

@kari.lahdensuo_150258 You'll need to figure out what those files are though they're most likely lots of SSTables. Cheers!

P.S. I've converted your post to a comment since it's not an answer. :)

0 ·

question