Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

kari.lahdensuo_150258 avatar image
kari.lahdensuo_150258 asked ·

Getting "failed to create a child event loop" after making changes to dse.yaml

I am installing a new DSE 6.8 cluster. Installation was successful, but I get error message "failed to create a child event loop" when changing the configuration (dse.yaml).

It similar as this case:

https://stackoverflow.com/questions/49760148/akkadatastax-getting-error-failed-to-create-a-child-event-loop-java-lang-ill

My stack trace:

2020-05-04 11:42:00,931Z [opscenterd] ERROR: Unexpected error processing configure job 08d030ba-e9ce-4ef5-9b47-64b42a24adfb on node name="xx" ssh-management-address="x.x.x.x" node-id="f39e820e-fc0c-48db-9876-2f4b6eba8f50" (async-thread-macro-51)
java.lang.IllegalStateException: failed to create a child event loop
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:50)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:70)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:65)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:56)
at com.datastax.driver.core.NettyUtil.newEventLoopGroupInstance(NettyUtil.java:130)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:86)
at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:990)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1778)
at com.datastax.driver.core.Cluster.init(Cluster.java:220)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:395)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:374)
at com.datastax.driver.core.Cluster.connect(Cluster.java:319)
at com.datastax.driver.core.DelegatingCluster.connect(DelegatingCluster.java:68)
at com.datastax.driver.dse.DseCluster.connect(DseCluster.java:521)
at opscenter_common.driver$validate_credentials_BANG_.invokeStatic(driver.clj:80)
at opscenter_common.driver$validate_credentials_BANG_.invoke(driver.clj:75)
at lcm.dse.auth$get_current_password$fn__8760$fn__8762.invoke(auth.clj:213)
at lcm.dse.auth$get_current_password$fn__8760.invoke(auth.clj:212)
at lcm.dse.auth$get_current_password.invokeStatic(auth.clj:203)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:176)
at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:150)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:103)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:50)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:70)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:65)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:56)
at com.datastax.driver.core.NettyUtil.newEventLoopGroupInstance(NettyUtil.java:130)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:86)
at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:990)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1778)
at com.datastax.driver.core.Cluster.init(Cluster.java:220)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:395)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:374)
at com.datastax.driver.core.Cluster.connect(Cluster.java:319)
at com.datastax.driver.core.DelegatingCluster.connect(DelegatingCluster.java:68)
at com.datastax.driver.dse.DseCluster.connect(DseCluster.java:521)
at opscenter_common.driver$validate_credentials_BANG_.invokeStatic(driver.clj:80)
at opscenter_common.driver$validate_credentials_BANG_.invoke(driver.clj:75)
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.makePipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:174)
at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:150)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:103)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:50)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:70)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:65)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:56)
at com.datastax.driver.core.NettyUtil.newEventLoopGroupInstance(NettyUtil.java:130)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:86)
at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:990)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1778)
at com.datastax.driver.core.Cluster.init(Cluster.java:220)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:395)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:374)
at com.datastax.driver.core.Cluster.connect(Cluster.java:319)
at com.datastax.driver.core.DelegatingCluster.connect(DelegatingCluster.java:68)

I was enabling the authorization_options under dse.yaml


I have

nofile 1048576
dselcm
2 comments
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@kari.lahdensuo_150258 We need more info to diagnose the problem. Please edit your original question to include:

  1. the full error message with the stack trace
  2. details of configuration changes you made
  3. the previous configuration values

Cheers!

0 Likes 0 · ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Erick Ramirez ♦♦ ·

@kari.lahdensuo_150258 I noticed that you posted an answer most likely because your update couldn't fit into a comment which has a maximum of 1000 characters. But your post was not an "answer" so I've rejected it.

A friendly reminder that in my comment, I requested that you edit your original question to include the updates. Cheers!

0 Likes 0 · ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

@kari.lahdensuo_150258 Do you know what the maximum number of open files is configured for the nodes? These sections of the stack trace show that one of the nodes has reached the limit:

Caused by: io.netty.channel.ChannelException: failed to open a new selector
        at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:176)
        at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:150)
        ...
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        ...

Please check the nodes are configured with the recommended settings. In particular, you will need to increase the maximum open files to:

nofile 1048576

For details, see Recommended production settings. Cheers!

UPDATE - If your nodes have more than 1M open files, you'll need to look into why that is the case (e.g. is compaction falling because the node or disk is overloaded). To get around this, temporarily increase the limit. Cheers!

2 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

I am setting up the DSE and there is nothing in the Cassandra yet... only DSE is loading this system and it has mainly the default settings. I wonder why there would be so many open files.


I have 3 nodes totally and actually there is another problem after rebooting the other node. It is similar as this and I need to investigate it https://stackoverflow.com/questions/38178719/meld-error-with-datastax-enterprise


0 Likes 0 · ·
Erick Ramirez avatar image Erick Ramirez ♦♦ kari.lahdensuo_150258 ·

@kari.lahdensuo_150258 You'll need to figure out what those files are though they're most likely lots of SSTables. Cheers!

P.S. I've converted your post to a comment since it's not an answer. :)

0 Likes 0 · ·