Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

stephen9834 avatar image
stephen9834 asked ·

Connection has been closed when running Spark job on remote server

I am getting below error when running spark job on remote server but in same network.

[cluster8-nio-worker-2] WARN  com.datastax.driver.core.RequestHandler  - casstest.example.com/172.31.5.120:9042 replied with server error (java.util.ConcurrentModificationException), defuncting connection.
[Executor task launch worker for task 5459] ERROR org.apache.spark.executor.Executor  - Exception in task 3.0 in stage 336.0 (TID 5459)
com.datastax.driver.core.exceptions.TransportException: [casstest.example.com/172.31.5.120:9042] Connection has been closed

in Cassandra logs getting below exceptions:

system.log:ERROR [CoreThread-0] 2020-12-07 01:20:07,779  Message.java:829 - Unexpected exception during request; channel = [id: 0x5b9c0eee, L:/172.31.5.120:9042 - R:/172.31.2.270:54864]
system.log-java.util.ConcurrentModificationException: null
system.log-    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
system.log-    at java.util.ArrayList$Itr.next(ArrayList.java:859)
system.log-    at org.apache.cassandra.transport.CBUtil.writeStringList(CBUtil.java:325)
system.log-    at org.apache.cassandra.transport.Message$ProtocolEncoder.makeFrame(Message.java:438)
system.log-    at org.apache.cassandra.transport.Message$ProtocolEncoder.encode(Message.java:392)
system.log-    at org.apache.cassandra.transport.Message$ProtocolEncoder.encode(Message.java:378)
system.log-    at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
system.log-    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
system.log-    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
system.log-    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
system.log-    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
system.log-    at org.apache.cassandra.transport.Message$Dispatcher$ChannelFlusher.add(Message.java:513)
system.log-    at org.apache.cassandra.transport.Message$Dispatcher$Flusher.run(Message.java:567)
system.log-    at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.process(EpollTPCEventLoopGroup.java:924)
Can you please help or guide what can be the cause of this issue ?
dseanalytics
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

There isn't enough information in your post so we're not able to give you a meaningful response.

As a side note, I want to remind you that we are only able to give you general recommendations in this Q&A forum. If you want assistance troubleshooting your cluster, please log a ticket with DataStax Support. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.