Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

stephen9834 avatar image
stephen9834 asked Erick Ramirez answered

Connection has been closed when running Spark job on remote server

I am getting below error when running spark job on remote server but in same network.

[cluster8-nio-worker-2] WARN  com.datastax.driver.core.RequestHandler  - casstest.example.com/172.31.5.120:9042 replied with server error (java.util.ConcurrentModificationException), defuncting connection.
[Executor task launch worker for task 5459] ERROR org.apache.spark.executor.Executor  - Exception in task 3.0 in stage 336.0 (TID 5459)
com.datastax.driver.core.exceptions.TransportException: [casstest.example.com/172.31.5.120:9042] Connection has been closed

in Cassandra logs getting below exceptions:

system.log:ERROR [CoreThread-0] 2020-12-07 01:20:07,779  Message.java:829 - Unexpected exception during request; channel = [id: 0x5b9c0eee, L:/172.31.5.120:9042 - R:/172.31.2.270:54864]
system.log-java.util.ConcurrentModificationException: null
system.log-    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
system.log-    at java.util.ArrayList$Itr.next(ArrayList.java:859)
system.log-    at org.apache.cassandra.transport.CBUtil.writeStringList(CBUtil.java:325)
system.log-    at org.apache.cassandra.transport.Message$ProtocolEncoder.makeFrame(Message.java:438)
system.log-    at org.apache.cassandra.transport.Message$ProtocolEncoder.encode(Message.java:392)
system.log-    at org.apache.cassandra.transport.Message$ProtocolEncoder.encode(Message.java:378)
system.log-    at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
system.log-    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
system.log-    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
system.log-    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
system.log-    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
system.log-    at org.apache.cassandra.transport.Message$Dispatcher$ChannelFlusher.add(Message.java:513)
system.log-    at org.apache.cassandra.transport.Message$Dispatcher$Flusher.run(Message.java:567)
system.log-    at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.process(EpollTPCEventLoopGroup.java:924)
Can you please help or guide what can be the cause of this issue ?
dseanalytics
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

There isn't enough information in your post so we're not able to give you a meaningful response.

As a side note, I want to remind you that we are only able to give you general recommendations in this Q&A forum. If you want assistance troubleshooting your cluster, please log a ticket with DataStax Support. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.