stephen9834 avatar image
stephen9834 asked Erick Ramirez answered

Connection has been closed when running Spark job on remote server

I am getting below error when running spark job on remote server but in same network.

[cluster8-nio-worker-2] WARN  com.datastax.driver.core.RequestHandler  - replied with server error (java.util.ConcurrentModificationException), defuncting connection.
[Executor task launch worker for task 5459] ERROR org.apache.spark.executor.Executor  - Exception in task 3.0 in stage 336.0 (TID 5459)
com.datastax.driver.core.exceptions.TransportException: [] Connection has been closed

in Cassandra logs getting below exceptions:

system.log:ERROR [CoreThread-0] 2020-12-07 01:20:07,779 - Unexpected exception during request; channel = [id: 0x5b9c0eee, L:/ - R:/]
system.log-java.util.ConcurrentModificationException: null
system.log-    at java.util.ArrayList$Itr.checkForComodification(
system.log-    at java.util.ArrayList$
system.log-    at org.apache.cassandra.transport.CBUtil.writeStringList(
system.log-    at org.apache.cassandra.transport.Message$ProtocolEncoder.makeFrame(
system.log-    at org.apache.cassandra.transport.Message$ProtocolEncoder.encode(
system.log-    at org.apache.cassandra.transport.Message$ProtocolEncoder.encode(
system.log-    at io.netty.handler.codec.MessageToMessageEncoder.write(
system.log-    at
system.log-    at
system.log-    at
system.log-    at
system.log-    at org.apache.cassandra.transport.Message$Dispatcher$ChannelFlusher.add(
system.log-    at org.apache.cassandra.transport.Message$Dispatcher$
system.log-    at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.process(
Can you please help or guide what can be the cause of this issue ?
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

There isn't enough information in your post so we're not able to give you a meaningful response.

As a side note, I want to remind you that we are only able to give you general recommendations in this Q&A forum. If you want assistance troubleshooting your cluster, please log a ticket with DataStax Support. Cheers!

10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.