question

kinomoto avatar image
kinomoto asked Erick Ramirez edited

Stream failed due to CompactionExecutor

While joining a node to the cluster, we intermittently get:

ERROR [CompactionExecutor:2] 2021-02-03 12:06:09,966 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:2,1,main]java.util.NoSuchElementException: null

This is immediately followed by:

ERROR [StreamReceiveTask:19] 2021-02-03 12:06:09,966 StreamSession.java:593 - [Stream #fba1aa60-6613-11eb-974b-391d84d34cb8] Streaming error occurred on session with peer 10.51.41.58
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.NoSuchElementException

Then it sends a fail message to the peer:

DEBUG [STREAM-OUT-/10.51.41.58:7001] 2021-02-03 12:06:09,967 ConnectionHandler.java:379 - [Stream #fba1aa60-6613-11eb-974b-391d84d34cb8] Sending Session Failed

On the peer's log there's only:

ERROR [STREAM-IN-/10.51.41.79:34226] 2021-02-03 12:06:09,970 StreamSession.java:706 - [Stream #fba1aa60-6613-11eb-974b-391d84d34cb8] Remote peer 10.51.41.79 failed stream session.

Any tips how to troubleshoot these and get rid of them? The bootstrap eventually fails and we believe it is because of these errors. Thanks!

compaction
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

You will need to review the logs on the source node to determine why the compaction thread CompactionExecutor:2 is getting a NoSuchElementException.

You will need to pay particular attention to the SSTable(s) which are generating the exception. Look into the schema + contents of the problematic SSTable(s) for clues as to why the compaction thread is failing. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

kinomoto avatar image kinomoto commented ·

Thanks Erick. It turned out to be a stack of problems. Data, JVM, settings related. So, there is no simple solution. Working with Datastax support now to get things sorted out.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ kinomoto commented ·

Good to hear that you're working with DataStax Support. You're in good hands. Cheers!

0 Likes 0 ·