kinomoto avatar image
kinomoto asked Erick Ramirez edited

Stream failed due to CompactionExecutor

While joining a node to the cluster, we intermittently get:

ERROR [CompactionExecutor:2] 2021-02-03 12:06:09,966 - Exception in thread Thread[CompactionExecutor:2,1,main]java.util.NoSuchElementException: null

This is immediately followed by:

ERROR [StreamReceiveTask:19] 2021-02-03 12:06:09,966 - [Stream #fba1aa60-6613-11eb-974b-391d84d34cb8] Streaming error occurred on session with peer
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.NoSuchElementException

Then it sends a fail message to the peer:

DEBUG [STREAM-OUT-/] 2021-02-03 12:06:09,967 - [Stream #fba1aa60-6613-11eb-974b-391d84d34cb8] Sending Session Failed

On the peer's log there's only:

ERROR [STREAM-IN-/] 2021-02-03 12:06:09,970 - [Stream #fba1aa60-6613-11eb-974b-391d84d34cb8] Remote peer failed stream session.

Any tips how to troubleshoot these and get rid of them? The bootstrap eventually fails and we believe it is because of these errors. Thanks!

10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

You will need to review the logs on the source node to determine why the compaction thread CompactionExecutor:2 is getting a NoSuchElementException.

You will need to pay particular attention to the SSTable(s) which are generating the exception. Look into the schema + contents of the problematic SSTable(s) for clues as to why the compaction thread is failing. Cheers!

2 comments Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

kinomoto avatar image kinomoto commented ·

Thanks Erick. It turned out to be a stack of problems. Data, JVM, settings related. So, there is no simple solution. Working with Datastax support now to get things sorted out.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ kinomoto commented ·

Good to hear that you're working with DataStax Support. You're in good hands. Cheers!

0 Likes 0 ·