Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

michael.guissine_30999 avatar image
michael.guissine_30999 asked ·

DSE 6.7.3, nodes are going down with Assertion errors

After upgrading DSE from 5.1.14 to 6.7.3 we are seeing nodes are going down with lots of errors like the one below in the logs, any thoughts?

ERROR [CoreThread-0] 2019-06-25 10:15:30,519  VerbHandlers.java:77 - Unexpected error during execution of request READS.SINGLE_READ (99097935): /10.16.6.6 -> /10.16.6.4
java.lang.AssertionError: Expected valid buffer or boundary crossed
                at org.apache.cassandra.utils.flow.Flow$ReduceSubscriber.onError(Flow.java:1221)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlatMap.onError(FlatMap.java:133)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.onError(FlatMap.java:185)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.onError(FlatMap.java:185)
                at org.apache.cassandra.io.sstable.format.AsyncPartitionReader$PartitionReader.onError(AsyncPartitionReader.java:365)


cassandradse 6.7.3
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

@michael.guissine_30999 the stack trace is incomplete but if it were, I'm pretty sure it would include o.a.c.utils.memory.buffers.TemporaryBufferPool. If it does, it's confirmation that it's for a known issue in DSE 6.7.3 (ticket ID DB-3172). We aim to get the fix included in the next release of DSE (no ETA yet).

You can workaround the issue by temporarily downgrading the binaries to DSE 6.7.2 -- this won't have any impact on the data. I've written about the issue in detail here -- https://support.datastax.com/hc/en-us/articles/360028462472. Cheers!

7 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

I get this issue too.

Maybe it fixed. (6.7.4)


Reference: the DataStax Enterprise 6.7 release notes:

Resolved issues:
AssertionError in temporary buffer pool causes CorruptSSTableException. (DB-3172, DB-3174)
1 Like 1 · ·

You're correct @Beck. DB-3172 which I wrote about in the article above was fixed in DSE 6.7.4. When I wrote the article, 6.7.4 was not release yet. Cheers!

1 Like 1 · ·

@michael.guissine_30999 DSE 6.7.4 is out now so you should be able to do a simpole binary upgrade. Cheers!

1 Like 1 · ·

thank you @Erick Ramirez, we managed to resolved the issue by downgrading to 6.7.2 along with disabling Asynchronous IO -Ddse.io.aio.enabled=false and limiting file size cache file_cache_size_in_mb: 1024 . Next, we will try upgrading to 6.7.4

1 Like 1 · ·

For future reference, disabling AIO is not recommended since it will significantly affect the performance of your cluster. There are very limited edge cases where it is suggested by a DataStax expert after exhaustive investigation. Cheers!

0 Likes 0 · ·
Show more comments