Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

pacoperalta_168099 avatar image
pacoperalta_168099 asked Erick Ramirez edited

Error attempted to serialize a 'remote' batch

Hi. We are running DSE 6.7.6 and get every so often the following Error messages,..

Could you tell me if he has noticed a similar issue?

ERROR [CoreThread-10] 2020-02-03 09:23:52,035 CassandraDaemon.java:129 - Exception in thread Thread[CoreThread-10,10,main]
java.lang.AssertionError: attempted to serialize a 'remote' batch
at org.apache.cassandra.batchlog.Batch$BatchSerializer.serializedSize(Batch.java:120)
at org.apache.cassandra.batchlog.Batch$BatchSerializer.serializedSize(Batch.java:108)
at org.apache.cassandra.net.Request.payloadSerializedSize(Request.java:293)
at org.apache.cassandra.metrics.DroppedMessageMetrics.onMessageDropped(DroppedMessageMetrics.java:123)
at org.apache.cassandra.net.DroppedMessages.onDroppedMessage(DroppedMessages.java:106)
at org.apache.cassandra.net.MessagingService.incrementDroppedMessages(MessagingService.java:985)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:54)
at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:68)
at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.process(EpollTPCEventLoopGroup.java:920)
at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.processTasks(EpollTPCEventLoopGroup.java:895)
at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.processTasks(EpollTPCEventLoopGroup.java:876)
at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.processEvents(EpollTPCEventLoopGroup.java:777)
at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.run(EpollTPCEventLoopGroup.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

dsethread-per-core
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image Erick Ramirez ♦♦ ·

@pacoperalta_168099 Apologies for the delay in responding. For some reason, this post went into moderation and I missed it. I'm posting a response now. Cheers!

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

@pacoperalta_168099 We've only ever seen this happen once before and in both occasions, they happened when recording the metrics for dropped messages (rejected reads, mutations, etc) because the node was overloaded.

What I mean is that the failed batch log was not while processing app data but when it was recording metrics. There seems to be a problem with the code path for recording dropped messages metrics but the real issue here is that your cluster is suffering from overload.

Are you seeing lots of dropped mutations/reads entries in the system.log? I apologise for the noisy AssertionError in the logs which we are looking into (internal ID DB-3905). The issue you need to address right now is the cluster being overloaded and it will be necessary to review the size of the cluster. Cheers!

3 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

pacoperalta_168099 avatar image pacoperalta_168099 ·

Hi @Erick Ramirez. Thank you for your answer. Yes there are dropped mutations in the log file. We will consider it

1 Like 1 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ pacoperalta_168099 ·

Not a problem, cheers!

0 Likes 0 ·
pacoperalta_168099 avatar image pacoperalta_168099 ·

Thankyou @Erick Ramirez. We will consider it.

Best regards

0 Likes 0 ·