DataStax Academy FAQ

DataStax Academy migrated to a new learning management system (LMS) in July 2020. We are also moving to a new Cassandra Certification process so there are changes to exam bookings, voucher system and issuing of certificates.

Check out the Academy FAQ pages for answers to your questions:


question

pacoperalta_168099 avatar image
pacoperalta_168099 asked ·

Error attempted to serialize a 'remote' batch

Hi. We are running DSE 6.7.6 and get every so often the following Error messages,..

Could you tell me if he has noticed a similar issue?

ERROR [CoreThread-10] 2020-02-03 09:23:52,035 CassandraDaemon.java:129 - Exception in thread Thread[CoreThread-10,10,main]
java.lang.AssertionError: attempted to serialize a 'remote' batch
at org.apache.cassandra.batchlog.Batch$BatchSerializer.serializedSize(Batch.java:120)
at org.apache.cassandra.batchlog.Batch$BatchSerializer.serializedSize(Batch.java:108)
at org.apache.cassandra.net.Request.payloadSerializedSize(Request.java:293)
at org.apache.cassandra.metrics.DroppedMessageMetrics.onMessageDropped(DroppedMessageMetrics.java:123)
at org.apache.cassandra.net.DroppedMessages.onDroppedMessage(DroppedMessages.java:106)
at org.apache.cassandra.net.MessagingService.incrementDroppedMessages(MessagingService.java:985)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:54)
at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:68)
at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.process(EpollTPCEventLoopGroup.java:920)
at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.processTasks(EpollTPCEventLoopGroup.java:895)
at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.processTasks(EpollTPCEventLoopGroup.java:876)
at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.processEvents(EpollTPCEventLoopGroup.java:777)
at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.run(EpollTPCEventLoopGroup.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

dse epolltpce
1 comment
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@pacoperalta_168099 Apologies for the delay in responding. For some reason, this post went into moderation and I missed it. I'm posting a response now. Cheers!

0 Likes 0 · ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

@pacoperalta_168099 We've only ever seen this happen once before and in both occasions, they happened when recording the metrics for dropped messages (rejected reads, mutations, etc) because the node was overloaded.

What I mean is that the failed batch log was not while processing app data but when it was recording metrics. There seems to be a problem with the code path for recording dropped messages metrics but the real issue here is that your cluster is suffering from overload.

Are you seeing lots of dropped mutations/reads entries in the system.log? I apologise for the noisy AssertionError in the logs which we are looking into (internal ID DB-3905). The issue you need to address right now is the cluster being overloaded and it will be necessary to review the size of the cluster. Cheers!

3 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Thankyou @Erick Ramirez. We will consider it.

Best regards

0 Likes 0 · ·

Hi @Erick Ramirez. Thank you for your answer. Yes there are dropped mutations in the log file. We will consider it

1 Like 1 · ·

Not a problem, cheers!

0 Likes 0 · ·