Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

aravinth_chakravarthyr_173918 avatar image
aravinth_chakravarthyr_173918 asked Erick Ramirez edited

Why does commitlog replay result in CommitLogReadException?

I need an information why commit log replay error occurs with the following log


ERROR [main] 2020-01-23 12:51:16,810 CommitLogReplayer.java:429 - Ignoring commit log replay error

org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Encountered bad header at position 291 of commit log /var/lib/cassandra/commitlog/CommitLog-600-1579614223044.log, with bad position but valid CRC

at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:111)

at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:84)

at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)

at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)

at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:226)

at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:132)

at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:152)

at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:207)

at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:188)

at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:323)

at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:499)

at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:678)

at com.datastax.bdp.DseModule.main(DseModule.java:96)



I guess it couldnot find some commitlog in the live commitlog directory for replay . Is that correct

Please give me a document / a view why this error occurs and what can be a possible solution for it

cassandracommitlog
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

@aravinth_chakravarthyr_173918 The error you posted indicates that the commit log CommitLog-600-1579614223044.log is corrupt and cannot be read.

The most common causes for this issue are:

  • there is a hardware failure such as bad sectors on the disk
  • the server crashed while the filesystem was in the middle of syncing the file to disk
  • the Cassandra process got interrupted mid-transaction

or some variation of the causes I described. What was the reason you had to restart Cassandra -- was it an unexpected outage or failure?

In any case, the workaround is to move the file out of the commit log directory and start Cassandra. Once C* is up and running, you will need to run repairs on the node to pick up the missing mutations which did not get replayed from the corrupt commit log. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

aravinth_chakravarthyr_173918 avatar image aravinth_chakravarthyr_173918 ·

The commit log directory file system was full and we extended the file system size and restarted it .

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ aravinth_chakravarthyr_173918 ·

That means that the commitlog didn't get completely written to disk which is why it is unreadable. Just remove it and it should allow you to restart the node. Cheers!

0 Likes 0 ·