question

shubham.gupta_182020 avatar image
shubham.gupta_182020 asked Erick Ramirez edited

Error indexing partition after upgrading to dse 6.7.5 from 5.0.7

Upgrading to version 6.7.5 in dse from 5.0.7.I have snapshot of keyspace which is restored properly on dse version 6.7.5 but it gives error during indexing dfter creating core

dsetool -l cassandra -p $CASSANDRA_PASSWORD reload_core us.lead reindex=true deleteAll=false
dsetool -l cassandra -p $CASSANDRA_PASSWORD reload_core us.product reindex=true deleteAll=false
ERROR [CoreThread-0] 2019-09-26 14:16:57,891 AbstractSolrSecondaryIndex.java:1482 - Error indexing partition 'CON-PSOP-CP-7975G' on 'us.product':
Error indexing partition '5270' on 'us.lead':
com.datastax.bdp.search.SearchReadBeforeWriteException: Unable to complete Search read-before-write.
    at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.readBeforeWriteError(AbstractSolrSecondaryIndex.java:2071)
    at io.reactivex.internal.operators.single.SingleResumeNext$ResumeMainSingleObserver.onError(SingleResumeNext.java:73)
    at org.apache.cassandra.utils.flow.Flow$1SingleFromFlow$1ReduceToSingle.signalError(Flow.java:1539)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/us/product-336e8d60df7d11e98ee3e99aa5a3a4fa/mc-439-big-Data.db

due to this error solr user interface is also not working but dse service is running on the node and in nodetool status both the nodes are up having approx same load.

Also tried to upgradesstables command but below issue occured

nodetool upgradesstables us lead
error: EOF after 245554130 bytes out of 1818583396
--StackTrace –
java.io.EOFException: EOF after 245554130 bytes out of 1818583396
    at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:70)
    at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:62)
    at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:472)
search
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

@shubham.gupta_182020 the errors you posted indicate that the SSTable is corrupted and cannot be read. As a workaround, you need to do the following:

  • run nodetool drain so all memtables get flushed to disk
  • stop DSE on the node
  • remove product-336e8d60df7d11e98ee3e99aa5a3a4fa/mc-439-big-*.db from /var/lib/cassandra/data/us/
  • start DSE

Once DSE has successfully started, repair the node. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.