question

manoj.sharma99765_192577 avatar image
manoj.sharma99765_192577 asked Erick Ramirez commented

Getting NullPointerException when running nodetool upgradesstables

this is not production data request. We are just performing testing of up gradation from dse 5.0.15 to 6.7.4 .

We are following below document for upgradation :

https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50to67.html

We have performed all the given steps on the test server & upgraded dse from 5.0.15 to 6.7.4 .

Post upgradation when we are running nodetool upgradesstable command to upgrade sstable to current version we are getting error :

root@ubuntu:/home/ubuntu# nodetool upgradesstables test_keyspace
error: Corrupt (negative) value length encountered
-- StackTrace --
java.io.IOException: Corrupt (negative) value length encountered
    at org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:372)
    at org.apache.cassandra.db.marshal.CollectionType$CollectionPathSerializer.deserialize(CollectionType.java:260)
    at org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:268)
    at org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:683)
    at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:620)
    at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:488)
    at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:444)
    at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86)
    at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:62)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:155)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:101)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:33)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:131)
    at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:74)
    at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:68)
    at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:28)

We checked logs at the same time we get below in it:

ERROR [CompactionExecutor:3] 2020-07-08 13:06:01,659 CassandraDaemon.java:129 - Exception in thread Thread[CompactionExecutor:3,5,RMI Runtime]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big-Data.db
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:167)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:101)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:33)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:131)
    at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:74)
    at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:68)
    at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:28)
    at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:93)
    at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:236)
    at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:208)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:87)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
    at org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:506)
    at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:375)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Corrupt (negative) value length encountered

We tried scrubbing this corrupted sstable using header fix command but again we are getting below error :

root@ubuntu:/home/ubuntu# sstablescrub -e fix-only test_keyspace quote
WARN 14:32:18,698 Only 46.123GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
WARN 14:32:27,821 Legacy view Keyspace(name='test_keyspace').VIEW_ASSIGNMENT_BY_UUID detected with unselected columns. Out of order updates to these columns may not be reflected on view. Re-create view to remove this limitation.
WARN 14:32:27,822 Legacy view Keyspace(name='test_keyspace').VIEW_ASSIGNMENT_BY_UUID detected with filtered columns. Expiration or removal of filtered columns may not remove view rows in some scenarios. Re-create view to remove this limitation.
Pre-scrub sstables snapshotted into snapshot pre-scrub-1594198947823
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'config_set' as type 'config_set', but schema mentions 'frozen<config_set>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'lead_info' as type 'lead_info', but schema mentions 'frozen<lead_info>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'maintainance_values' as type 'maintainance_values', but schema mentions 'frozen<maintainance_values>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'pricing_summary' as type 'pricing_summary', but schema mentions 'frozen<pricing_summary>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'project_component_details' as type 'project_component_details', but schema mentions 'frozen<project_component_details>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'quote_details' as type 'quote_details', but schema mentions 'frozen<quote_details>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'validation_status' as type 'validation_status', but schema mentions 'frozen<validation_status>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'line_items' as type 'set<line_item>', but schema mentions 'set<frozen<line_item>>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'quote_profiles' as type 'set<quote_profile>', but schema mentions 'set<frozen<quote_profile>>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'site_details' as type 'set<party>', but schema mentions 'set<frozen<party>>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-7-big: contains column 'site_wise_pricing_summary' as type 'set<map_type>', but schema mentions 'set<frozen<map_type>>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'config_set' as type 'config_set', but schema mentions 'frozen<config_set>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'lead_info' as type 'lead_info', but schema mentions 'frozen<lead_info>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'maintainance_values' as type 'maintainance_values', but schema mentions 'frozen<maintainance_values>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'pricing_summary' as type 'pricing_summary', but schema mentions 'frozen<pricing_summary>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'project_component_details' as type 'project_component_details', but schema mentions 'frozen<project_component_details>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'quote_details' as type 'quote_details', but schema mentions 'frozen<quote_details>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'validation_status' as type 'validation_status', but schema mentions 'frozen<validation_status>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'line_items' as type 'set<line_item>', but schema mentions 'set<frozen<line_item>>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'quote_profiles' as type 'set<quote_profile>', but schema mentions 'set<frozen<quote_profile>>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'site_details' as type 'set<party>', but schema mentions 'set<frozen<party>>'
ERROR sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-6-big: contains column 'site_wise_pricing_summary' as type 'set<map_type>', but schema mentions 'set<frozen<map_type>>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'config_set' needs to be updated from type 'config_set' to 'frozen<config_set>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'lead_info' needs to be updated from type 'lead_info' to 'frozen<lead_info>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'maintainance_values' needs to be updated from type 'maintainance_values' to 'frozen<maintainance_values>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'pricing_summary' needs to be updated from type 'pricing_summary' to 'frozen<pricing_summary>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'project_component_details' needs to be updated from type 'project_component_details' to 'frozen<project_component_details>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'quote_details' needs to be updated from type 'quote_details' to 'frozen<quote_details>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'validation_status' needs to be updated from type 'validation_status' to 'frozen<validation_status>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'line_items' needs to be updated from type 'set<line_item>' to 'set<frozen<line_item>>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'quote_profiles' needs to be updated from type 'set<quote_profile>' to 'set<frozen<quote_profile>>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'site_details' needs to be updated from type 'set<party>' to 'set<frozen<party>>'
INFO sstable /var/lib/cassandra/data/test_keyspace/quote-622b9ad1bb6411eaa5f16143bebae437/mc-20-big: Column 'site_wise_pricing_summary' needs to be updated from type 'set<map_type>' to 'set<frozen<map_type>>'
INFO Stopping due to previous errors. Either fix the errors or specify the ignore-errors option.
Errors in serialization-header detected, aborting.

Please check all this information and help us to resolve this

dseupgrade
1594103370190.png (59.8 KiB)
6 comments
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

rakeshgouda1992_96602 avatar image rakeshgouda1992_96602 commented ·

Hi @manoj.sharma99765_192577 ,

Please check if any tables with COMPACT STORAGE option which needs to be removed before upgrade to 6.7.x version as explained in the below link:

https://docs.datastax.com/en/dse/5.1/cql/cql/cql_using/dropCompactStorage.html

Thanks,

Rakesh

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ rakeshgouda1992_96602 commented ·

Thanks for helping out @rakeshgouda1992_96602 but Manoj is facing a completely different issue that requires assistance from DataStax Support.

We appreciate you being part of the community. Cheers!

1 Like 1 ·
manoj.sharma99765_192577 avatar image manoj.sharma99765_192577 rakeshgouda1992_96602 commented ·

@rakeshgouda1992_96602 as we checked all from all ks compact storage options has been removed. We have only excluded system keyspace while removing .

As i can check keyspace and table had udt type frozen before up-gradation and it is same post upgrade then how it is possible that sstable of such table can not be of type frozen?


Please refere this error :
technology_to_gsp_mapping-3ea5abf1bb6411eaa5f16143bebae437/mc-6-big: contains column 'product_store_filter_set' as type 'set<product_store_filter>', but schema mentions 'set<frozen<product_store_filter>>'


Here in old version column was of type frozen. But in sstable Why sstable column is not set to frozen ?

0 Likes 0 ·
rakeshgouda1992_96602 avatar image rakeshgouda1992_96602 manoj.sharma99765_192577 commented ·

Hi @manoj.sharma99765_192577,

after DSE 5.1, UDT used with collection types only supports frozen type.Please check from your old(backup) schema, which all tables using UDT with collection type that are not frozen.

As this is not production,Please try to use ignore-error option with "-e fix-only" and once this is done , run a normal scrub to fix other errors.

Thanks,

Rakesh

0 Likes 0 ·
1594720539564.png (10.2 KiB)
Show more comments

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

There isn't enough detail in your post to be able to diagnose your problem and given it's production data, I don't want to make any requests in a public forum.

Please log a ticket with DataStax Support and attach a Diagnostics Report from OpsCenter if possible so one of our engineers can assist you directly. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

manoj.sharma99765_192577 avatar image manoj.sharma99765_192577 commented ·

@Erick Ramirez Please check updated question . Thanks

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ manoj.sharma99765_192577 commented ·

This requires assistance from Support and you'll need to work with one of our engineers. Please log a ticket with DataStax Support and attach a Diagnostics Report from OpsCenter. Cheers!

0 Likes 0 ·