question

igor.rmarinho_185445 avatar image
igor.rmarinho_185445 asked Erick Ramirez edited

Why am I getting "ConfigurationException: Cannot change the number of tokens from 8 to 1" after upgrading to DSE 6.7.7?

After the upgrade when I'm starting cassadra dse service I have the error below:

ERROR [main] 2020-03-04 16:18:32,016  CassandraDaemon.java:862 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot change the number of tokens from 8 to 1
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1127)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:689)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:402)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:527)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:702)
at com.datastax.bdp.DseModule.main(DseModule.java:96)

the fix for that I found was

Step 1 - Calculate the token for the node as described in the Generating tokens document.

Step 2 - Reconfigure the token properties in cassandra.yaml:

comment out num_tokens
set the token assignment initial_token
leave auto_bootstrap: true
Step 3 - Delete the following directories. This will ensure that all their contents are cleaned out completely.

<data_file_directories>
commitlog
saved_caches
Step 4 - Recreate the directories above and make sure Cassandra has full permissions.

Step 5 - Bootstrap the node again by starting DSE.

But if I do step 3 it will delete my data, what make no sense for me.

When I'm doing 1 and 2 I have issues like:

tatus=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load   TokensOwns    Host ID                               Rack
UN  10.*.*.1  542.54 MiB  8            ? 5b7238df-92ee-4e44-8aa2-ac1819934df0  rack1
UN  10.*.*.2  787.82 MiB  8            ? 5aa41102-4621-4174-8c94-24e82d2187d8  rack1
UN  10.*.*.3  182.31 KiB  1            ? 5be1ed48-8fac-4c0a-ab35-703bf5eebb97  rack1

In cassandra.yaml file I add my values gave by

python -c 'print [str(((2**64 / 8) * i) - 2**63) for i in range(8)]'
initial_token: 1,2,3,4,5,6,7,8
ERROR [main] 2020-03-04 21:25:59,900  DseModule.java:100 - {}. Exiting...
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) An exception was caught and reported. Message: The number of initial tokens (by initial_token) specified is different from num_tokens value
  at com.datastax.bdp.DseModule.configure(Unknown Source)

How to fix this issue and keep my data, since I'm upgrading and my node 3 won't communicate with my nodes 5.1.17?

Thanks.

virtual nodes
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

2 Answers

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

@igor.rmarinho_185445 It looks like you got off-track a bit. This exception indicates that you have not set num_tokens so it defaulted to 1:

ERROR [main] 2020-03-04 16:18:32,016  CassandraDaemon.java:862 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot change the number of tokens from 8 to 1

If you check the existing DSE 5.1.17 nodes, it should have this in cassandra.yaml:

num_tokens: 8

If you set it to 8, the upgraded node should start as normal. You don't need to calculate/assign new tokens nor do you need to re-bootstrap the node. Cheers!

8 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

igor.rmarinho_185445 avatar image igor.rmarinho_185445 commented ·

Hi Erick, thanks for your answer! It really helped! When I change the parameter num_tokens: 8 I have another issue:


 CassandraDaemon.java:862 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot change the number of tokens from 1 to 8
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1127)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.ja

iIs it possible to change the number of tokens without deleting my /data, /commitlog etc? In a simple way without deleting everything, decomision the node, adding another datacenter etc ... as below the refrences below

https://docs.datastax.com/en/archived/cassandra/2.0/cassandra/configuration/configVnodesProduction_t.html

.


Changing Vnode token size

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ igor.rmarinho_185445 commented ·

It looks like what you did before already changed the node to only allocate 1 token. Unfortunately, there's no workaround for the changes you've already implemented so you'll need to decommission the node from the cluster and add the node back in as if it was a brand new node. Make sure you delete all the contents of the following directories:

  • data
  • commitlog
  • saved_caches

Cheers!

1 Like 1 ·
igor.rmarinho_185445 avatar image igor.rmarinho_185445 Erick Ramirez ♦♦ commented ·

Thank you Erick you have been very helpful!

So when I finish the upgrade I have to uncommit the num_tokens before I start the dse service otherwise I lose all my data, right?

After doing the decommissioning and add the node back I should upgrade node 2 and 1, change num_tokes: 8 before start the service and in the run a repair to apply the replica to node 3?



0 Likes 0 ·
Show more comments
ben.krug_85176 avatar image
ben.krug_85176 answered

Hello Igor -

The default in 6.7 is to use one token, so if you use the new cassandra.yaml after the upgrade, you'll get this error (if you use more than one).


In your case, you don't want to use initial_token (or erase your data). If you comment out initial_token and instead uncomment num_tokens and set it to 8, your node should start up.

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.