question

amit82ocp_172818 avatar image
amit82ocp_172818 asked Erick Ramirez commented

Lost permissions after upgrading cluster from Cassandra 2.1.5 to 3.11

Hello All,

We have recently perform the cassandra upgrade from 2.15 to 3.11 version and during the upgrade we have dropped the following tables.

system_auth.users.

system_auth.permissions

system_auth.credentials.

however after upgrade the database we lost all the user permissions, which are working fine in other cluster.

We do not have roles in 2.15 and due to that these tables need to drop before upgrade so that it would be managed by roles in 3.11.

Though we have resolved the issue be granting the permission manually however how can we avoid same situation in further upgrade ?

upgrade
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

Based on your description, it looks like you did the steps in the wrong order -- you were supposed to drop the legacy authentication tables after all nodes have been upgraded as it states in NEWS.txt:

For systems already using the internal auth implementations, the process

for converting existing data during a rolling upgrade is straightforward.
As each node is restarted, it will attempt to convert any data in the

legacy tables into the new schema. Until enough nodes to satisfy the

replication strategy for the system_auth keyspace are upgraded and so have

the new schema, this conversion will fail with the failure being reported

in the system log.

...

Once all nodes are upgraded, an operator with superuser privileges should

drop the legacy tables, system_auth.users, system_auth.credentials and

system_auth.permissions. Doing so will prompt Cassandra to switch over to

the new tables without requiring any further intervention.

Since you dropped the legacy tables before the nodes were upgraded, Cassandra didn't have any legacy tables to convert to the new schema. Cheers!

[UPDATE] During startup after a node is upgraded, Cassandra will run/re-run the conversion process if:

  1. Internal authentication is enabled, AND
  2. Legacy tables exist on the node, AND
  3. At least a QUORUM of nodes have been upgraded.

Here are example log entries on a node indicating successful conversion of legacy users and credentials:

INFO  [OptionalTasks:1] CassandraRoleManager.java:410 - Converting legacy users
INFO  [OptionalTasks:1] CassandraRoleManager.java:420 - Completed conversion of legacy users
INFO  [OptionalTasks:1] CassandraRoleManager.java:425 - Migrating legacy credentials data to new system table
INFO  [OptionalTasks:1] CassandraRoleManager.java:438 - Completed conversion of legacy credentials

If internal authorizer is enabled, here are example log entries on a node indicating successful conversion of the legacy permissions:

INFO  [OptionalTasks:1] CassandraAuthorizer.java:396 - Converting legacy permissions data
INFO  [OptionalTasks:1] CassandraAuthorizer.java:435 - Completed conversion of legacy permissions

If the conversion is not successful for whatever reason, here is an example log entry on a node:

INFO  [OptionalTasks:1] CassandraAuthorizer.java:468 - Unable to complete conversion of legacy auth data (perhaps not enough nodes are upgraded yet). Conversion should not be considered complete

You should only drop the legacy authentication tables when ALL nodes in cluster report successful conversion of the tables in the logs.

For details, see the blog post Role Based Access Control In Cassandra. Cheers!

4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

amit82ocp_172818 avatar image amit82ocp_172818 commented ·

Thanks Erick for reply on this.

Please note that we have dropped these tables after all the nodes upgrade only and it went fine for other cluster only one cluster reported this privilege's issue.

Regards,

Amit

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ amit82ocp_172818 commented ·

That indicates that something went wrong when Cassandra attempted to convert the legacy authentication tables. You will need to review the logs around the time after the nodes were upgraded for clues. Cheers!

0 Likes 0 ·
amit82ocp_172818 avatar image amit82ocp_172818 commented ·

@Erick Ramirez, could you please let us know what all precaution we should take as a sanity check before dropping next time?

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ amit82ocp_172818 commented ·

I've updated my answer with details of how to verify successful conversion of legacy tables. Cheers!

0 Likes 0 ·