question

adityajain22_141041 avatar image
adityajain22_141041 asked Erick Ramirez edited

Why is LIST ROLES output different from what's in system_auth.roles?

Found some strange behavious as List roles gives output

cassandra@cqlsh> list ROLES ;

 role           | super | login | options
----------------+-------+-------+---------
      cassandra |  True |  True |        {}
       kaicloud | False |  True |        {}
    kaicloud_ro | False |  True |        {}
 kaicloud_subpo | False | False |        {}

Above output says kaicloud_subpo role is NOT a SUPER

but doing select * from system_auth.roles

cassandra@cqlsh> SELECT * from system_auth.roles ;

 role           | can_login | is_superuser | member_of | salted_hash
----------------+-----------+--------------+-----------+--------------------------------------------------------------
    kaicloud_ro |      True |        False |      null | $#################
 kaicloud_subpo |      True |         True |      null | $#################
       kaicloud |      True |        False |      null | $#################
      cassandra |      True |         True |      null | $##################

in above output it says kaicloud_subpo IS super and Login is true also.

AND same strange behavior in granting permissions too.

cassandra@cqlsh> grant all PERMISSIONS ON KEYSPACE kaicloud_subpo to kaicloud;
cassandra@cqlsh> 
cassandra@cqlsh> list all;

 role        | username    | resource                    | permission
-------------+-------------+-----------------------------+------------
    kaicloud |    kaicloud | <keyspace app_metrics_prod> |     CREATE
    kaicloud |    kaicloud | <keyspace app_metrics_prod> |     MODIFY
    kaicloud |    kaicloud |         <keyspace kaicloud> |     CREATE
    kaicloud |    kaicloud |         <keyspace kaicloud> |      ALTER
    kaicloud |    kaicloud |         <keyspace kaicloud> |     SELECT
    kaicloud |    kaicloud |         <keyspace kaicloud> |     MODIFY
 kaicloud_ro | kaicloud_ro |         <keyspace kaicloud> |     SELECT

(7 rows)
cassandra@cqlsh> 
cassandra@cqlsh> select * from system_auth.role_permissions ;

 role        | resource                 | permissions
-------------+--------------------------+--------------------------------------------------------------
 kaicloud_ro |            data/kaicloud |                                                   {'SELECT'}
    kaicloud |           data/OpsCenter | {'ALTER', 'AUTHORIZE', 'CREATE', 'DROP', 'MODIFY', 'SELECT'}
    kaicloud |    data/app_metrics_prod |                                         {'CREATE', 'MODIFY'}
    kaicloud |            data/kaicloud |                      {'ALTER', 'CREATE', 'MODIFY', 'SELECT'}
    kaicloud |      data/kaicloud_subpo | {'ALTER', 'AUTHORIZE', 'CREATE', 'DROP', 'MODIFY', 'SELECT'}
   cassandra |      data/kaicloud_subpo | {'ALTER', 'AUTHORIZE', 'CREATE', 'DROP', 'MODIFY', 'SELECT'}
   cassandra | functions/kaicloud_subpo |          {'ALTER', 'AUTHORIZE', 'CREATE', 'DROP', 'EXECUTE'}
   cassandra |     roles/kaicloud_subpo |                               {'ALTER', 'AUTHORIZE', 'DROP'}

Why i cant see permission i just grant in list all and can see in system_auth.role_permissions

> run full repair on system_auth

>nodetool describecluster || schema version is same across the cluster.

Please help to understand the behavior.

security
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

smadhavan avatar image smadhavan ♦ commented ·

@adityajain22_141041, could you also update your original question with the version of Cassandra/DSE that you are using? Also post the output of SELECT * FROM system_schema.keyspaces; command from cqlsh. Thanks!

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

@adityajain22_141041
error:
I realise you stated that you ran repairs but the symptoms you posted indicate to me that the replicas out-of-sync. The LIST ROLES command is executed with QUORUM consistency so I'm inclined to believe that output.

In contrast, unless you've overridden the default consistency in cqlsh then you are reading from the system_auth.roles table with a consistency level of ONE.

It would be easy to confirm it by forcing read-repair on the table with tracing enabled:

admin@cqlsh> TRACING ON;
Now Tracing is enabled
admin@cqlsh> CONSISTENCY ALL;
Consistency level set to ALL.
admin@cqlsh> SELECT * FROM system_auth.roles;

From the trace output, you should be able to see whether a read-repair was triggered and if the problem was out-of-sync replicas, you'll be able to see the expected rows. Cheers!

[UPDATED] - Role-based access control introduced in C* 2.2 made several tables in the system_auth keyspace obsolete. If the cluster was upgraded from C* 2.1, once all nodes have been upgraded to C* 2.2 or C* 3.x, the following obsolete tables needed to be dropped:

  • system_auth.users
  • system_auth.credentials
  • system_auth.permissions

If these tables are not dropped, it would result in unexpected behaviour from the authentication subsystem.

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

adityajain22_141041 avatar image adityajain22_141041 commented ·

Thank @Erick Ramirez, Issue was something else, We were seeing some legacy tables of cassandra 2 version like users , credentials etc in system_auth which are no more supported in cassandra3 versions.


Thanks


0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ adityajain22_141041 commented ·

Right. Role-based access control introduced in C* 2.2 made several tables in the system_auth keyspace obsolete. If the cluster was upgraded from C* 2.1, once all nodes have been upgraded to C* 2.2 or C* 3.x, the following obsolete tables needed to be dropped:

  • system_auth.users
  • system_auth.credentials
  • system_auth.permissions

If these tables are not dropped, it would result in unexpected behaviour from the authentication subsystem like the one you experienced. Cheers!

0 Likes 0 ·