question

ranjeet_ranjee avatar image
ranjeet_ranjee asked ranjeet_ranjee commented

Why am I getting "AuthenticationFailed" exception when one node is down?

I am facing an Authentication issue. We have 2 node of cluster. If DSE stop on any one node I am unable to login inside DSE on another node. Due to Authentication issue application also get down.

Command

[root@cassandra-7 ~]# cqlsh 10.130.1.204 -u user -p password

Error

Connection error: ('Unable to connect to any servers', {'10.130.1.204': AuthenticationFailed('Failed to authenticate to 10.130.1.204: Error from server: code=0100 [Bad credentials] message="Failed to login. Please re-try."',)})

Cluster Info 

[root@cassandra-7 ~]# dsetool status
Warning: Not able to read Analytics Master data, SparkMaster data may not be correct.

DC: singaporedo     Workload: Cassandra       Graph: no     
======================================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--   Address             Load             Owns   Token                        Rack         Health [0,1] 
                                                 1051212759506365289                                       
UN   Node-1-Public-IP   380.01 GiB        ?      -921962553594005710          rack-1       0.90         
UN   Node-2-Public-IP   407.31 GiB        ?      1051212759506365289          rack-1       0.90         

Kindly help to fix this.

dseauthentication
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ranjeet_ranjee commented

Since you only have 2 nodes and 1 node is down, that indicates that you have not setup the replication correctly.

I recommend you bring both nodes back online and update the system_auth and dse_security keyspaces to have 2 replicas in the singaporedo DC with:

user@cqlsh> ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy','singaporedo' : 2 };
user@cqlsh> ALTER KEYSPACE dse_security WITH REPLICATION= {'class' : 'NetworkTopologyStrategy','singaporedo' : 2 };

You will need to repair both keyspaces after you've updated the replication settings.

NOTE - This approach does not apply to the default cassandra superuser. Authentication uses a consistency level of LOCAL_ONE authenticating with the default superuser requires a QUORUM of nodes. You should only ever use the default user to create other accounts then drop the default cassandra superuser.

For more information, see Configuring the security keyspaces replication factors. Cheers!

14 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

ranjeet_ranjee avatar image ranjeet_ranjee commented ·

Thanks Erick.

If I execute your 2 recommended command will it impact any of the applications or do I need to make any changes?

0 Likes 0 ·
ranjeet_ranjee avatar image ranjeet_ranjee commented ·

Thanks Erick.

If I execute your 2 recommended command will it impact any of the applications or do I need to make any changes?

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ ranjeet_ranjee commented ·

No, there isn't any impact. All operations on C* are pretty much online. I forgot to mention that you need to repair the keyspaces so I've updated my answer. Cheers!

0 Likes 0 ·
ranjeet_ranjee avatar image ranjeet_ranjee Erick Ramirez ♦♦ commented ·

So you mean to say I only need to repair keyspace using below 2 command


nodetool repair --full system_auth
nodetool repair --full dse_security
0 Likes 0 ·
Show more comments