Getting DseAuthenticator warning after upgrading Kubernetes environment

Question

question

igor.rmarinho asked Dec 3, '20 Erick Ramirez commented Jan 22, '21

Getting DseAuthenticator warning after upgrading Kubernetes environment

Hi,

After upgrading my kubernetes environments I'm having the issue below:

WARN  [mainIOThread-4] 2020-12-03 16:11:07,234  DseAuthenticator.java:666 - Failed to check permissions
org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at org.apache.cassandra.cql3.QueryProcessor.validateKey(QueryProcessor.java:234)
    at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:1047)
    at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:397)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:610)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:348)
    at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:526)
    at org.apache.cassandra.auth.CassandraRoleManager.getRoleData(CassandraRoleManager.java:347)
    at com.datastax.bdp.cassandra.auth.DseRoleManager.getRoleData(DseRoleManager.java:407)
    at org.apache.cassandra.auth.AuthCache$AsyncLoader.lambda$asyncLoad$0(AuthCache.java:317)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
    at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:101)
    at org.apache.cassandra.concurrent.IOScheduler$PooledTaskWorker.run(IOScheduler.java:319)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
    at org.apache.cassandra.concurrent.IOThread.run(IOThread.java:46)
03-Dec-2020 16:11:09.474 WARNING [mainIOThread-1] com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.lambda$get$3 Exception thrown during asynchronous load
 java.util.concurrent.CompletionException: org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
    at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:101)
    at org.apache.cassandra.concurrent.IOScheduler$PooledTaskWorker.run(IOScheduler.java:319)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
    at org.apache.cassandra.concurrent.IOThread.run(IOThread.java:46)
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at org.apache.cassandra.cql3.QueryProcessor.validateKey(QueryProcessor.java:234)
    at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:1047)
    at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:397)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:610)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:348)
    at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:526)
    at org.apache.cassandra.auth.CassandraRoleManager.getRoleData(CassandraRoleManager.java:347)
    at com.datastax.bdp.cassandra.auth.DseRoleManager.getRoleData(DseRoleManager.java:407)
    at org.apache.cassandra.auth.AuthCache$AsyncLoader.lambda$asyncLoad$0(AuthCache.java:317)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
... 12 more

kubernetes

1

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

igor.rmarinho · Dec 03, 2020 at 04:18 PM

dse.yaml

dse.txt

0 ·

dse.txt (3.4 KiB)

Answer 1 · 2020-12-03T23:06:48Z

Erick Ramirez answered Dec 3, '20 Erick Ramirez commented Jan 22, '21

There isn't enough information in your post for us to give a meaningful response. Did you deploy your cluster with the cass-operator?

It would be good if you update your question with background details including:

version pre- and post-upgrade
information about your k8s configuration
security features you configured
any other relevant information

Cheers!

4 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

igor.rmarinho · Dec 04, 2020 at 01:41 AM

It was from 5.1.17 to 6.8.3.

The install and upgrade was done like this:

helm repo add artifactory https://artifactory.*******.com/artifactory/helm --username artifactory_helm --password <password>
helm repo update
helm install --name "namespace"-datastax-dse-server artifactory/datastax-dse-server --namespace="namespace" --set image.tag=<DSE version>

###Upgrade Datastax Cassandra

helm repo add artifactory https://artifactory.********.com/artifactory/helm --username artifactory_helm --password <password>
helm repo update
helm upgrade --namespace <namespace> <namespace>-datastax-dse-server artifactory/datastax-dse-server --set image.tag=<DSE versio

the authentication in the dse.yaml is

authentication_options:
     enabled: true
     default_scheme: internal
role_management_options:
    mode: internal
authorization_options:
     enabled: true
     transitional_mode: normal
     allow_row_level_security: true

I did not use cass-operator.

0 ·

Erick Ramirez ♦♦ igor.rmarinho · Dec 07, 2020 at 11:40 PM

My ability to assist you with this issue is very limited since this isn't a supported configuration and this isn't something we test.

If you log a ticket with DataStax Support, they might be able to assist you in determining why the role lookup is failing. Cheers!

0 ·

DonBeau igor.rmarinho · Jan 21 at 07:14 PM

@Erick Ramirez, by "this isn't a supported configuration and this isn't something we test", what do you mean ? Can you get more specific on what you do not support ? I'm trying to get work dse 6.8.3 in container/kubernetes with dseauthenticator activated with ldap. But I was not able yet to get it worked.

0 ·

Erick Ramirez ♦♦ DonBeau · Jan 22 at 12:14 AM

We don't test Helm upgrades. We instead recommend using the cass-operator. So if you're having issues with upgrading DSE with Helm, please go through DataStax Support. Cheers!

1 ·

question