Getting DseAuthenticator warning after upgrading Kubernetes environment

Question

question

igor.rmarinho asked · Dec 03, 2020 at 04:14 PM

Getting DseAuthenticator warning after upgrading Kubernetes environment

Hi,

After upgrading my kubernetes environments I'm having the issue below:

WARN  [mainIOThread-4] 2020-12-03 16:11:07,234  DseAuthenticator.java:666 - Failed to check permissions
org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at org.apache.cassandra.cql3.QueryProcessor.validateKey(QueryProcessor.java:234)
    at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:1047)
    at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:397)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:610)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:348)
    at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:526)
    at org.apache.cassandra.auth.CassandraRoleManager.getRoleData(CassandraRoleManager.java:347)
    at com.datastax.bdp.cassandra.auth.DseRoleManager.getRoleData(DseRoleManager.java:407)
    at org.apache.cassandra.auth.AuthCache$AsyncLoader.lambda$asyncLoad$0(AuthCache.java:317)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
    at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:101)
    at org.apache.cassandra.concurrent.IOScheduler$PooledTaskWorker.run(IOScheduler.java:319)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
    at org.apache.cassandra.concurrent.IOThread.run(IOThread.java:46)
03-Dec-2020 16:11:09.474 WARNING [mainIOThread-1] com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.lambda$get$3 Exception thrown during asynchronous load
 java.util.concurrent.CompletionException: org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
    at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:101)
    at org.apache.cassandra.concurrent.IOScheduler$PooledTaskWorker.run(IOScheduler.java:319)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
    at org.apache.cassandra.concurrent.IOThread.run(IOThread.java:46)
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at org.apache.cassandra.cql3.QueryProcessor.validateKey(QueryProcessor.java:234)
    at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:1047)
    at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:397)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:610)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:348)
    at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:526)
    at org.apache.cassandra.auth.CassandraRoleManager.getRoleData(CassandraRoleManager.java:347)
    at com.datastax.bdp.cassandra.auth.DseRoleManager.getRoleData(DseRoleManager.java:407)
    at org.apache.cassandra.auth.AuthCache$AsyncLoader.lambda$asyncLoad$0(AuthCache.java:317)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
... 12 more

kubernetes

1

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

igor.rmarinho · Dec 03, 2020 at 04:18 PM

dse.yaml

dse.txt

0 ·

dse.txt (3.4 KiB)

Answer 1 · 2020-12-03T23:06:48Z

Erick Ramirez answered · Dec 03, 2020 at 11:06 PM

There isn't enough information in your post for us to give a meaningful response. Did you deploy your cluster with the cass-operator?

It would be good if you update your question with background details including:

version pre- and post-upgrade
information about your k8s configuration
security features you configured
any other relevant information

Cheers!

4 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

igor.rmarinho · Dec 04, 2020 at 01:41 AM

It was from 5.1.17 to 6.8.3.

The install and upgrade was done like this:

helm repo add artifactory https://artifactory.*******.com/artifactory/helm --username artifactory_helm --password <password>
helm repo update
helm install --name "namespace"-datastax-dse-server artifactory/datastax-dse-server --namespace="namespace" --set image.tag=<DSE version>

###Upgrade Datastax Cassandra

helm repo add artifactory https://artifactory.********.com/artifactory/helm --username artifactory_helm --password <password>
helm repo update
helm upgrade --namespace <namespace> <namespace>-datastax-dse-server artifactory/datastax-dse-server --set image.tag=<DSE versio

the authentication in the dse.yaml is

authentication_options:
     enabled: true
     default_scheme: internal
role_management_options:
    mode: internal
authorization_options:
     enabled: true
     transitional_mode: normal
     allow_row_level_security: true

I did not use cass-operator.

0 ·

Erick Ramirez ♦♦ igor.rmarinho · Dec 07, 2020 at 11:40 PM

My ability to assist you with this issue is very limited since this isn't a supported configuration and this isn't something we test.

If you log a ticket with DataStax Support, they might be able to assist you in determining why the role lookup is failing. Cheers!

0 ·

DonBeau igor.rmarinho · 5 days ago

@Erick Ramirez, by "this isn't a supported configuration and this isn't something we test", what do you mean ? Can you get more specific on what you do not support ? I'm trying to get work dse 6.8.3 in container/kubernetes with dseauthenticator activated with ldap. But I was not able yet to get it worked.

0 ·

Erick Ramirez ♦♦ DonBeau · 5 days ago

We don't test Helm upgrades. We instead recommend using the cass-operator. So if you're having issues with upgrading DSE with Helm, please go through DataStax Support. Cheers!

1 ·

question