Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

igor.rmarinho avatar image
igor.rmarinho asked Erick Ramirez commented

Getting DseAuthenticator warning after upgrading Kubernetes environment

Hi,

After upgrading my kubernetes environments I'm having the issue below:

WARN  [mainIOThread-4] 2020-12-03 16:11:07,234  DseAuthenticator.java:666 - Failed to check permissions
org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at org.apache.cassandra.cql3.QueryProcessor.validateKey(QueryProcessor.java:234)
    at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:1047)
    at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:397)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:610)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:348)
    at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:526)
    at org.apache.cassandra.auth.CassandraRoleManager.getRoleData(CassandraRoleManager.java:347)
    at com.datastax.bdp.cassandra.auth.DseRoleManager.getRoleData(DseRoleManager.java:407)
    at org.apache.cassandra.auth.AuthCache$AsyncLoader.lambda$asyncLoad$0(AuthCache.java:317)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
    at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:101)
    at org.apache.cassandra.concurrent.IOScheduler$PooledTaskWorker.run(IOScheduler.java:319)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
    at org.apache.cassandra.concurrent.IOThread.run(IOThread.java:46)
03-Dec-2020 16:11:09.474 WARNING [mainIOThread-1] com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.lambda$get$3 Exception thrown during asynchronous load
 java.util.concurrent.CompletionException: org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
    at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:101)
    at org.apache.cassandra.concurrent.IOScheduler$PooledTaskWorker.run(IOScheduler.java:319)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
    at org.apache.cassandra.concurrent.IOThread.run(IOThread.java:46)
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at org.apache.cassandra.cql3.QueryProcessor.validateKey(QueryProcessor.java:234)
    at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:1047)
    at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:397)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:610)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:348)
    at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:526)
    at org.apache.cassandra.auth.CassandraRoleManager.getRoleData(CassandraRoleManager.java:347)
    at com.datastax.bdp.cassandra.auth.DseRoleManager.getRoleData(DseRoleManager.java:407)
    at org.apache.cassandra.auth.AuthCache$AsyncLoader.lambda$asyncLoad$0(AuthCache.java:317)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
... 12 more
kubernetes
1 comment
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

There isn't enough information in your post for us to give a meaningful response. Did you deploy your cluster with the cass-operator?

It would be good if you update your question with background details including:

  • version pre- and post-upgrade
  • information about your k8s configuration
  • security features you configured
  • any other relevant information

Cheers!

4 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

It was from 5.1.17 to 6.8.3.

The install and upgrade was done like this:

helm repo add artifactory https://artifactory.*******.com/artifactory/helm --username artifactory_helm --password <password>
helm repo update
helm install --name "namespace"-datastax-dse-server artifactory/datastax-dse-server --namespace="namespace" --set image.tag=<DSE version>

###Upgrade Datastax Cassandra

helm repo add artifactory https://artifactory.********.com/artifactory/helm --username artifactory_helm --password <password>
helm repo update
helm upgrade --namespace <namespace> <namespace>-datastax-dse-server artifactory/datastax-dse-server --set image.tag=<DSE versio


the authentication in the dse.yaml is

authentication_options:
     enabled: true
     default_scheme: internal
role_management_options:
    mode: internal
authorization_options:
     enabled: true
     transitional_mode: normal
     allow_row_level_security: true


I did not use cass-operator.


0 Likes 0 ·

My ability to assist you with this issue is very limited since this isn't a supported configuration and this isn't something we test.

If you log a ticket with DataStax Support, they might be able to assist you in determining why the role lookup is failing. Cheers!

0 Likes 0 ·
DonBeau avatar image DonBeau igor.rmarinho ·

@Erick Ramirez, by "this isn't a supported configuration and this isn't something we test", what do you mean ? Can you get more specific on what you do not support ? I'm trying to get work dse 6.8.3 in container/kubernetes with dseauthenticator activated with ldap. But I was not able yet to get it worked.

0 Likes 0 ·

We don't test Helm upgrades. We instead recommend using the cass-operator. So if you're having issues with upgrading DSE with Helm, please go through DataStax Support. Cheers!

1 Like 1 ·