Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

igor.rmarinho avatar image
igor.rmarinho asked ·

Getting DseAuthenticator warning after upgrading Kubernetes environment

Hi,

After upgrading my kubernetes environments I'm having the issue below:

WARN  [mainIOThread-4] 2020-12-03 16:11:07,234  DseAuthenticator.java:666 - Failed to check permissions
org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at org.apache.cassandra.cql3.QueryProcessor.validateKey(QueryProcessor.java:234)
    at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:1047)
    at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:397)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:610)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:348)
    at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:526)
    at org.apache.cassandra.auth.CassandraRoleManager.getRoleData(CassandraRoleManager.java:347)
    at com.datastax.bdp.cassandra.auth.DseRoleManager.getRoleData(DseRoleManager.java:407)
    at org.apache.cassandra.auth.AuthCache$AsyncLoader.lambda$asyncLoad$0(AuthCache.java:317)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
    at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:101)
    at org.apache.cassandra.concurrent.IOScheduler$PooledTaskWorker.run(IOScheduler.java:319)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
    at org.apache.cassandra.concurrent.IOThread.run(IOThread.java:46)
03-Dec-2020 16:11:09.474 WARNING [mainIOThread-1] com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.lambda$get$3 Exception thrown during asynchronous load
 java.util.concurrent.CompletionException: org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
    at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:101)
    at org.apache.cassandra.concurrent.IOScheduler$PooledTaskWorker.run(IOScheduler.java:319)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
    at org.apache.cassandra.concurrent.IOThread.run(IOThread.java:46)
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
    at org.apache.cassandra.cql3.QueryProcessor.validateKey(QueryProcessor.java:234)
    at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:1047)
    at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:397)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:610)
    at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:348)
    at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:526)
    at org.apache.cassandra.auth.CassandraRoleManager.getRoleData(CassandraRoleManager.java:347)
    at com.datastax.bdp.cassandra.auth.DseRoleManager.getRoleData(DseRoleManager.java:407)
    at org.apache.cassandra.auth.AuthCache$AsyncLoader.lambda$asyncLoad$0(AuthCache.java:317)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
... 12 more
kubernetes
1 comment
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

There isn't enough information in your post for us to give a meaningful response. Did you deploy your cluster with the cass-operator?

It would be good if you update your question with background details including:

  • version pre- and post-upgrade
  • information about your k8s configuration
  • security features you configured
  • any other relevant information

Cheers!

4 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

It was from 5.1.17 to 6.8.3.

The install and upgrade was done like this:

helm repo add artifactory https://artifactory.*******.com/artifactory/helm --username artifactory_helm --password <password>
helm repo update
helm install --name "namespace"-datastax-dse-server artifactory/datastax-dse-server --namespace="namespace" --set image.tag=<DSE version>

###Upgrade Datastax Cassandra

helm repo add artifactory https://artifactory.********.com/artifactory/helm --username artifactory_helm --password <password>
helm repo update
helm upgrade --namespace <namespace> <namespace>-datastax-dse-server artifactory/datastax-dse-server --set image.tag=<DSE versio


the authentication in the dse.yaml is

authentication_options:
     enabled: true
     default_scheme: internal
role_management_options:
    mode: internal
authorization_options:
     enabled: true
     transitional_mode: normal
     allow_row_level_security: true


I did not use cass-operator.


0 Likes 0 ·

My ability to assist you with this issue is very limited since this isn't a supported configuration and this isn't something we test.

If you log a ticket with DataStax Support, they might be able to assist you in determining why the role lookup is failing. Cheers!

0 Likes 0 ·
DonBeau avatar image DonBeau igor.rmarinho ·

@Erick Ramirez, by "this isn't a supported configuration and this isn't something we test", what do you mean ? Can you get more specific on what you do not support ? I'm trying to get work dse 6.8.3 in container/kubernetes with dseauthenticator activated with ldap. But I was not able yet to get it worked.

0 Likes 0 ·

We don't test Helm upgrades. We instead recommend using the cass-operator. So if you're having issues with upgrading DSE with Helm, please go through DataStax Support. Cheers!

1 Like 1 ·