question

nagasree963_193876 avatar image
nagasree963_193876 asked nagasree963_193876 commented

Getting error while deploying cass-operator 1.4.0, "CassandraDatacenter dc1 is invalid"

I am deploying cassandra operator 1.4.o in a GKE cluster, My operator pod is running. But Application pods are not coming. Below were the logs of operator pod:

kubectl -n <ns> logs cass-operator-b58f486c4-9n7lt
{"level":"info","ts":1619190243.0803065,"logger":"reconciliation_handler","msg":"Reconcile loop completed","requestNamespace":"do-cass-upgrade","requestName":"dc1","loopID":"eb86a455-ede2-41c1-9e23-e6031975dc70","duration":0.017864647}
{"level":"error","ts":1619190243.0803647,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"cassandradatacenter-controller","request":"do-cass-upgrade/dc1","error":"CassandraDatacenter.cassandra.datastax.com \"dc1\" is invalid: [status.conditions.message: Required value, status.conditions.reason: Required value]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:88"}

I just deployed using the below documentation:

https://github.com/datastax/cass-operator/blob/1.4.x/operator/example-cassdc-yaml/cassandra-3.11.x/example-cassdc-minimal.yaml

https://github.com/datastax/cass-operator/blob/1.4.x/docs/user/cass-operator-manifests-v1.18.yaml

Can anyone suggest me on this.

cass-operator
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered nagasree963_193876 commented

It isn't clear to me what problem you're reporting. As a start, it would be great if you could update the body section of your question with the following:

  • output of kubectl get pods
  • description of how you configured your k8s cluster
  • commands you ran to deploy cass-operator
  • replace the screenshot you posted with actual text

A lot of people who are willing to help you are responding on their mobiles and it's difficult to read the screenshots so please paste the actual text instead.

Please be aware that version 1.4.0 was released in August 2020. Version 1.6.0 of the cass-operator was released in February. The issue you are facing may have been a known issue which has already been fixed.

A friendly final reminder to please NOT post your updates as an "answer" but instead edit your original question. Cheers!

7 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

nagasree963_193876 avatar image nagasree963_193876 commented ·

Thank You. I have updated in the above question.

0 Likes 0 ·
nagasree963_193876 avatar image nagasree963_193876 commented ·

We are currently using cassandra operator:1.1.0. we want to upgrade from 1.1.0 to 1.6.0. As per the below documentation, it was mention like first upgrade to 1.4.0 and then 1.6.0.

https://docs.datastax.com/en/cass-operator/doc/cass-operator/cassOperatorUpgrades.html

But while upgrading to 1.4.0 , cassandra pods were not restarting and getting above error. So, is that okay, if we upgrade directly to 1.6.0. Can you please suggest on the approach.

0 Likes 0 ·
john.sanda_194109 avatar image john.sanda_194109 nagasree963_193876 commented ·

Unfortunately the log message isn't very helpful. It does mention though that dc1 is invalid and something about a required value. Can you please share the output of the following:

kubectl get cassandradatacenter dc1 -o yaml


0 Likes 0 ·
nagasree963_193876 avatar image nagasree963_193876 john.sanda_194109 commented ·

Attaching a file for the above command dc1 -o yaml.txt

Can you please check it.


0 Likes 0 ·
dc1-o-yaml.txt (5.9 KiB)
Show more comments
Erick Ramirez avatar image Erick Ramirez ♦♦ nagasree963_193876 commented ·

Thanks for pointing this out. I've discussed it with the cass-operator engineers and confirmed that it is not necessary to upgrade to cass-operator 1.4.0.

You can upgrade from cass-operator 1.1.0 directly to 1.6.0. The upgrade document has been updated to reflect this. Cheers!

0 Likes 0 ·
nagasree963_193876 avatar image nagasree963_193876 Erick Ramirez ♦♦ commented ·

Thanks @Erick Ramirez. For 1.6.0, does it support scaling down the cassandra datacenter nodes. I try to scale down from 4 to 3, but nodes are not scaling down.

kubectl -n <ns> get po

NAME READY STATUS RESTARTS AGE

cass-operator-74c55b94b-hlnj9 1/1 Running 0 3h55m

cluster1-dc1-default-sts-0 2/2 Running 0 3h40m

cluster1-dc1-default-sts-1 2/2 Running 0 3h42m

cluster1-dc1-default-sts-2 2/2 Running 0 3h43m

cluster1-dc1-default-sts-3 2/2 Running 0 100m


kubectl -n do-cass-110 get sts

NAME READY AGE

cluster1-dc1-default-sts 4/4 2d23h


cass-cr.txt Attaching the file for the refernce.



0 Likes 0 ·
cass-cr.txt (1.8 KiB)