I'm testing the cass-operator and have a very simple 3-node cluster setup.
Then I try to give the operator a serverImage configuration that does not work, e.g.
Now my pods (and statefulset) looks like this:
> kubectl -n cass-operator get pods,statefulset NAME READY STATUS RESTARTS AGE pod/cass-operator-78884f4f84-lmkbj 1/1 Running 0 18m pod/cluster1-dc1-default-sts-0 2/2 Running 0 18m pod/cluster1-dc1-default-sts-1 2/2 Running 0 18m pod/cluster1-dc1-default-sts-2 1/2 CrashLoopBackOff 6 7m26s NAME READY AGE statefulset.apps/cluster1-dc1-default-sts 2/3 18m
The statefulset try to roll out the new image to pod #2 but it fails!
And the operator status is
> kubectl -n cass-operator describe cassdc ... Status: Cassandra Operator Progress: Updating ...
So even though I try to fallback (apply a old config without a failing image), the operator is "stuck" in Updating state and will not "feed" the old (working) image down to the statefulset...
> kubectl -n cass-operator describe statefulset |grep Image Image: datastax/cass-config-builder:1.0.0 <--- Init container Image: cassandra:3.11 <--- non-working image - not "fallback:ed" Image: busybox <--- I'm using a sidecar image as well, that's why its 2 containers/pod
So HOW can I force the operator to ignore its updating state and feed the correct image to the statefulset?
I want to do this without deleting the statefulset (then all pods will be deleted and traffic will be lost).