Hello, I have some issues deploying Cassandra on my cluster using the operator. "Warning FailedScheduling default-scheduler pod has unbound immediate PersistentVolumeClaims". I use the same StorageClass to deploy Cassandra manually, and it works. Any ideas?
K8s version: 1.13
Storageclass provider: netapp.io/trident
kdp cluster1-dc1-default-sts-0 -n cass-operator
Name: cluster1-dc1-default-sts-0
Namespace: cass-operator
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: app.kubernetes.io/managed-by=cass-operator
cassandra.datastax.com/cluster=cluster1
cassandra.datastax.com/datacenter=dc1
cassandra.datastax.com/node-state=Ready-to-Start
cassandra.datastax.com/rack=default
controller-revision-hash=cluster1-dc1-default-sts-6dcbf85bb5
statefulset.kubernetes.io/pod-name=cluster1-dc1-default-sts-0
Annotations: <none>
Status: Pending
IP:
Controlled By: StatefulSet/cluster1-dc1-default-sts
Init Containers:
server-config-init:
Image: datastax/cass-config-builder:1.0.1
Port: <none>
Host Port: <none>
Environment:
CONFIG_FILE_DATA: {"cassandra-yaml":{"authenticator":"org.apache.cassandra.auth.PasswordAuthenticator","authorizer":"org.apache.cassandra.auth.CassandraAuthorizer","role_manager":"org.apache.cassandra.auth.CassandraRoleManager"},"cluster-info":{"name":"cluster1","seeds":"cluster1-seed-service"},"datacenter-info":{"name":"dc1"},"jvm-options":{"initial_heap_size":"800M","max_heap_size":"800M"}}
POD_IP: (v1:status.podIP)
RACK_NAME: default
PRODUCT_VERSION: 3.11.6
PRODUCT_NAME: cassandra
DSE_VERSION: 3.11.6
Mounts:
/config from server-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mv8lz (ro)
Containers:
cassandra:
Image: datastax/cassandra-mgmtapi-3_11_6:v0.1.5
Ports: 9042/TCP, 8609/TCP, 7000/TCP, 7001/TCP, 8080/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
Liveness: http-get http://:8080/api/v0/probes/liveness delay=15s timeout=1s period=15s #success=1 #failure=3
Readiness: http-get http://:8080/api/v0/probes/readiness delay=20s timeout=1s period=10s #success=1 #failure=3
Environment:
DS_LICENSE: accept
DSE_AUTO_CONF_OFF: all
USE_MGMT_API: true
MGMT_API_EXPLICIT_START: true
DSE_MGMT_EXPLICIT_START: true
Mounts:
/config from server-config (rw)
/var/lib/cassandra from server-data (rw)
/var/log/cassandra from server-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mv8lz (ro)
server-system-logger:
Image: busybox
Port: <none>
Host Port: <none>
Args:
/bin/sh
-c
tail -n+1 -F /var/log/cassandra/system.log
Environment: <none>
Mounts:
/var/log/cassandra from server-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mv8lz (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
server-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: server-data-cluster1-dc1-default-sts-0
ReadOnly: false
server-config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
server-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-mv8lz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mv8lz
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 114s (x190 over 112m) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 7 times)
here the storageclass:
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
name: server-storage
parameters:
backendType: ontap-nas-economy
encryption: "true"
media: ssd
provisioningType: thin
provisioner: netapp.io/trident
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
Deploying manually Cassandra with the same StorageClass works. It generates new PVCs and bound to new PVs.
✦2 ➜ kgp NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 20h spectrum-77f9f76c58-gwstc 1/1 Running 0 31h spectrum-cassandra-0 1/1 Running 0 3d7h spectrum-cassandra-1 1/1 Running 0 3d7h spectrum-cassandra-2 1/1 Running 0 3d7h spectrum-elasticsearch-0 1/1 Running 0 3d21h
Here the Statefulset:
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: spectrum-cassandra
labels:
app: spectrum-cassandra
spec:
serviceName: spectrum-cassandra
replicas: 3
template:
metadata:
labels:
app: spectrum-cassandra
zone: bb-hsec
spec:
containers:
- name: spectrum-cassandra
imagePullPolicy: IfNotPresent
image: spectrum-cassandra:latest
ports:
- containerPort: 7000
name: internal
- containerPort: 7001
name: internaltls
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
- containerPort: 9160
name: thrift
livenessProbe:
exec:
command:
- /bin/bash
- -c
- nodetool statusgossip |grep "^running$"
initialDelaySeconds: 60
periodSeconds: 20
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 20
readinessProbe:
exec:
command:
- /bin/bash
- -c
- if [[ $(nodetool status | grep $POD_IP) == *"UN"* ]]; then exit 0; else exit 1; fi
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 1
successThreshold: 1
timeoutSeconds: 10
resources:
limits:
cpu: "2000m"
memory: 4Gi
requests:
cpu: "500m"
memory: 2Gi
securityContext:
capabilities:
add:
- IPC_LOCK
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- nodetool drain
envFrom:
- configMapRef:
name: spectrum-cassandra
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: ca-pv-data
mountPath: /cassandra_data
terminationGracePeriodSeconds: 300
volumeClaimTemplates:
- metadata:
name: ca-pv-data
spec:
accessModes:
- ReadWriteOnce
storageClassName: trident-nfs
resources:
requests:
storage: 10Gi
PVC:
✦2 ➜ kg pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ca-pv-data-spectrum-cassandra-0 Bound oneira-sandbox-ca-pv-data-spectrum-cassandra-0-1f88d 10737418240 RWO trident-nfs 3d19h ca-pv-data-spectrum-cassandra-1 Bound oneira-sandbox-ca-pv-data-spectrum-cassandra-1-742ba 10737418240 RWO trident-nfs 3d19h ca-pv-data-spectrum-cassandra-2 Bound oneira-sandbox-ca-pv-data-spectrum-cassandra-2-aa2ad 10737418240 RWO trident-nfs 3d19h es-pv-data-spectrum-elasticsearch-0 Bound oneira-sandbox-es-pv-data-spectrum-elasticsearch-0-4e172 10737418240 RWO trident-nfs 4d2h
