Build Cloud-Native apps with Apache Cassandra

GOT QUESTIONS from the Workshop? You're in the right place! Post a question here and we'll get you answers.

Click here for Week 8 Materials and Homework.

Follow us on Eventbrite to get notified when new workshops are scheduled!


question

oded.erner_188979 avatar image
oded.erner_188979 asked ·

Why is my pod in pending status?

Hi,
I'm getting the following:

what should be done to get it out of pending?

its running on ec2 cloud.

$ kubectl -n cass-operator get pods                                                                                                     NAME                             READY   STATUS    RESTARTS   AGE                                       cass-operator-56fcb9ff47-qnvfs   1/1     Running   0          39m                                       cluster1-dc1-default-sts-0       0/2     Pending   0          31m            

[UPDATE]

$ kubectl -n cass-operator describe pods cluster1-dc1-default-sts-0
Name:           cluster1-dc1-default-sts-0                                                                                                                                                                         Namespace:      cass-operator                                                                                                                                                                                      Priority:       0                                                                                                                                                                                                  Node:           <none>                                                                                                                                                                                                                                                                                                                                                                 Status:         Pending                                                                                                                                                                                            IP:                                                                                                                                                                                                                IPs:            <none>                                                                                                                                                                                             Controlled By:  StatefulSet/cluster1-dc1-default-sts                                                                                                                                                               Init Containers:                                                                                                                                                                                                         server-config-init:                                                                                                                                                                                                  Image:      datastax/cass-config-builder:1.0.1                                                                                                                                                                     Port:       <none>                                                                                                                                                                                                 Host Port:  <none>                                                                                                                                                                                                 Environment:                                                                                                                                                                                                         CONFIG_FILE_DATA:  {"cassandra-yaml":{"authenticator":"org.apache.cassandra.auth.PasswordAuthenticator","authorizer":"org.apache.cassandra.auth.CassandraAuthorizer","role_manager":"org.apache.cassandra.auth.CassandraRoleManager"},"cluster-info":{"name":"cluster1","seeds":"cluster1-seed-service"},"datacenter-info":{"name":"dc1"},"jvm-options":{"initial_heap_size":"800M","max_heap_size":"800M"}}                          POD_IP:             (v1:status.podIP)                                                                                                                                                                              RACK_NAME:         default                                                                                                                                                                                         PRODUCT_VERSION:   3.11.6                                                                                                                                                                                          PRODUCT_NAME:      cassandra                                                                                                                                                                                       DSE_VERSION:       3.11.6                                                                                                                                                                                        Mounts:                                                                                                                                                                                                              /config from server-config (rw)                                                                                                                                                                                    /var/run/secrets/kubernetes.io/serviceaccount from default-token-js6zp (ro)                                                                                                                                  Containers:                                                                                                                                                                                                          cassandra:                                                                                                                                                                                                           Image:       datastax/cassandra-mgmtapi-3_11_6:v0.1.5                                                                                                                                                              Ports:       9042/TCP, 8609/TCP, 7000/TCP, 7001/TCP, 8080/TCP                                                                                                                                                      Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP                                                                                                                                                                     Liveness:    http-get http://:8080/api/v0/probes/liveness delay=15s timeout=1s period=15s #success=1 #failure=3                                                                                                    RReadiness:   http-get http://:8080/api/v0/probes/readiness delay=20s timeout=1s period=10s #success=1 #failure=3                                                                                                   Environment:                                                                                                                                                                                                         DS_LICENSE:               accept                                                                                                                                                                                   DSE_AUTO_CONF_OFF:        all                                                                                                                                                                                      USE_MGMT_API:             true                                                                                                                                                                                     MGMT_API_EXPLICIT_START:  true                                                                                                                                                                                     DSE_MGMT_EXPLICIT_START:  true                                                                                                                                                                                   Mounts:                                                                                                                                                                                                              /config from server-config (rw)                                                                                                                                                                                    /var/lib/cassandra from server-data (rw)                                                                                                                                                                           /var/log/cassandra from server-logs (rw)                                                                                                                                                                           /var/run/secrets/kubernetes.io/serviceaccount from default-token-js6zp (ro)                                                                                                                                    server-system-logger:                                                                                                                                                                                                Image:      busybox                                                                                                                                                                                                Port:       <none>                                                                                                                                                                                                 Host Port:  <none>                                                                                                                                                                                                 Args:                                                                                                                                                                                                                /bin/sh                                                                                                                                                                                                            -c                                                                                                                                                                                                                 tail -n+1 -F /var/log/cassandra/system.log                                                                                                                                                                       Environment:  <none>                                                                                                                                                                                               Mounts:                                                                                                                                                                                                              /var/log/cassandra from server-logs (rw)                                                                                                                                                                           /var/run/secrets/kubernetes.io/serviceaccount from default-token-js6zp (ro)                                                                                                                                  Conditions:                                                                                                                                                                                                          Type           Status                                                                                                                                                                                              PodScheduled   False                                                                                                                                                                                             Volumes:                                                                                                                                                                                                             server-data:                                                                                                                                                                                                         Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)                                                                                                                   ClaimName:  server-data-cluster1-dc1-default-sts-0                                                                                                                                                                 ReadOnly:   false                                                                                                                                                                                                server-config:                                                                                                                                                                                                       Type:       EmptyDir (a temporary directory that shares a pod's lifetime)                                                                                                                                          Medium:                                                                                                                                                                                                            SizeLimit:  <unset>                                                                                                                                                                                              server-logs:                                                                                                                                                                                                         Type:       EmptyDir (a temporary directory that shares a pod's lifetime)                                                                                                                                          Medium:                                                                                                                                                                                                            SizeLimit:  <unset>                                                                                                                                                                                              default-token-js6zp:                                                                                                                                                                                                 Type:        Secret (a volume populated by a Secret)                                                                                                                                                               SecretName:  default-token-js6zp                                                                                                                                                                                   Optional:    false                                                                                                                                                                                             QoS Class:       BestEffort                                                                                                                                                                                        Node-Selectors:  <none>                                                                                                                                                                                            Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s                                                                                                                                                                    node.kubernetes.io/unreachable:NoExecute for 300s                                                                                                                                                 Events:                                                                                                                                                                                                              Type     Reason            Age        From               Message                                                                                                                                                   ----     ------            ----       ----               -------                                                                                                                                                   Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "cluster1-dc1-default-sts-0": pod has unbound immediate PersistentVolumeClaims
Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "cluster1-dc1-default-sts-0": pod has unbound immediate PersistentVolumeClaims     
workshopcass-operator
6 comments
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

[Converted to comment, content posted in original question]

0 Likes 0 · ·

[Converted to comment, content posted in original question]

0 Likes 0 · ·

[Converted to comment, content posted in original question]

0 Likes 0 · ·

[Converted to comment, content posted in original question]

0 Likes 0 · ·

Hi @oded.erner_188979,

your post was seemingly forwarded to moderation queue by spam filter, it does this sometimes. I will delete your answers, they seem to be attempts of repeated posts, and not answers. Hope that's ok!

0 Likes 0 · ·
oded.erner_188979 avatar image oded.erner_188979 bettina.swynnerton ♦♦ ·

sure - but please leave just one of them which is enough.
thanks

0 Likes 0 · ·
jim.dickinson_187342 avatar image
jim.dickinson_187342 answered ·

You should describe that pod and the persistentvolumeclaim (pvc) for it for more information. There are a few different reasons why Kubernetes wouldn't be able to schedule the pod.

3 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

hi, I updated the thread with get pods.

thanks.

0 Likes 0 · ·

Great - now we know it's strictly about

pod has unbound immediate PersistentVolumeClaims

Can you describe the pvc as well?

0 Likes 0 · ·

@oded.erner_188979 There's something strange that happens when you're pasting the outputs here, as if the underlying text is getting re-formatted and it's triggering the spam detector on this site.

If you wouldn't mind for next time, please upload large amounts of text to a site like https://gist.github.com/ then post the URL to your Gist here. Cheers!

0 Likes 0 · ·
Cedrick Lunven avatar image
Cedrick Lunven answered ·

Hi @oded.erner_188979

Root cause here is Kubernetes cannot mount any disk to the pod, multiple root causes:

How did you create the `storage Class` in the namespace `cass-operation` ?

Could you show us the a get storageClass and describe server-storage created in first part.

Also what about the storage, do you have enough space remaining ?

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Hi Cedric,

My ec2 machine got erased.
I got a new one up and running.
This time all went fine.

Thanks

1 Like 1 · ·