question

nkanth_152448 avatar image
nkanth_152448 asked Erick Ramirez commented

Cannot "describe pods", getting "Readiness probe failed: Get http://10.244.1.5:8080/api/v0/probes/readiness"

As part of week 5 exercise I am trying to create a single node cluster in my laptop. Everything seems working till the point of pulling the image and starting the container. However, it is not completing and I am getting the below messages with the command. Can you please help here?

Thanks,

Ravi

> kubectl -n cass-operator describe pods
Warning Unhealthy 54s kubelet, kind-cassandra-worker Readiness probe failed: Get http://10.244.1.5:8080/api/v0/probes/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 37s (x2 over 48s) kubelet, kind-cassandra-worker Liveness probe failed: Get http://10.244.1.5:8080/api/v0/probes/liveness: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Normal Killing 35s kubelet, kind-cassandra-worker Container cassandra failed liveness probe, will be restarted
Warning Unhealthy 12s (x5 over 47s) kubelet, kind-cassandra-worker Readiness probe failed: Get http://10.244.1.5:8080/api/v0/probes/readiness: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 3s kubelet, kind-cassandra-worker Readiness probe failed: Get http://10.244.1.5:8080/api/v0/probes/readiness: dial tcp 10.244.1.5:8080: connect: connection refused
workshopcass-operator
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

Those errors indicate that the service cluster1-dc1-service is not listening on port 8080 and that your Kubernetes cluster isn't operational.

If you try to get the list of services configured, it would probably not show up. Here is a sample output from my cluster:

$ kubectl -n cass-operator get svc
NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
cass-operator-metrics                 ClusterIP   10.102.92.173    <none>        8383/TCP,8686/TCP   8d
cassandradatacenter-webhook-service   ClusterIP   10.107.9.223     <none>        443/TCP             8d
cluster1-dc1-all-pods-service         ClusterIP   None             <none>        <none>              8d
cluster1-dc1-service                  ClusterIP   None             <none>        9042/TCP,8080/TCP   8d
cluster1-seed-service                 ClusterIP   None             <none>        <none>              8d
mycontactpoints                       ClusterIP   10.109.125.238   <none>        30942/TCP           19h

In my experience, a common cause is that your machine doesn't have enough memory to run the cluster. You might be able to get it working with 12GB but we recommend 16GB of RAM minimum so there's enough resources for both the Kubernetes cluster and the Windows operating system. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

nkanth_152448 avatar image nkanth_152448 commented ·

Hi Eric, On my laptop I could see a similar output but no luck. Do you suspect it is all about only memory issue or can we tweak something else? Since when I see memory usage I see the below metrics. Can you please throw some light here?

Thanks,

Ravi


0 Likes 0 ·
1596605005358.png (9.6 KiB)
1596605066701.png (16.7 KiB)
Erick Ramirez avatar image Erick Ramirez ♦♦ nkanth_152448 commented ·

If you're lucky, you might get it to work on a machine with 12GB of memory but in my experience you'll eventually run into other weird issues. Cheers!

0 Likes 0 ·