Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

edmundbett_47126 avatar image
edmundbett_47126 asked Erick Ramirez edited

Setting up DSE cluster for the week 5 Metrics Collector is too difficult to get to work

The section 5. Start DSE Cluster of Monitor a Cluster with the Metrics Collector for Week5 is very difficult to get to work -> I have after 14 hours - managed to get cluster2-dc2-default-sts-0 into running status - but cluster2-dc2-default-sts-1 and cluster2-dc2-default-sts-2 are not. I have 32GB of memory so I do not believe that Memory is an issue. All 3 seem to have running status now - after some 14 hours or so. Last night I had to add a command [not specified] -

kubectl -n cass-operator apply -f ./prometheus_grafana/promotheus/instance.yaml

in order to see

prometheus-operated

with

kubectl get svc -n cass-operator --show-labels=true
workshop
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

bettina.swynnerton avatar image bettina.swynnerton ♦♦ ·

Hi @edmundbett_47126,

I agree that 14 hours to get the three DSE pods up and running is a very long time. To set it all up, it took me about one hour, and I am not sure what would explain the 14 hours. Downloading the DSE images took quite a bit of time, and in my case I was CPU bound on my Mac laptop, it was running hot.

To be sure that I understand your post correctly: is all up and running correctly now?

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

It sounds like something has gone really wrong with your setup. It shouldn't take that long to setup. The whole idea with using the cass-operator is to make it simple.

At the time that you experienced the issue, did you run a describe on the problematic pods? Without additional diagnostic information, it would be difficult to determine what happened in your case.

Sorry that it didn't work so well for you. Cheers!

[UPDATE] I forgot to mention in my previous response that this isn't quite right:

Last night I had to add a command [not specified] - 
kubectl -n cass-operator apply -f ./prometheus_grafana/promotheus/instance.yaml

This command is in fact included in step 3b of the week 5 of the Metrics section.

I got really curious with your experience so I spent some time yesterday testing various scenarios. I've determined that a 2-core + 8GB machine is not sufficient to run the Metrics Collector exercise.

Here are some key points:

  • test ran on server with Intel Xeon Gold 6152 CPU @ 2.1GHz
  • 1 Gigabit internet link
  • Prometheus operator installation took 40+ minutes
  • Grafana operator installation took 20+ minutes
  • installation was CPU-bound
  • most time spent pulling down images

In my test, memory did not appear to be an issue despite only having 8GB of RAM.

An attempt to install the DSE cluster with 3 nodes (size: 3) failed. The pods were stuck trying to initialise and the test server was CPU-bound. They never got past pod initialisation after 1 hour.

After deleting the DSE pods, I was able to install a single-node DSE cluster (size: 1) in 4 minutes. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.