question

pradeep42.zak_187279 avatar image
pradeep42.zak_187279 asked Erick Ramirez edited

Cassandra metrics are not available in Prometheus

Dear friends,

I'm trying to configure prometheus , grafana based monitoring for cassandra cluster.

Prometheus and grafana are successfully installed on standalone machine.

I have configured jmx exported on all cassandra ndoes. Whoever cassandra nodes are showing down on prometheus dashboard.

DId someone experience same issue while configuring the prometheus , grafana based monitoring for cassandra cluster?

Any help will be highly appreciated.

Thanks,

Pradeep

monitoring
3 comments
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Aleks Volochnev avatar image Aleks Volochnev ♦ commented ·

First I'd check a simple network access: can Prometheus machine reach any of the Cassandra nodes? Ping it? Are the ports open? Security groups? These things usually make 90% of issues.

1 Like 1 ·
pradeep42.zak_187279 avatar image pradeep42.zak_187279 Aleks Volochnev ♦ commented ·

Dear Aleks,

Yes prometheus machine is able to reach cassandra nodes on port 7070 / 9042 using telnet commands ( prometheus, grafana and cassandra nodes are configured on same subnet ) . We are trying to fetch the metric with the help of jmx on cassandra

Metrics is not accessible locally from cassandra nodes using curl or elinks.

I am stuck in this issues from more than 45 days :(

Any help will be highly appreciated.

Thanks,

Pradeep

0 Likes 0 ·
pradeep42.zak_187279 avatar image pradeep42.zak_187279 pradeep42.zak_187279 commented ·

just to add, connections which are used for jmx exported are going to close_wait / time_wait status.

e.g. netstat | grep 7070

all the connections are going to close_wait / time_wait status

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

@pradeep42.zak_187279 You will need to check all the components of your environment since 1 configuration mistake means the whole thing doesn't work.

First up, check that you've got the JMX-to-Prometheus exporter agent configured correctly. On each node, you should be able to connect to the metrics with a web browser. Depending on the which JMX exporter you installed and how you configured it, the URL would look like:

http://cassandra_node_ip:8080/metrics

NOTE: The IP and port depends on how you configured the agent.

You mentioned that you couldn't curl the metrics from the nodes themselves. That indicates an issue with the agent configuration. You need to consult the manual for the JMX exporter you installed to work out why it isn't working. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.