question

rakshit.sourabh21_99595 avatar image
rakshit.sourabh21_99595 asked Erick Ramirez answered

Why is there no data displayed on the Grafana dashboard when using MCAC?

i am using MCAC on apache cassandra but for condensed graph not getting any metrics .am i missing something ??? was i supposed to import any lib of collectd as it already comes with datastax-mcac-agent-0.1.13.tar.gz (cassandra related metrics are not avialable in condensed metric dashboard and system dashboard.

1613470867987.png

monitoringmetrics collector
1613470867987.png (368.1 KiB)
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

There are several possible reasons why there is no data displayed in the Grafana dashboards. You need to make sure that the agents are configured and operational, plus metrics are exported to the Prometheus server.

Agent configuration

You need to check that the agents are running on the Cassandra nodes. Confirm that the collector daemon is listening on port 9103 (default) using netstat or lsof. Here are example outputs on my Cassandra node:

$ netstat -tlnp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:7199          0.0.0.0:*               LISTEN      19138/java          
tcp        0      0 127.0.0.1:36007         0.0.0.0:*               LISTEN      19138/java          
tcp        0      0 10.101.33.36:9042       0.0.0.0:*               LISTEN      19138/java          
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 10.101.33.36:7000       0.0.0.0:*               LISTEN      19138/java          
tcp6       0      0 :::9103                 :::*                    LISTEN      19216/collectd_wrap 
tcp6       0      0 :::22                   :::*                    LISTEN      -
$ sudo lsof -nPi -sTCP:LISTEN
COMMAND     PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd-r   996 systemd-resolve   13u  IPv4  18430      0t0  TCP 127.0.0.53:53 (LISTEN)
sshd       1576            root    3u  IPv4  24615      0t0  TCP *:22 (LISTEN)
sshd       1576            root    4u  IPv6  24617      0t0  TCP *:22 (LISTEN)
java      19138          ubuntu   84u  IPv4  54039      0t0  TCP 127.0.0.1:7199 (LISTEN)
java      19138          ubuntu   85u  IPv4  54042      0t0  TCP 127.0.0.1:36007 (LISTEN)
java      19138          ubuntu  116u  IPv4  54136      0t0  TCP 10.101.33.36:7000 (LISTEN)
java      19138          ubuntu  126u  IPv4  54143      0t0  TCP 10.101.33.36:9042 (LISTEN)
ld-2.23.s 19216          ubuntu    3u  IPv6  54155      0t0  TCP *:9103 (LISTEN)

Monitoring server

On your Prometheus server, confirm that your nodes are listed as targets with status UP in the web UI. You can check by browsing to http://your_prometheus_server_ip:9090/targets. If your Cassandra nodes are not listed as endpoints for MCAC (or are not showing as UP), you need to investigate the agents on the nodes.

If the nodes are UP, click on the endpoint URL (for example http://node_ip:9103/metrics) and confirm that metrics are being reported to Prometheus. Here's an example:

# HELP collectd_collectd_cache_size write_prometheus plugin: 'collectd' Type: 'cache_size', Dstype: 'gauge', Dsname: 'value'
# TYPE collectd_collectd_cache_size gauge
collectd_collectd_cache_size{collectd="cache",instance="10.101.33.36",cluster="MCAC test",dc="datacenter1",rack="rack1"} 4327 1613542533085
# HELP collectd_collectd_derive_total write_prometheus plugin: 'collectd' Type: 'derive', Dstype: 'derive', Dsname: 'value'
# TYPE collectd_collectd_derive_total counter
collectd_collectd_derive_total{collectd="write_queue",type="dropped",instance="10.101.33.36",cluster="MCAC test",dc="datacenter1",rack="rack1"} 0 1613542533085
...

If these aren't working, it indicates that the Prometheus server is not configured correctly. You need to make sure you're using the prometheus.yaml and tg_mcac.json files from here. If you need assistance with Prometheus, you can get support from https://prometheus.io/community/.

Visualisation

Grafana gets the metrics data from Prometheus. You need to add the Prometheus URL http://your_prometheus_server_ip:9090 as the data source.

If you need assistance with Grafana, you can get support from https://community.grafana.com/. Cheers!

Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.