Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

vkayanala_42513 avatar image
vkayanala_42513 asked ·

Is querying system.local for Cassandra ring health check right or wrong?

Hello,

Is querying system.local for Cassandra ring health check right or wrong? And how will be the system.local table performance will be?

https://github.com/spring-projects/spring-boot/blob/master/spring-boot-project/spring-boot-actuator/src/main/java/org/springframework/boot/actuate/cassandra/CassandraHealthIndicator.java#L38

- More importantly, there's no way to tell Spring via configuration to not include it.

We have been noticing system.local taking more than a 1 second to reply. Can we know where it is slowing down and reasons for it?

Thanks in Advance.

-Varun

cassandramonitoring
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

alexandre.dutra avatar image
alexandre.dutra answered ·

Hi, querying system.local is usually very fast, but indeed we cannot guarantee that such a query will always be fast.

While Erick is right when he says that this is a good way to check the availability of one node, this is definitely NOT the right thing to do to check the availability of the cluster as a whole.

As a matter fo fact, we are right now rethinking Spring Boot's health check mechanism, see JAVA-2472. The idea is to replace that check by a token ring inspection. This would give us a much more accurate picture of the cluster health, and has the advantage of not requiring executing any query.

This enhanced health check should be released soon.

In the meanwhile:

there's no way to tell Spring via configuration to not include it

Actually there is: add this line to your application.properties:

management.health.cassandra.enabled=false


2 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@alexandre.dutra I think the thought is that we lose any healthcheck if we disable it, no?

0 Likes 0 · ·

The property

management.health.cassandra.enabled

is only for Cassandra, it does not disable other health checks.

2 Likes 2 · ·
Erick Ramirez avatar image
Erick Ramirez answered ·

@vkayanala_42513 Querying the system.local table is a good way to do an end-to-end check on the availability of a node:

  • confirms that the services are operational
  • node is listening for requests from clients
  • confirms reads are working for a specific node because the table is local to each node

It is also very performant because there is only ever 1 partition in the table (partition key local). And for what it's worth, one of its authors @alexandre.dutra is one of the leading engineers for drivers and tools here at DataStax.

If the check is slow, it might indicate a problem with read latency on the node (it's busy serving lots of requests) or the network between your app servers(s) and the cluster is slow so check for those. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.