Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

vkayanala_42513 avatar image
vkayanala_42513 asked ·

One bad Cassandra node causing Local_quorum failure for both writes and reads.

Hi,

One bad Cassandra node causing Local_quorum failure for both Writes and reads.

- Caused by WriteTimeoutException: Cassandra timeout during write query at consistency LOCAL_QUORUM (2 replica were required but only 0 acknowledged the write)
- Caused by ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 0 replica responded)

After bad node taken down (dse services stopped) application running without any issues.

Can you educate me, what caused this timeouts why one bad node causing all this?


Thanks in advance.

cassandra
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Russell Spitzer avatar image
Russell Spitzer answered ·

The errors say that 0 responses were received. This can be many things because in networking we can't tell the difference between "late" and "dead".

Assuming RF 3,

It can mean the remaining two nodes were overloaded, and could not respond to the request from the coordinator
The two nodes were responding but the responses were lost or were too late
The the nodes were down


So if this began when a single node went down, it could be that the cluster was at capacity for 3 nodes worth of load. When one node went down the amount of traffic overwhelmed the last 2 nodes for some requests.

If this happens with a single request (no possible overload issues) it is most likely related to the replication of the keyspace. It may be using a strategy which does not correctly configure replicas in the datacenter being used with "Local".

1 comment Share
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

vkayanala_42513 avatar image vkayanala_42513 ·

Yeah, we are using RF 3.

Things are getting better after bad node taken down, and there wasn't a huge load on these servers.

And it's not just about a single request, it seems like all queries effected during that time.

Thanks.

0 Likes 0 · ·