question

sharathsai666_167144 avatar image
sharathsai666_167144 asked sharathsai666_167144 edited

Changing Read and WriteTimeout configuration

Currently, we have a 3 node Cassandra cluster. we are facing ReadTimeOutException and WriteTimeOutException from few tables thrown by our Cassandra JAVA Driver(3.10.0).

The load on the cluster is low, but also we are getting ReadTimeOuts, write timeouts, we are not getting this frequently, but once in awhile these timeouts were hitting, that's why we thought to increase timeout configuration.
we added query logger to trace the query time, the query is crossing default timeout,

2020-09-15 18:17:21:596*[DEBUG]*cluster1-nio-worker-0*c.d.driver.core.QueryLogger.ERROR*logQuery*[cluster1] [cassandra-0.cassandra.default.svc.cluster.local/x.x.x.x:9042] Query error after 5569 ms: [1 bound values] select * from req_resp_data where  id=:id;
com.datastax.driver.core.exceptions.ReadTimeoutException


we have deployed our cluster on Kubernetes in Gcp Cloud,
Cassandra version - 3.11.3
Cluster Load - 7-8 GB
Ram - 10 GB
CPU - 6 Core
Disk - 500 GB SSD

Cpu scaling was not enabled.
Coming to consistency from the code side, which consistency is better to fetch (ONE or QUORUM) from code side (Cassandra Dao)
what would be the good level of Read and Write timeout configuration in the YAML file/Code side.
What will be after-effects, after changing this configuration?

java driver
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

bettina.swynnerton avatar image
bettina.swynnerton answered sharathsai666_167144 edited

Hi @sharathsai666_167144,

If you are getting a ReadTimeoutError, the following happened:

A read request reached the coordinator, which initially believed that there were enough live replicas to process it. But, for some reason, one or several replicas were too slow to answer within the predefined timeout (read_request_timeout_in_ms in cassandra.yaml), and the coordinator replied to the client with a READ_TIMEOUT error.
This could be due to a temporary overloading of these replicas, or even that they just failed or were turned off. During reads, Cassandra doesn’t request data from every replica to minimize internal network traffic; instead, some replicas are only asked for a checksum of the data. A read timeout may occur even if enough replicas responded to fulfill the consistency level, but only checksum responses were received (the method’s dataRetrieved parameter allow you to check if you’re in that situation).
If the policy rethrows the error, the user code will get a ReadTimeoutException.

see the relevant driver docs: https://docs.datastax.com/en/developer/java-driver/3.10/manual/retries/

By default, the read_request_timeout_in_ms is set to 5 seconds in the cassandra.yaml. You can of course increase the timeout values, but this would only hide the problem. 5 seconds is already a very long timeout for a read request, and your error more likely indicates that your cluster is under-resourced.

It seems that your query is a direct lookup by primary key:

select * from req_resp_data where  id=:id

Unless your nodes are seriously overloaded and unresponsive, for example due to excessive GC pauses, this should not come anywhere close to 5 seconds.

I can see that there is only very little load on the cluster, however I am concerned about the low amount of RAM available on the cluster. Is this 10GB RAM for the whole cluster, or per node? What is the heap per node?

Check into the system.log of your cassandra nodes to see what server side issues are logged when you are getting the read timeouts - I would check for GC pauses and other pauses first.

I hope this gives you some ideas where to look next.

Re the question on consistency: What is your replication factor for these tables?

4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

sharathsai666_167144 avatar image sharathsai666_167144 commented ·

Thanks, @bettina.swynnerton

Above provided ram, disk, CPU is per node.
Our Replication factor is 3

heap is 1/4 of Ram

we have observed system.log on that particular day we didn't find any error logs.
coming to GC log we find below log after our ReadTimeoutException. only one GC log is generated throughout the whole day.

2020-09-15 18:17:22:669*[WARN]*GossipTasks:1*o.a.c.g.FailureDetector*interpret*Not marking nodes down due to 
local pause of 6890653503 > 5000000000
0 Likes 0 ·
sharathsai666_167144 avatar image sharathsai666_167144 sharathsai666_167144 commented ·

Hi @bettina.swynnerton, can I get your inputs

0 Likes 0 ·
bettina.swynnerton avatar image bettina.swynnerton ♦♦ sharathsai666_167144 commented ·

Hi @sharathsai666_167144,

sorry, I missed your earlier comment.

The message you see Not marking nodes down due to local pause is due to the JVM pausing. The cassandra process recognises this as an externally caused pause, hence not marking nodes down.

It is normal to see these FailureDetector pauses after a restart once per node, so double check that you are not seeing a false warning here.

Other than the one expected pause during startup, long GC pauses can also trigger these FailureDetector pauses, but if you do not observe other GC pauses in your system.log, then it might be a pause due to some other factors. Only pauses longer than 5 seconds are being reported, so it is possible that you are experiencing more pauses than are being reported in the logs.

You might want to check with GCP if your instances are being paused during low load, I have seen this before.

In this particular error message, You have a pause of 6.8 seconds, so this exceeds the timeout and would cause this error.



0 Likes 0 ·
Show more comments