Currently, we have a 3 node Cassandra cluster. we are facing ReadTimeOutException and WriteTimeOutException from few tables thrown by our Cassandra JAVA Driver(3.10.0).
The load on the cluster is low, but also we are getting ReadTimeOuts, write timeouts, we are not getting this frequently, but once in awhile these timeouts were hitting, that's why we thought to increase timeout configuration.
we added query logger to trace the query time, the query is crossing default timeout,
2020-09-15 18:17:21:596*[DEBUG]*cluster1-nio-worker-0*c.d.driver.core.QueryLogger.ERROR*logQuery*[cluster1] [cassandra-0.cassandra.default.svc.cluster.local/x.x.x.x:9042] Query error after 5569 ms: [1 bound values] select * from req_resp_data where id=:id; com.datastax.driver.core.exceptions.ReadTimeoutException
we have deployed our cluster on Kubernetes in Gcp Cloud,
Cassandra version - 3.11.3
Cluster Load - 7-8 GB
Ram - 10 GB
CPU - 6 Core
Disk - 500 GB SSD
Cpu scaling was not enabled.
Coming to consistency from the code side, which consistency is better to fetch (ONE or QUORUM) from code side (Cassandra Dao)
what would be the good level of Read and Write timeout configuration in the YAML file/Code side.
What will be after-effects, after changing this configuration?