question

gpss1979 avatar image
gpss1979 asked gpss1979 commented

Java Driver Timeouts

Hi All,

I need your assistance in better understanding how the Java driver calculates timeouts.

I use Apache Cassandra 3.11.4 and the latest Datastax Java driver (4.8).

I see the following timeout exceptions:

com.datastax.oss.driver.api.core.DriverTimeoutException Query timed out after 2 seconds.

I found that the default timeout is 2 seconds (for reads?), however, from Cassandra’s point of view, the highest latency I see is 4-5 milliseconds.

So I’m trying to understand how the driver calculates its timeout in relation to Cassandra’s. I believe that it’s different from the read_request_timeout_in_ms or the request_timeout_in_ms in the Cassandra yaml file, right?

Thanks a lot :-)

java driver
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

2 Answers

smadhavan avatar image
smadhavan answered gpss1979 commented

@gpss1979, you're correct that the Java Driver timeout is different from the server timeout properties that are set in the server-side configuration.

Since you're using the DataStax Java Driver 4.8 version, you could tweak the datastax-java-driver.basic.request.timeout value which by default is 2 seconds.

There are also other resources that quite helpful in this regard,

Hope that helps!

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

gpss1979 avatar image gpss1979 commented ·

@smadhavan and @Erick Ramirez.

Thanks a lot for your assistance. It's really helpful! :)

0 Likes 0 ·
Erick Ramirez avatar image
Erick Ramirez answered gpss1979 commented

To add to Madhavan's response, a DriverTimeoutException is a specific timeout exception that gets thrown by the driver when the coordinator doesn't reply to the driver's CQL request within the driver timeout period (client-side configuration). The error message plus the full stack trace should give you clues to the cause.

In contrast, server-side timeouts would be the coordinator returning exceptions such as ReadTimeoutException or WriteTimeoutException. These exceptions are based on the timeouts in cassandra.yaml.

If you are constantly seeing driver timeouts in the application logs but can't correlate it with the latency histograms on the nodes (nodetool proxyhistograms outputs, for example), there's a chance that network latency or connectivity might be causing the driver to not get a response back from the coordinator in time.

Otherwise, validate the CQL queries to see if the application is possibly sending large requests which take too long to complete. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

gpss1979 avatar image gpss1979 commented ·

Hi @Erick Ramirez

I fail to understand one thing. Why the default timeout of the client is 2 seconds, whereas the default read timeout of the coordinator is 5 seconds? Shouldn't the client have a timeout of at least 5 seconds as well?

0 Likes 0 ·
gpss1979 avatar image gpss1979 commented ·

One more question that came to mind. Could it be that the driver gets timed out because of excessive GC behavior or a high CPU load on the client itself, regardless of Cassandra or the query?

0 Likes 0 ·