Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

kallmekunal_131749 avatar image
kallmekunal_131749 asked Erick Ramirez answered

Getting DriverTimeoutException after upgrading from Java driver 3.5 to 4.9

I have upgraded from DSE 3.5 to 4.9to my product. Have noticed dip in microservices capabilities. the tps is down by 15 percent. after a certain level getting

com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2S
    at com.datastax.oss.driver.api.core.DriverTimeoutException.copy(DriverTimeoutException.java:34)
    at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
    at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
    at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProces

Increased the timeout to 30 sec experimentally from 2 sec default.

then the timeout vanishes as expected but server at times giving service unavailable 503 to the client.Seems something is oveburdening it.

Driver configuration:

final DriverConfigLoader configLoader = DriverConfigLoader.programmaticBuilder()
        .withString(DefaultDriverOption.REQUEST_CONSISTENCY, ConsistencyLevel.LOCAL_QUORUM.name())
        .withString(DefaultDriverOption.RECONNECTION_POLICY_CLASS, ExponentialReconnectionPolicy.class.getName())
        .withInt(DefaultDriverOption.RECONNECTION_BASE_DELAY, baseReconnectionIntervalSec * 1000)
        .withInt(DefaultDriverOption.RECONNECTION_MAX_DELAY, maxReconnectionIntervalSec * 1000)
        .withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofSeconds(30))//Test changes for driver timeout
        .withString(DefaultDriverOption.RETRY_POLICY_CLASS, CassandraCustomRetryPolicyXXXX.class.getName()).build();
java driver
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

The queries timing out is indicative that you have queries which are expensive. An example of this is when retrieving multiple partitions using the IN operator with multiple keys.

Previously, the default client-side timeout in Java driver 3.5 was 12 seconds. In Java driver 4.9, this was reduced to 2 seconds because it is generally accepted that queries taking longer than 2 seconds is too long except for very limited cases.

The reason for the lower timeout threshold is for the driver to give up on the query in a quicker time and free up the connection to be used by other requests.

When you increase the timeout to an excessively high value like 30 seconds, the driver will keep the connection open and will wait a long time for the response. If your microservices are making multiple expensive requests, you get into a situation where the connections are held up by the expensive queries and your apps are waiting for connections to free up instead of getting a timeout.

As a side note, we recommend configuring the driver options using the reference configuration file. Using a configuration file instead of hard-coding values in your apps means that you don't need to recompile your app every time you change a driver option. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.