question

spoon99 avatar image
spoon99 asked Erick Ramirez answered

How can I configure heartbeat timeout in the spark-cassandra-connector?

Hi, I'm using spark-cassandra-connector and having following exception.

Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: Heartbeat request: timed out after 5000 ms

Want to know how can I configure heartbeat timeout configuration on spark-cassandra-connector.

Thanks!

spark-cassandra-connector
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

steve.lacerda avatar image
steve.lacerda answered steve.lacerda edited

I believe the configuration parameter you're looking for is this:

https://docs.datastax.com/en/developer/java-driver/4.9/manual/core/pooling/#heartbeat

The error being referred to is a driver configuration timeout and not something from the scc.

Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image
Erick Ramirez answered

The quick answer is that you do not change the heartbeat timeout. Doing so will just hide the underlying issue and you will still eventually run into other problems.

The Spark connector uses the Java driver under the hood. Heartbeats are a client-side mechanism similar to the Linux TCP keepalive. It's the driver's way of sending fake traffic on a connection to make sure network devices like firewalls don't close idle sockets.

If a driver fails to get a response from a heartbeat, it either indicates:

  • it lost network connectivity to the nodes in the cluster, even if it was just a temporary network glitch, or
  • the nodes are unresponsive usually because they are overloaded.

You need to review the logs on the Cassandra nodes for clues as to why a heartbeat request timed out and address that issue. The heartbeat itself is not the problem -- it's a symptom of another problem which you need to identify and resolve. Cheers!

Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.