Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

mishra.anurag643_153409 avatar image
mishra.anurag643_153409 asked ·

Can you please explain spark.cassandra.input.readsPerSec?

I want to less overload cassandra nodes , in order so I am assuming spark.cassandra.input.readsPerSec will solve this problem . I am reading cassandra table from spark . In order to overload less cassandra node , is this property enough to solve the problem .

Can someone please illustrate this with example and values ?

spark-cassandra-connector
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

The readsPerSec property puts a limit on the maximum number of read requests a Spark app ca perform per second.

For example, if you set it to 10:

--conf spark.cassandra.input.readsPerSec=10

then only 10 read requests will run every second. By default read requests are not limited so Spark will send as many requests as it needs to unless you throttle it. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.