I am reading a table from cassandra usign pyspark program and then writing data into aws s3 bucket . It seems spark job is taking so much of time reading data from cassandra .
What could be possible reasons for that ? and what are configurations settings I should check in order to have fast read ?
In your previous question (#10306), you stated that nodes are overloaded as a result of your PySpark app.
This is a good place to start. As Jaroslaw Grabowski suggested, you need to look at tuning the read parameters so as not to overload the nodes.
Ultimately, you will need to review the performance of your cluster including the throughput of the disks and the number of nodes. There's a good chance that you're hitting the upper limit of your cluster and will need to add more nodes to increase its capacity. Cheers!
7 People are following this question.
