Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

Ryan Quey avatar image
Ryan Quey asked bettina.swynnerton edited

How can I debug the error "too many continuous paging sessions are already running"?

I'm running a Spark job on my DSE 6.8.0 cluster, and when I do a certain `regexp_replace`, it returns the error `Invalid request, too many continuous paging sessions are already running: 60`

val punctuation = """!"#$%&\'()*+,-./:;<=>?@[\]^_\{|\}\~"""
val descNoPunct = df.withColumn("desc_no_punct", regexp_replace($"description", punctuation, ""))

(The `description` column is TEXT type in Cassandra)

Here is more of the error message, in case it's helpful:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 21.0 failed 10 times, most recent failure: Lost task 0.9 in stage 21.0 (TID 174, 127.0.0.1, executor 2): java.io.IOException: Exception during execution of SELECT "episode_guid", "description" FROM "podcast_analysis_tool"."episodes_by_podcast" WHERE token("podcast_api", "podcast_api_id") > ?   ALLOW FILTERING: Invalid request, too many continuous paging sessions are already running: 60
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.com$datastax$spark$connector$rdd$CassandraTableScanRDD$$fetchTokenRange(CassandraTableScanRDD.scala:349)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD$$anonfun$17.apply(CassandraTableScanRDD.scala:366)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD$$anonfun$17.apply(CassandraTableScanRDD.scala:366)
...

I could give more of my code if it's helpful, but actually my guess is that it's happening because I'm just running this code too many times in a row, without doing any `LIMIT` on it.

I have three related questions:

  1. Is there a way to check how many continuous paging sessions are currently running at a given moment?
  2. Are there up to date docs on continuous paging? (Google searches tend to mostly show this blog post, but it's from three years ago and refers to DSE 5.0, and some points seem to be out of date)
  3. Is there a way in Spark to just tell it to wait rather than adding more continuous paging sessions?
spark-cassandra-connector
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

bettina.swynnerton avatar image
bettina.swynnerton answered bettina.swynnerton edited

Hi @Ryan Quey,

unless you are submitting this job with more than 60 executors to a single node, I would not expect to hit this limit with a healthy Spark job. If continuous paging is enabled, each executor should not request more than one session.

So, I would first focus the debugging effort onto the job itself: Check the executor logs, both stdout and stderr for each executor to see if the job executes without error before it hits this limit.

You can check the logs from the Spark UI, or directly on the DSE nodes. On a DSE package install, you can find the executor logs in the worker directories for the relevant job, i.e. ./var/lib/spark/worker/app-20200521133901-0008

Certain errors can cause an executor to request sessions repeatedly, which then causes this limit to be exceeded. The setting limits the number of concurrent sessions per node.

If the executor logs are error-free, we can enable further debug logging, as mentioned in the blog that you referenced. It was written when the feature introduced, but it is still valid. Let me know where you see it as being out of date.

https://www.datastax.com/blog/2017/04/dse-continuous-paging-tuning-and-support-guide

I would also be happy to look at your Spark job, if you want to update your question.

I hope this helps to give more insight where this is going wrong.


Update based on the discussion in the comments:

The referenced blog post is still mostly valid regarding what continuous paging is and how it works.

But a few things have changed, as was correctly pointed out by @Ryan Quey

Since the release of DSE 6.0, continuous paging is now enabled by default, and it can now be configured via a section in the cassandra.yaml. The setting is only relevant for Spark reads.

See here for the yaml configuration.

The other configuration options through spark-defaults.conf and command line with the --conf option are still valid.

The parameter spark.dse.continuous_paging_enabled has been deprecated as of DSE 6.0, but can still be used, and is automatically replaced with spark.dse.continuousPagingEnabled

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Ryan Quey avatar image Ryan Quey ·

Thanks this is very helpful. I haven't gotten a chance to go through the debugging steps you outlined, but you're right, it is strange to have hit the limit, when I'm only running this job on my local computer (i.e, far less than 60 executors).

The main thing that jumped out at me as being out of date includes that unless I'm mistaken, the article says that continuous paging is disabled by default, whereas for me I am pretty sure I did not turn it on, yet clearly it is in use. It is very possible I did turn it on somewhere along the line though and just forgot, haha!

Thanks again

0 Likes 0 ·
bettina.swynnerton avatar image bettina.swynnerton ♦♦ Ryan Quey ·

Oh yes, you are completely right there! I will raise a documentation request to include more about this feature into the docs.

Cheers!

1 Like 1 ·