Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

mishra.anurag643_153409 avatar image
mishra.anurag643_153409 asked mishra.anurag643_153409 commented

How do I resolve FrameTooLongException error?

I am running pyspark job to read cassandra data , but it is throwing an error . I am using spark-cassandra-connector-2.3.2_patched.jar

below is error stack trace :

ERROR:root:exception during cassandra table read >>>>>:An error occurred while calling o380.count.: org.apache.spark.SparkException: Job aborted due to stage failure: Task 438 in stage 19.0 failed 4 times, most recent failure: Lost task 438.3 in stage 19.0 (TID 775, ip-10-221-46-130.ec2.internal, executor 13): com.datastax.driver.core.exceptions.FrameTooLongException: Response frame exceeded maximum allowed lengthat com.datastax.driver.core.exceptions.FrameTooLongException.copy(FrameTooLongException.java:48) 

1) Is this issue happening due to connector and I have to upgrade connector version ?

2) if I have to update version , is there some work around to resolve this issue without

upgrading connector ?


spark code:


def read_data_cassandra(spark_session, keyspace, table):
    """ Read Cassandra table

    :param spark_session: Spark session
    :param keyspace: Cassandra keyspace
    :param table: Cassandra table
    :return: DataFrame object
    """
    try:
        data = spark_session.read \
            .format('org.apache.spark.sql.cassandra') \
            .options(table=table, keyspace=keyspace) \
            .load()
        return data

    except Exception as e:
        print("issue reading cassandra table name:", table)
        logging.exception("exception during cassandra table read >>>>>:" + str(e))
        raise e
cassandra_data = read_data_cassandra(spark, c_keyspace, table_name)
cassandra_data.cache()
cassandra_data.count()

while running count() above error is coming .

versions for all the software components used including Cassandra cluster:

cassandra: 3.11.3

spark: 2.4.2

connector : spark-cassandra-connector-2.3.2


spark-cassandra-connector
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered mishra.anurag643_153409 commented

There isn't enough information in your post to determine the cause of the problem.

It is an error generated by the Java driver which the Spark connector uses to connect to Cassandra. It indicates that the response sent back to the driver was too large (the maximum size is 256MB) but it's impossible for us to determine why without additional information.

Please update your original question with the following:

  • minimal Spark app code that reproduces the issue
  • versions for all the software components used including Cassandra cluster
  • any other relevant information needed to reproduce the problem

Cheers!

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@Erick Ramirez I have updated question and added more information .

0 Likes 0 ·