question

The input.split.size_in_mb property is a notional target for the Spark connector to split the Cassandra data into Spark partitions. As I explained to you in question #11500, the calculations are based on estimates so they will never be exact.

In reality, the Spark partitions will never be 64MB in size (size_in_mb) unless the Cassandra partitions are exactly the same. Spark doesn't read the table and stop reading when it's reached 64MB.

As an example let's say that based on the estimated table size and estimated number of C* partitions, each Spark partition is estimated to map to 200 token ranges in Cassandra.

For the first Spark partition, the token range might only contain 2 Cassandra partitions of size 3MB and 15MB so the actual size of the data in Spark partition is just 18MB.

But in the next Spark partition, the token range contains 28 Cassandra partitions that are mostly 1 to 4MB but there is one partition that is 56MB. The total size of this Spark partition ends up being a lot more than 64MB.

If you happen to have really large partitions, they would skew the size of the Spark partitions. For example, your table might have 99% of partitions with 3 to 10MB but 1% of partitions are larger than 250MB. Every time Spark retrieves data from a token range which has a large partition, the Spark partition will be larger than 250MB.

Note that Cassandra partitions are not distributed across multiple Spark partitions -- a Cassandra partition can only belong to one Spark partition. If you have a really bad case where you have large partitions which do not fit in the executor memory, the task will run into out-of-memory errors.

There isn't a straightforward way to deal with large partitions. It requires deep knowledge of the data and isn't something that can be answered in a Q&A forum.

Russell Spitzer who is one of the authors of the Spark connector provides some advice in his blog post which might interest you -- Dealing with large partitions. Cheers!