question

When using datasets and an In Clause, you should not use caching. We talked about this a little in the past thread but if you are doing a single filtering step, it is not important to do a caching. Caching is only for repeated uses of the same RDD.

Secondary indexes may be acceptable on Scylla, which you were using in the logs you previously submitted. If this is the case I would try an approach without spark at all since it doesn't seem like you actually need Spark for a Scatter Gather approach. For this I would just issue all the requests Async and do a AwaitAll to wait for the results.

If Secondary Indexes aren't acceptable then I would have a hard time believing that you can do better than a single pass looking up all your keys at the same time. This would be similar code in RDD's if we are avoiding the pushdown. rdd.filter( ParentHash.contains(row.getCol("key")) or something like that. I am not sure what you mean about ordering since sets are not intrinsically ordered in Scala and i'm pretty sure you would lose ordering on combining the sets. The best you could get is a SortedSet which if sorted on the ParentHash would be the same as if you looked up the records in order more or less. If that isn't acceptable you can always index the records you retrieve based on the ordinal of the parent used to look them up and sort afterwards.

Again this is difficult to tell from your question since you are asking for many various things at once and I would recommend you make small focused questions if possible, but it does sound like perhaps youare looking up records based on a Primary Key at some point? For this you would want to use one of the custom cassandra joins in the connector. Methods like `joinWithCassandraTable`.

All of this assumes you are working with a lot of data and it does not sound like this actual result set is ever that large, so Spark may not actually be helpful here.

In summation my approaches would be (Psudo Code below),

1) Are secondary indexes good in Scylla? Do Async Requests

Futures.awaitAll(
  for (parent <- parents ) yield { 
    session.executeAsync("SELECT from Child ...")  
  }
) // This will return back the results of all these queries in order

2) Are secondary indexes bad in Scylla? Full Table Scan non Cache unless repeated Scan's are used.

indexOfParent = parents.zipWithIndex.swap.map // I want a map of parentId -> order
rdd.flatMap( row => parents.find(row.get("parent")).map( parent => (indexOfParent, row))

// DataFrame version, you would have to do an index lookup for this as well but I haven't written it out
df.filter(col(parent).in("parents")