If I create a table in spark-sql that is back by a cassandra table:
CREATE TABLE my_spark_table USING org.apache.spark.sql.cassandra OPTIONS ( keyspace "my_keyspace", table "my_cassandra_table", pushdown "true");
Would it be possible to call cassandra udfs in spark, for example:
select my_cassandra_udf(column_a) from my_spark_table;
No, but you could use Spark UDFs and UDAF to achieve the same goal.
https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-create-function.html
https://spark.apache.org/docs/3.0.0-preview/sql-getting-started#aggregations
The quick answer is you can call a user-defined function (UDF) in the Spark connector using FunctionCallRef (SPARKC-280). Unfortunately, there isn't a lot of detail on it so I'm not sure if you can call it from Spark SQL.
There's this example from the User Defined Aggregations with Spark blog post:
val result = sensorsAndDaysRDD.
map( _._1 ).
joinWithCassandraTable("udfdemo","measurements",
SomeColumns("sensor",
"time_bucket",
FunctionCallRef("temp_avg", Seq(Right("temp")), Some("avg_temp"))))
I'm going to reach out to the Analytics devs in DataStax and get them to respond on this ticket. Cheers!
5 People are following this question.
