alfhv avatar image
alfhv asked alfhv commented

How do I publish to Kafka from Spark dataset?

I'm trying to do a spark query (to cassandra) and I want to push the resulted dataset to kafka, basically the code is as follow:

.withColumn("key", col("pk")) // kafka partition key
.withColumn("value", to_json(struct("*"))) // kafka msg content

but I'm having error: - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ClassNotFoundException: Failed to find data source: kafka. Please find packages at] with root cause 
java.lang.ClassNotFoundException: kafka.DefaultSource

I have DSE install nodetool version:
DSE version: 6.7.14

My jar is a uber-jar, I have added also following section in my maven-shade-plugin in pom:

<transformer     implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">

my dependencies are:


I'm doing "dse spark-submit" directly on cluster nodes (for testing purpose for now)

The kafka datasource needs to be installed separatly in cluster ??

Can I achive the same using DSE KafkaConnector ? it is possible to have an example ?

10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered alfhv commented

Unfortunately there is no Kafka connector available where DSE or Cassandra is the source. The only connector available is a sink --

I'm unable to provide you assistance for this issue in a Q&A forum. My suggestion is that you log a ticket with DataStax Support so one of our engineers can assist you directly. Cheers!

1 comment Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

alfhv avatar image alfhv commented ·

I’m ont using Cassandra itself as datasource, my source us a dataset. My code works ok with Spark open source..

The Kafka module (sa me as for Spark open source) is installed with Spark dse ? If not, there is a way to add it ?

0 Likes 0 ·