Bringing together the Apache Cassandra experts from the community and DataStax.
Want to learn? Have a question? Want to share your expertise? You are in the right place!
Not sure where to begin? Getting Started
I'm using
spark 3.0.2 scala 2.12.10 cassandra 3.11.2 spark-cassandra-connector 3.0.0
it's unable to see my keyspaces even it's there
the error is changable !
- can't see this keyspace, Table or view not found:
- query 'SELECT * FROM system_schema.columns' timed out after PT5S
if it's a spark/cassandra application do I need to include these :
spark-cassandra-connector-driver and spark-cassandra-connector-assembly
I don't understand what's the difference here
spark-cassandra-connector-assembly is a fat jar version of the main artifact. It includes all of the SCC depedencies. It was introduced to shade some of the classes that caused classpath conflicts in cloud environments.
spark-cassandra-connector is the main artifact
spark-cassandra-connector-driver is one of the dependencies of the main artifact
When the main artifact is provided to your application as a maven coordinate, Spark downloads and includes all of the dependencies.
When the main artifact is provided to your application as a filesystem path, Spark doesn't download dependencies and you have to include them manually (including the driver).
When the assembly artifact is provided to your application (either way) all of the needed dependencies are provided to your app.
I suspect that you don't have a connection to your C* instance. We can't help further without a full output log.
Cheers
@jaroslaw.grabowski_50515
I'm preparing the dependency on a deployment server manually for the cassandra spark connector
so I need to copy all the
which comes with
Only compile dependencies. But note that you need the whole dependency tree (you also need dependencies of the dependencies).
If I understand correctly you don't want to rely on Spark to resolve the SCC and it's dependencies. In this you may use spark-cassandra-connector-assembly. It should be very convenient in your case. It's just a single jar with all the needed classes. You don't have to worry about the dependencies at all.
after using spark-cassandra-connector-assembly.
I could retrieve from cassandra via
val res = CassandraConnector(conf).withSessionDo { session =>
session.execute("select * From test.t5")
}
but got can't see any keyspaces via spark.read.format
val typeData = spark.read.format("org.apache.spark.sql.cassandra")
.options(Map( "table" -> t5, "keyspace" -> test))
.load().
6 People are following this question.
