question

randas avatar image
randas asked mouna.tayahi_167584 commented

Why can't Spark connector access my keyspaces?

I'm using

spark 3.0.2
scala 2.12.10
cassandra 3.11.2
spark-cassandra-connector 3.0.0

it's unable to see my keyspaces even it's there
the error is changable !
- can't see this keyspace, Table or view not found:
- query 'SELECT * FROM system_schema.columns' timed out after PT5S
if it's a spark/cassandra application do I need to include these :

spark-cassandra-connector-driver and  spark-cassandra-connector-assembly

I don't understand what's the difference here

spark-cassandra-connector
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

jaroslaw.grabowski_50515 avatar image
jaroslaw.grabowski_50515 answered mouna.tayahi_167584 commented

spark-cassandra-connector-assembly is a fat jar version of the main artifact. It includes all of the SCC depedencies. It was introduced to shade some of the classes that caused classpath conflicts in cloud environments.

spark-cassandra-connector is the main artifact

spark-cassandra-connector-driver is one of the dependencies of the main artifact


When the main artifact is provided to your application as a maven coordinate, Spark downloads and includes all of the dependencies.

When the main artifact is provided to your application as a filesystem path, Spark doesn't download dependencies and you have to include them manually (including the driver).

When the assembly artifact is provided to your application (either way) all of the needed dependencies are provided to your app.


I suspect that you don't have a connection to your C* instance. We can't help further without a full output log.

Cheers

4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

randas avatar image randas commented ·

@jaroslaw.grabowski_50515
I'm preparing the dependency on a deployment server manually for the cassandra spark connector
so I need to copy all the

Provided Dependencies (9)

Compile Dependencies (2)

Test Dependencies (10)

which comes with

Spark Cassandra Connector ?

0 Likes 0 ·
jaroslaw.grabowski_50515 avatar image jaroslaw.grabowski_50515 ♦ randas commented ·

Only compile dependencies. But note that you need the whole dependency tree (you also need dependencies of the dependencies).

If I understand correctly you don't want to rely on Spark to resolve the SCC and it's dependencies. In this you may use spark-cassandra-connector-assembly. It should be very convenient in your case. It's just a single jar with all the needed classes. You don't have to worry about the dependencies at all.

1 Like 1 ·
randas avatar image randas jaroslaw.grabowski_50515 ♦ commented ·

after using spark-cassandra-connector-assembly.
I could retrieve from cassandra via

val res = CassandraConnector(conf).withSessionDo { session =>
  session.execute("select * From test.t5")
}

but got can't see any keyspaces via spark.read.format 

val typeData = spark.read.format("org.apache.spark.sql.cassandra")
  .options(Map( "table" -> t5, "keyspace" -> test))
  .load().



2 Likes 2 ·
Show more comments