question

NiD avatar image
NiD asked NiD commented

Issue with spring-boot-starter-data-cassandra and spark-cassandra-connector?

Am trying to create a spring boot project (2.2.6.RELEASE) in which i want save data to Cassandra using spring data Cassandra (spring-boot-starter-data-cassandra) and read data with spark (spark-cassandra-connector). But am facing many issue one after another.

Here is my pom.xml. Can anyone help me?

<properties>
<java.version>1.8</java.version>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.4.2</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.11</artifactId>
<version>2.4.2</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-cassandra</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

With above mentioned pom while building the the first error i got was

SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/cinesoft/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/cinesoft/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] log4j:WARN No appenders could be found for logger (org.springframework.test.context.BootstrapUtils). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

then i exclude slf4j-log4j12 from spark sql like this

<dependency> 
<groupId>org.apache.spark</groupId> 
<artifactId>spark-sql_2.11</artifactId> 
<version>2.4.2</version> 
<exclusions> 
<exclusion> 
<groupId>org.slf4j</groupId> 
<artifactId>slf4j-log4j12</artifactId> 
</exclusion> 
</exclusions> 
</dependency>

then i got

Caused by: java.lang.NoSuchMethodError: com.datastax.driver.core.TypeCodec.getJavaType()Lcom/google/common/reflect/TypeToken; at java.lang.invoke.MethodHandleNatives.resolve(Native Method) ~[na:1.8.0_191] at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:975) ~[na:1.8.0_191] at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1000) ~[na:1.8.0_191] at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:1394) ~[na:1.8.0_191] at java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(MethodHandles.java:1750) ~[na:1.8.0_191] at java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:477) ~[na:1.8.0_191] ... 231 common frames omitted
sparkconnectorspring-data-cassandra
2 comments
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image Erick Ramirez ♦♦ commented ·

@nithinthomas.pro_188349 You stated you have a problem but you didn't specify what they are. Please update your question with additional information like what you tried and what errors you were getting. Cheers!

0 Likes 0 ·
NiD avatar image NiD Erick Ramirez ♦♦ commented ·

Thanks for the response, i have updated my question . Please check it and let me if you need more details.

0 Likes 0 ·

1 Answer

Russell Spitzer avatar image
Russell Spitzer answered NiD commented

This is almost certainly caused by a conflict in Java Driver versions being included. Although we have done our best to shade the Java Drvier and all of it's guava references with the Spark Cassandra Connector object, it looks like reflection is failing us at runtime.

You can see a brief explanation of why here
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md#why-is-the-cassandra-java-driver-embedded-in-spark-cassandra-connector-artifacts

The solution to this is to use the unshaded Spark Cassandra Connector artifact and reshade guava yourself if Spark has issues with multiple Guava Versions on the classpath.

https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector-unshaded_2.11/2.4.3

Or upgrade once we have a release, to Spark Cassandra Connector 2.5 which will no longer require shading because Guava will no longer be a requirement.


1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

NiD avatar image NiD commented ·

Thank you for your response.

As you suggested i replaced my spark Cassandra connector with unshaded one, now am able to build and run the project. But while i try to read cassandra table using spark session it gave me an error like this.

Caused by: java.lang.ClassNotFoundException: com.codahale.metrics.JmxReporter
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 70 more

Hope you can help me .

0 Likes 0 ·