Issue with spring-boot-starter-data-cassandra and spark-cassandra-connector?

Question

question

NiD asked Apr 22, '20 NiD commented Apr 22, '20

Issue with spring-boot-starter-data-cassandra and spark-cassandra-connector?

Am trying to create a spring boot project (2.2.6.RELEASE) in which i want save data to Cassandra using spring data Cassandra (spring-boot-starter-data-cassandra) and read data with spark (spark-cassandra-connector). But am facing many issue one after another.

Here is my pom.xml. Can anyone help me?

<properties>
<java.version>1.8</java.version>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.4.2</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.11</artifactId>
<version>2.4.2</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-cassandra</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

With above mentioned pom while building the the first error i got was

SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/cinesoft/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/cinesoft/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] log4j:WARN No appenders could be found for logger (org.springframework.test.context.BootstrapUtils). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

then i exclude slf4j-log4j12 from spark sql like this

<dependency> 
<groupId>org.apache.spark</groupId> 
<artifactId>spark-sql_2.11</artifactId> 
<version>2.4.2</version> 
<exclusions> 
<exclusion> 
<groupId>org.slf4j</groupId> 
<artifactId>slf4j-log4j12</artifactId> 
</exclusion> 
</exclusions> 
</dependency>

then i got

Caused by: java.lang.NoSuchMethodError: com.datastax.driver.core.TypeCodec.getJavaType()Lcom/google/common/reflect/TypeToken; at java.lang.invoke.MethodHandleNatives.resolve(Native Method) ~[na:1.8.0_191] at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:975) ~[na:1.8.0_191] at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1000) ~[na:1.8.0_191] at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:1394) ~[na:1.8.0_191] at java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(MethodHandles.java:1750) ~[na:1.8.0_191] at java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:477) ~[na:1.8.0_191] ... 231 common frames omitted

spark connector spring-data-cassandra

2

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez ♦♦ · Apr 21, 2020 at 10:53 PM

@nithinthomas.pro_188349 You stated you have a problem but you didn't specify what they are. Please update your question with additional information like what you tried and what errors you were getting. Cheers!

0 ·

NiD Erick Ramirez ♦♦ · Apr 22, 2020 at 07:01 AM

Thanks for the response, i have updated my question . Please check it and let me if you need more details.

0 ·

Answer 1 · 2020-04-22T12:55:12Z

Russell Spitzer answered Apr 22, '20 NiD commented Apr 22, '20

This is almost certainly caused by a conflict in Java Driver versions being included. Although we have done our best to shade the Java Drvier and all of it's guava references with the Spark Cassandra Connector object, it looks like reflection is failing us at runtime.

You can see a brief explanation of why here
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md#why-is-the-cassandra-java-driver-embedded-in-spark-cassandra-connector-artifacts

The solution to this is to use the unshaded Spark Cassandra Connector artifact and reshade guava yourself if Spark has issues with multiple Guava Versions on the classpath.

https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector-unshaded_2.11/2.4.3

Or upgrade once we have a release, to Spark Cassandra Connector 2.5 which will no longer require shading because Guava will no longer be a requirement.

1 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

NiD · Apr 22, 2020 at 05:40 PM

Thank you for your response.

As you suggested i replaced my spark Cassandra connector with unshaded one, now am able to build and run the project. But while i try to read cassandra table using spark session it gave me an error like this.

Caused by: java.lang.ClassNotFoundException: com.codahale.metrics.JmxReporter
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 70 more

Hope you can help me .

0 ·

question