Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

ajay.prajakta_128320 avatar image
ajay.prajakta_128320 asked Erick Ramirez answered

Getting "ClassNotFoundException: com.twitter.jsr166e.LongAdder" with Spark connector v1.2

Hi Team,

In our one of the old deployment we are using Datastax spark cassandra connector version 1.2.

Following are the versions of our components

  1. Cassandra - 2.0.4
  2. Spark - 1.2
  3. Spark Cassandra connector - 1.2

During our write from spark to cassandra we are getting java.lang.ClassNotFoundException: com.twitter.jsr166e.LongAdder exception. When we checked the Connector jar we could see the JSR166e is available. This jar is coming with connector pom. But in run time it is not able to find out the jar.

Code Snippet :

CassandraJavaUtil.javaFunctions(myJavaRdd)
  .writerBuilder(keyspace COLUMN_FAMILY, CassandraJavaUtil.mapToRow(Class, myMap))
  .withPerRowTTL(Constants.TTL).saveToCassandra();

Exception:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 3.0 failed 4 times, most recent failure: Lost task 8.3 in stage 3.0 (TID 118, vmc0251):
java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
  at com.datastax.spark.connector.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
  at com.datastax.spark.connector.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:146)
  at com.datastax.spark.connector.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:76)
  at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:119)
  at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
  at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
  at org.apache.spark.scheduler.Task.run(Task.scala:56)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.twitter.jsr166e.LongAdder
  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
  ... 12 more Driver stacktrace:

Please help us on that issue.

Regards,

Ajay

spark-cassandra-connector
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

The exception you posted is usually caused misconfiguration where the connector or its dependencies are not in the classpath.

You need to use the --packages method to add the connector + dependencies to the runtime classpath. Otherwise, you need to check that your app is correctly configured as described in the spark-cassandra-connector FAQ. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.