question

baid_manish_187433 avatar image
baid_manish_187433 asked bettina.swynnerton answered

Why does graphFrame() on BYOS return "NoClassDefFoundError: com/datastax/driver/core/TupleImpl"?

hi,

When using external spark (byos: https://docs.datastax.com/en/dse/6.8/dse-dev/datastax_enterprise/spark/byosIntro.html), if there is a tuple data type property, spark integration fails

val g = DseGraphFrameBuilder.apply("graph-test", spark).graphFrame()

Getting the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/driver/core/TupleImpl
at com.datastax.bdp.graphv2.dsedb.schema.Column$Type$4.of(Column.java:407)
at com.datastax.bdp.graphv2.dsedb.DataStoreUtil.getTypeFromDriver(DataStoreUtil.java:208)
at com.datastax.bdp.graphv2.dsedb.DataStoreUtil.getTypeFromDriver(DataStoreUtil.java:203)
at com.datastax.bdp.graphv2.dsedb.ExternalDataStore.lambda$extractColumns$11(ExternalDataStore.java:421)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at com.datastax.bdp.graphv2.dsedb.ExternalDataStore.extractColumns(ExternalDataStore.java:417)
at com.datastax.bdp.graphv2.dsedb.ExternalDataStore.lambda$null$3(ExternalDataStore.java:227)
at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707)
at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1082)
at com.datastax.bdp.graphv2.dsedb.ExternalDataStore.lambda$createSchema$4(ExternalDataStore.java:225)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at com.datastax.bdp.graphv2.dsedb.ExternalDataStore.createSchema(ExternalDataStore.java:218)
at com.datastax.bdp.graphv2.dsedb.ExternalDataStore.<init>(ExternalDataStore.java:149)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$$anonfun$withDataStoreDo$1$$anonfun$3.apply(DseGraphFrameBuilder.scala:186)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$$anonfun$withDataStoreDo$1$$anonfun$3.apply(DseGraphFrameBuilder.scala:186)
at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$$anonfun$withDataStoreDo$1.apply(DseGraphFrameBuilder.scala:186)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$$anonfun$withDataStoreDo$1.apply(DseGraphFrameBuilder.scala:183)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withClusterDo$1.apply(CassandraConnector.scala:136)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withClusterDo$1.apply(CassandraConnector.scala:135)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:115)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:114)
at com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:158)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:114)
at com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:135)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.withDataStoreDo(DseGraphFrameBuilder.scala:182)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.schema(DseGraphFrameBuilder.scala:131)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.schemaKeyspace(DseGraphFrameBuilder.scala:143)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:118)
at com.confluxsys.kb.analytics.test.spark.remote.TestSparkApp$.delayedEndpoint$com$confluxsys$kb$analytics$test$spark$remote$TestSparkApp$1(TestSparkApp.scala:36)
at com.confluxsys.kb.analytics.test.spark.remote.TestSparkApp$delayedInit$body.apply(TestSparkApp.scala:17)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at com.confluxsys.kb.analytics.test.spark.remote.TestSparkApp$.main(TestSparkApp.scala:17)
at com.confluxsys.kb.analytics.test.spark.remote.TestSparkApp.main(TestSparkApp.scala)
Caused by: java.lang.ClassNotFoundException: com.datastax.driver.core.TupleImpl
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 58 more


dsegraphbyos
3 comments
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

bettina.swynnerton avatar image bettina.swynnerton ♦♦ commented ·

Would you be able to share the graph schema for the graph "graph-test", or just for the vertex/edge where the tuple is defined?

0 Likes 0 ·
baid_manish_187433 avatar image baid_manish_187433 bettina.swynnerton ♦♦ commented ·

Here is the schema...


//--------------vertex : Alert --------------------------------------------------//
schema.vertexLabel('Alert').
        ifNotExists().
        partitionBy('tenantId', Ascii).
        partitionBy('appId', Ascii).
        partitionBy('nativeType', Ascii).
        clusterBy('level', Tinyint).
        clusterBy('gid', Uuid, Asc).
        property('createTime', Timestamp).
        property('updateTime', Timestamp).
        property('type', Ascii).
        property('processId', Uuid).
        property('instanceId', Uuid).
        property('summary', Text).
        property('taskId', Smallint).
        property('taskName', Ascii).
        property('valuePath', Ascii).
        property('valueSignature', Text).
        property('messages', listOf(tupleOf(Text,Text))).
        create()
0 Likes 0 ·
bettina.swynnerton avatar image bettina.swynnerton ♦♦ baid_manish_187433 commented ·

Thanks, I see the same when accessing via BYOS (from Spark 2.4.5), I'll see what I find.

0 Likes 0 ·

1 Answer

bettina.swynnerton avatar image
bettina.swynnerton answered

Hi. @baid_manish_187433,

thank you for raising this issue, and yes, we have confirmed that this is a problem with the DSE 6.8 BYOS jar.

We are working on a fix (internal reference DSP-21015), and I am also investigating workarounds. I will update this answer with the progress.

Thanks for your patience!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.