question

Hi,

for a short term workaround, please try with this driver version, this one has the com.datastax.driver.dse.DseCluster class

<dependency>
  <groupId>com.datastax.dse</groupId>
  <artifactId>dse-java-driver-core</artifactId>
  <version>1.9.0</version>
</dependency>

The cluster handling changed with the newer unified drivers. I will investigate this further what this means for java spark jobs through DSE BYOS, but let us know this works with the older driver.

Note that for graph queries through script API or fluent API, this driver version will fail due to protocol differences when accessing DSE Graph 6.8.

Let me know how you get on with the different dependency.

Thanks!

As this issue was still not quite resolved, I came back to this problem and did some further tests, and I believe the DSE Cluster Casting issue is a result of incorrectly submitting the Spark job.

For reference, here is an example how to use DSE Graphframes in DSE 6.8.1 with external Spark, using the DSE BYOS jar.

My test setup:

DSE Graph 6.8.1

Apache Spark 2.4.6 running in a different cluster

I am building and packaging the Spark job in IntelliJ as an sbt project, using sbt 0.13.18

Here is the build.sbt file:

name := "graphframe68"

version := "0.1"

scalaVersion := "2.11.12"

resolvers += "DataStax Repo" at "https://repo.datastax.com/public-repos/"

val dseVersion = "6.8.1"

libraryDependencies += "com.datastax.dse" % "dse-spark-dependencies" % dseVersion % "provided" exclude(
  "org.slf4j", "log4j-over-slf4j")

Here is the scala job, GraphFrameTest.scala

import com.datastax.spark.connector._
import com.datastax.bdp.graph.spark.graphframe._
import org.apache.spark.sql.SparkSession

object GraphFrameTest extends App {

  val spark = SparkSession.builder
    .appName("Datastax Scala example")
    .enableHiveSupport()
    .getOrCreate()

  val g = spark.dseGraph("test")

  g.V().show()

}

Note: you need to build and package and then load to the Spark cluster where you want to submit the job. You cannot run this directly from IntelliJ.

To build and package:

sbt package

Before you can submit the job on the Spark node, make sure you have imported the dse BYOS jar from the DSE Graph cluster. You can find it on the DSE Graph cluster (in case of a package installation) here: /usr/share/dse/clients/dse-byos_2.11-6.8.1.jar

In my test, I copied this file to the SPARK_HOME directory on the Spark cluster master node.

In addition, you need to export the byos.properties from the DSE Graph cluster.

To get the byos.properties, run the following command on a DSE Graph cluster node:

dse client-tool configuration byos-export ~/byos.properties

Copy it to the Spark cluster. I placed it into the SPARK_HOME on the master node.

Then, to submit your job, you need to specify the DSE BYOS jar and the properties file, along with the class you want to run and the job jar file.

Here is the complete submit command:

$SPARK_HOME/bin/spark-submit --master spark://xxx.xxx.xxx.xxx:7077 --jars dse-byos_2.11-6.8.1.jar --properties-file byos.properties --class GraphFrameTest graphframe68_2.11-0.1.jar

I hope these steps help to understand how to use DSE Graphframes in a Spark job.