question

Ryan Quey avatar image
Ryan Quey asked Erick Ramirez commented

Can I use BYOS jar at the same time as DSE Java Driver?

I'm using DSE 6.7 and have a project that was originally written for running Spark using DSE Analytics nodes. However, now I want to try getting the same Spark submit job jar to run on a different Spark runtime (specifically, Databricks) instead.

Accordingly, I am trying to use the byos jar as described here. However, it seems like some of the classes in that jar are conflicting with the DSE java driver classes that we are using, and it seems to be causing some strange behaviour. Specifically, I noticed that both the java driver and byos are defining com.datastax.driver.dse.graph.GraphJsonUtils, though I'm sure there are others as well.

I could give more details about the specific error I'm running into, but before that just wanted to ask more generally if there is a standard procedure for using both BYOS jar and DSE Java Driver together. Or is it that if I'm using BYOS, I should just use the classes defined there and avoid also specifying the DSE java driver as a dependency?

java driverbyos
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

3 Answers

jaroslaw.grabowski_50515 avatar image
jaroslaw.grabowski_50515 answered Erick Ramirez converted comment to answer
..., I should just use the classes defined there and avoid also specifying the DSE java driver as a dependency

Yes.

Alternatively you may use Spark Cassandra Connector 2.5.1/3.0.0 if you don't use DSEFS/ Kerberos/Token authentication. https://github.com/datastax/spark-cassandra-connector

Note that if you decide to go with SCC we recommend using spark-cassandra-connector-assembly artifact (as opposed to spark-cassandra-connector artifact) which is a fat jar with some of the dependencies shaded. It was created to avoid classpath conflicts when SCC is used in Databricks env.

Similarly to BYOS, SCC comes with a version of the driver that should not be overridden in user apps.



2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Ryan Quey avatar image Ryan Quey commented ·

Great thanks! Is there documentation or articles on spark-cassandra-connector-assembly for reference? I see these comments here, but is there anything more descriptive and exhaustive?

0 Likes 0 ·
jaroslaw.grabowski_50515 avatar image jaroslaw.grabowski_50515 ♦ Ryan Quey commented ·

We don't have its content documented. Here is the definition, at the moment we only shade typesafe.config https://github.com/datastax/spark-cassandra-connector/blob/master/build.sbt#L51. Other dependencies are simply packaged into the archive.

1 Like 1 ·
Erick Ramirez avatar image
Erick Ramirez answered jaroslaw.grabowski_50515 commented

The short answer is to just stick with BYOS.

If you provide some sample code, I'd be happy to socialise it here in DataStax with the Analytics devs. Cheers!

3 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Ryan Quey avatar image Ryan Quey commented ·

Got it, thanks.

Is there a chart or documentation somewhere that shows what version of the DSE Java driver is bundled up within each version of the BYOS jar? E.g., if I'm using BYOS for dse 6.7.7 and scala 2.11 (dse-byos_2.11-6.7.7.jar), what is the comparable dse java driver code? E.g., is it DSE java driver 2.3, or C* OSS java driver 4.x or something else?

Is there a chart that maps this out for us? I need to know so I can see if our current code that relies on the dse java driver will be compatible with the DSE java driver classes defined in byos jar.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Ryan Quey commented ·

I believe it's the same DSE Java driver (not OSS) that ships with the particular version of DSE. For example, DSE 6.7.7 includes DSE Java driver 1.7.1.

@jaroslaw.grabowski_50515 could you please confirm this is true for the BYOS JAR? Cheers!

1 Like 1 ·
jaroslaw.grabowski_50515 avatar image jaroslaw.grabowski_50515 ♦ Erick Ramirez ♦♦ commented ·

Yes! Note that we plan to upgrade to 1.8.3-dse (unreleased yet) in the near future.

1 Like 1 ·
Ryan Quey avatar image
Ryan Quey answered Erick Ramirez commented

For future reference, this blog by Alex Ott addresses the use of the spark-cassandra-connector-assembly jar with Databricks:


"Because we have dependency conflict between SCC and Databricks runtime we must not use the spark-cassandra-connector artifact, but the assembly version of it: spark-cassandra-connector-assembly (see SPARKC-601 for details). For runtime version 6.x we need to use com.datastax.spark:spark-cassandra-connector-assembly_2.11:2.5.1, and for runtime 7.0 we need to take com.datastax.spark:spark-cassandra-connector-assembly_2.12:3.0.0-beta (or released version when it's done)."

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image Erick Ramirez ♦♦ commented ·

Thanks for circling back and posting an update. I'm sure it will come in handy for someone in the future. Cheers!

0 Likes 0 ·