Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

Ryan Quey avatar image
Ryan Quey asked Erick Ramirez edited

How do I connect remotely to DSE from a non-DSE Livy installation?

The documentation for Livy integration with DSE 6.7 describes how to connect Livy to DSE when Livy is running on one of the DSE Analytics nodes. However, is there a way to connect Livy to DSE when Livy is running on a separate node?

The use case is that we have an existing DSE cluster that we're not able to migrate over to Kubernetes, but want Livy running on Kubernetes so that it can scale and be managed along with a number of other services we're integrating with our DSE cluster.

What I tried so far:

The way I have it setup is that Livy is co-located with a Spark binary, which is used as the Spark Driver. The SPARK_HOME env var is set to where that Spark binary is in the filesystem. Then `livy.spark.master` in livy.conf is set to the Spark Master, in this case a DSE Node:

# assuming 1.2.3.4 is my DSE Analytics node:
livy.spark.master = dse://1.2.3.4:7077
livy.repl.enable-hive-context = true

This is my best guess of how it should work by following the documentation for Livy integration. However, it then throws this error:

org.apache.spark.SparkException: Master must either be yarn or start with spark, mesos, k8s, or local

I'm guessing that the dse:// protocol is only possible when the Spark Driver is a Spark Node, which makes sense, but still leaves me not knowing how to connect Livy.

It does work with non-DSE Spark:

I used a similar process for connecting to a non-DSE Spark Master, and it worked fine, though using spark:// protocol instead:

livy.spark.master = spark://5.6.7.8:7077

It does not work with spark:// protocol on a DSE node:

Since it worked when used with non-DSE Spark Master, I figured I would give it a shot, and tried using the spark:// protocol but with the DSE node's ip, e.g., something like this:

# assuming 1.2.3.4 is my DSE Analytics node:
livy.spark.master = spark://1.2.3.4:7077

However, this returned a different error:

ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem

Any suggestions for this?

byoslivy
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

jaroslaw.grabowski_50515 avatar image
jaroslaw.grabowski_50515 answered Ryan Quey commented

Hi! The command

dse exec livy-server

starts Livy with DSE jars on the classpath. To mimic this you could try adding 6.7.x BYOS jar on your livy classpath and additional config from BYOS properties file (https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/spark/byosGeneratingConfigFile.html).

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Ryan Quey avatar image Ryan Quey ·

Thanks for the tip, I'll give it a shot!

0 Likes 0 ·