The documentation for Livy integration with DSE 6.7 describes how to connect Livy to DSE when Livy is running on one of the DSE Analytics nodes. However, is there a way to connect Livy to DSE when Livy is running on a separate node?
The use case is that we have an existing DSE cluster that we're not able to migrate over to Kubernetes, but want Livy running on Kubernetes so that it can scale and be managed along with a number of other services we're integrating with our DSE cluster.
What I tried so far:
The way I have it setup is that Livy is co-located with a Spark binary, which is used as the Spark Driver. The SPARK_HOME env var is set to where that Spark binary is in the filesystem. Then `livy.spark.master` in livy.conf is set to the Spark Master, in this case a DSE Node:
# assuming 220.127.116.11 is my DSE Analytics node: livy.spark.master = dse://18.104.22.168:7077 livy.repl.enable-hive-context = true
This is my best guess of how it should work by following the documentation for Livy integration. However, it then throws this error:
org.apache.spark.SparkException: Master must either be yarn or start with spark, mesos, k8s, or local
I'm guessing that the dse:// protocol is only possible when the Spark Driver is a Spark Node, which makes sense, but still leaves me not knowing how to connect Livy.
It does work with non-DSE Spark:
I used a similar process for connecting to a non-DSE Spark Master, and it worked fine, though using spark:// protocol instead:
livy.spark.master = spark://22.214.171.124:7077
It does not work with spark:// protocol on a DSE node:
Since it worked when used with non-DSE Spark Master, I figured I would give it a shot, and tried using the spark:// protocol but with the DSE node's ip, e.g., something like this:
# assuming 126.96.36.199 is my DSE Analytics node: livy.spark.master = spark://188.8.131.52:7077
However, this returned a different error:
ERROR SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
Any suggestions for this?