Connecting Spark in docker container

Question

question

tapan.sharma_186956 asked · Mar 24 at 06:33 AM

Connecting Spark in docker container

In order to run Spark SQL queries on DSE Graph, I want to connect to Spark engine running inside the docker container. I have mapped 9042, 7077 ports so that they can be accessed outside docker.

Following is the Java code:

spark = SparkSession.builder().appName("My application name").config(new SparkConf())
            .master("dse://localhost:9042?connection.local_dc=dc1;connection.host=;").getOrCreate();

I am getting this error:

"Couldn't parse Master URL: dse://localhost:9042?connection.local_dc=dc1;connection.host=;"

I also tried "spark://localhost:9042?connection.local_dc=dc1;connection.host=;" master URL but in that case, it gives "Invalid Master URL" error.

What am I missing here?

dse spark graph spark-connector java api

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-03-24T13:26:00Z

Russell Spitzer answered · Mar 24 at 01:26 PM

Networking aside (this may still be an issue), the dse:// url can only be used with an application running with all of the DSE libraries on the classpath. Usually this means running with dse spark-submit from the command line. If your docker image does not have an install of DSE this will not be possible. In addition you should not include connection.host=; if you aren't specifying additional endpoints. Leave out this parameter entirely unless you actually have a value to set it to.

spark:// is the url for the OSS Spark Standalone manager, it doesn't know anything about Cassandra so parameters like "connection.local_dc" are invalid. It can only use single ip:port urls.

So the question here is whether or not you are attempting to contact a Dse Spark Master or a Resource Manager provided from some other Spark Service. If it's DSE you must make sure DSE is in your submitting docker image, or use another submission process like the CQL based RPC (which is a more advanced technique).

If the resource manager is not DSE you need to match the URL format specific to your cluster.

Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

question