Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

scano_183208 avatar image
scano_183208 asked ·

Getting DseFsConnectionException using Livy client

Getting the following error when using livy client to dowload a jar file from dsefs directory to a tmp folder.

When I sudo su or simply use dse fs I do not get any errors.

20/12/28 22:06:45 INFO BatchSession: Batch session 19 created [appid: null, state: running, info: {driverLogUrl=null, sparkUiUrl=null}]
20/12/28 22:06:55 INFO LineBufferedStream: Exception in thread "main" com.datastax.bdp.fs.client.DseFsConnectionException: Could not connect to DSEFS. Last endpoint tried: 192.168.21.4:5598
20/12/28 22:06:55 INFO LineBufferedStream:      at com.datastax.bdp.fs.client.RestClientProxy.<init>(RestClientProxy.scala:161)
20/12/28 22:06:55 INFO LineBufferedStream:      at com.datastax.bdp.fs.client.DseFsClient.liftedTree1$1(DseFsClient.scala:62)
20/12/28 22:06:55 INFO LineBufferedStream:      at com.datastax.bdp.fs.client.DseFsClient.<init>(DseFsClient.scala:62)
20/12/28 22:06:55 INFO LineBufferedStream:      at com.datastax.bdp.fs.hadoop.DseFileSystem.initialize(DseFileSystem.scala:129)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:362)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.spark.deploy.SparkSubmit$.downloadFile(SparkSubmit.scala:885)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.spark.deploy.DseSparkSubmit$$anonfun$prepareSubmitEnvironment$1.apply(DseSparkSubmit.scala:290)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.spark.deploy.DseSparkSubmit$$anonfun$prepareSubmitEnvironment$1.apply(DseSparkSubmit.scala:290)
20/12/28 22:06:55 INFO LineBufferedStream:      at scala.Option.map(Option.scala:146)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.spark.deploy.DseSparkSubmit$.prepareSubmitEnvironment(DseSparkSubmit.scala:290)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.spark.deploy.DseSparkSubmit$.submit(DseSparkSubmit.scala:140)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.spark.deploy.DseSparkSubmit$.main(DseSparkSubmit.scala:106)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.spark.deploy.DseSparkSubmitBootstrapper$.main(DseSparkSubmitBootstrapper.scala:109)
20/12/28 22:06:55 INFO LineBufferedStream:      at org.apache.spark.deploy.DseSparkSubmitBootstrapper.main(DseSparkSubmitBootstrapper.scala)
20/12/28 22:06:55 INFO LineBufferedStream: Caused by: com.datastax.bdp.fs.rest.UnknownServerResponseException: The server sent an error response this client did not understand: HTTP 401:
20/12/28 22:06:55 INFO LineBufferedStream:      at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$331$1.apply(RestResponse.scala:51)
20/12/28 22:06:55 INFO LineBufferedStream:      at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$331$1.apply(RestResponse.scala:44)
20/12/28 22:06:55 INFO LineBufferedStream:      at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
20/12/28 22:06:55 INFO LineBufferedStream:      at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
20/12/28 22:06:55 INFO LineBufferedStream:      at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
20/12/28 22:06:55 INFO LineBufferedStream:      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465)
20/12/28 22:06:55 INFO LineBufferedStream:      at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
20/12/28 22:06:55 INFO LineBufferedStream:      at java.lang.Thread.run(Thread.java:748)
20/12/28 22:07:00 WARN BatchSession$: spark-submit exited with code 1
20/12/28 22:07:00 ERROR SparkProcApp: spark-submit exited with code 1
dsefslivy
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

The stack trace indicates you have an authentication issue and that your app has not been provided with the correct credentials to access DSEFS.

We have limited ability to assist you in a Q&A forum so please log a ticket with DataStax Support so one of our engineers can assist you directly. You will need to provide the sample code that replicates the problem as well as a Diagnostics Report from OpsCenter to assist with the investigation. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.