question

scano_183208 avatar image
scano_183208 asked scano_183208 commented

spark-submit unable to connect to DSEFS

When i try the following command:

dse spark-submit --class main.ReportExporter --conf spark.master=dse:/// dsefs:/sparkbin/report-exporter-assembly-0.2.jar

I keep receiving the following error:

Exception in thread "main" com.datastax.bdp.fs.client.DseFsConnectionException: Could not connect to DSEFS. Last endpoint tried: 192.168.21.4:5598
    at com.datastax.bdp.fs.client.RestClientProxy.<init>(RestClientProxy.scala:161)
    at com.datastax.bdp.fs.client.DseFsClient.liftedTree1$1(DseFsClient.scala:62)
    at com.datastax.bdp.fs.client.DseFsClient.<init>(DseFsClient.scala:62)
    at com.datastax.bdp.fs.hadoop.DseFileSystem.initialize(DseFileSystem.scala:129)

I am having trouble finding to root cause of this and could use some help wherever possible.

Thank you,

Smon

dsefs
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

2 Answers

jaroslaw.grabowski_50515 avatar image
jaroslaw.grabowski_50515 answered scano_183208 commented

Hi!

Could you paste the full stack trace?

Are you able to connect to DSEFS with dsefs shell (`dse fs` command)? If no, could you paste the contents of `cat ~/.dsefs-shell.log` file?

Are you able to open spark-shell (`dse spark` command)? If no, could you paste the contents of `cat ~/.spark-shell.log` file?

Which DSE version is this?

Cheers,

Jarosław

9 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

scano_183208 avatar image scano_183208 commented ·

@jaroslaw.grabowski_50515 Hi, Thank you for getting back to me. I am able to connect/shell into dsefs shell with dse fs and I am also able to shell into spark with dse spark which is why I am baffled that I am getting this error: dseError.txt


Also, I am using version 6.7.7

0 Likes 0 ·
dseerror.txt (2.6 KiB)
jaroslaw.grabowski_50515 avatar image jaroslaw.grabowski_50515 ♦ scano_183208 commented ·

This is an authentication problem. What kind of authentication do you use?

0 Likes 0 ·
scano_183208 avatar image scano_183208 jaroslaw.grabowski_50515 ♦ commented ·

@jaroslaw.grabowski_50515

I am using the internal authentication that comes with DSE not KERBEROS or LDAP.


I am also using the superuser cassandra with the correct password in the app.

0 Likes 0 ·
Show more comments
Erick Ramirez avatar image
Erick Ramirez answered

As I stated in your other question (#9778), the stack trace indicates you have an authentication issue and that your app has not been provided with the correct credentials to access DSEFS.

A friendly note to let you know that this site is not intended to replace support issues and is designed for general questions.

We have limited ability to assist you in a Q&A forum so please log a ticket with DataStax Support so one of our engineers can assist you directly. You will need to provide the sample code that replicates the problem as well as a Diagnostics Report from OpsCenter to assist with the investigation. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.