question

t.ariunbat_189192 avatar image
t.ariunbat_189192 asked t.ariunbat_189192 commented

Is it possible to use the Kafka HDFS sink connector with DSEFS?

I guess it is possible to use Kafka HDFS sink connector with DSEFS. Do we need to provide any credentials on connector? Is the HDFS version 1.3 as mentioned in the version doc? Do we write directly to dsefs://192.168.1.72:5598 this port or another port?

Thanks in advance

dsefs
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered t.ariunbat_189192 commented

[UPDATED] You should be able to use DSEFS since it is HDFS-compatible and will work in place of HDFS but you will need to work out on your own how to configure the connector since we don't test or support that connector.

To respond to your questions directly:

  1. You will need to provide credentials in order to access DSEFS if you have enabled authentication in your DSE cluster. How you provide the credentials depends on how you are accessing DSEFS. For details, see DSEFS authentication.
  2. DSE 6.8 ships with Hadoop core 1.0.3.
  3. DSEFS listens on port 5598 for client requests by default.

If you have any follow up questions, please log a ticket with DataStax Support so one of our engineers can assist you. Cheers!

4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

t.ariunbat_189192 avatar image t.ariunbat_189192 commented ·

Here is the link: https://docs.datastax.com/en/landing_page/doc/landing_page/3rdpartysoftware/dse686.html

It was hadoop-core 1.0.3 my mistake.

I am using DSE 6.8.5 in docker env. I guess docker or not it is the same right? I am making POC in docker and prod is not docker just fyi.

I am trying to use this connector assuming it is for hadoop v1.x https://www.confluent.io/hub/confluentinc/kafka-connect-hdfs there are other version 2 and version 3 connector as well.

0 Likes 0 ·
t.ariunbat_189192 avatar image t.ariunbat_189192 commented ·

@Erick Ramirez Have you asked about it?


0 Likes 0 ·
jaroslaw.grabowski_50515 avatar image jaroslaw.grabowski_50515 ♦ t.ariunbat_189192 commented ·

Hi, just for the record, DSE 6.8 uses Apache Spark 2.4.x which depends on Apache Hadoop 2.6.5. So in DSE you have Hadoop 2.6.5.

2 Likes 2 ·
t.ariunbat_189192 avatar image t.ariunbat_189192 commented ·

I was able to configure the connector. I have added dse-byos.jar and scala-library.jar into the connectors' library and it worked without a problem. Thanks for the information.

0 Likes 0 ·