question

Ryan Quey avatar image
Ryan Quey asked Tomasz Lelek commented

How do I configure authentication for Astra when using DataStax Kafka connector?

I am trying to setup Kafka connector to sink messages into Astra. I am working from the

conf/cassandra-sink-standalone.properties.sample

file provided in the connector zip. I am having trouble with authentication though.

The docs on the Datastax Kafka connector page seem to say that I should use DSE or GSSAPI or None. However, in the `connect-standalone.properties` template, it says "Authentication provider to use, if any. Valid choices: None, PLAIN, GSSAPI."

And indeed, if I try DSE it errors out saying:

[2020-08-25 21:11:49,758] ERROR WorkerSinkTask{id=cassandra-sink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
org.apache.kafka.common.config.ConfigException: Invalid value DSE for configuration auth.provider: valid values are None, PLAIN, GSSAPI

On the other hand, if I use PLAIN, I'm getting a different error, perhaps because of some other setting. Now it says:

[2020-08-25 20:58:44,186] WARN [s1|<my_host>-us-east1.db.astra.datastax.com:30185:413ab5b7-63ea-4d90-9fa6-57d40766c08a]  Error while opening new ch
annel (DriverTimeoutException: [s1|id: 0xeba1f6f0, L:/172.20.10.21:37394 - R:<my_host>-us-east1.db.astra.datastax.com/34.73.106.126:30185] Protocol
 initialization request, step 2 (AUTH_RESPONSE): timed out after 500 ms) (com.datastax.oss.driver.internal.core.pool.ChannelPoo
ERROR WorkerSinkTask{id=cassandra-sink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
java.lang.RuntimeException: Prepare failed for statement: 
...
Caused by: com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the query

Is this an Authentication problem, since it is saying that it is failing in AUTH_RESPONSE above. Or I am doing something else wrong?

astraauthenticationkafka-connector
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Tomasz Lelek commented

The setup for connecting to DataStax Astra is a little different since it has additional requirements.

The high level steps are:

  1. Download the Astra secure connect bundle for your database instance.
  2. For your Kafka installation, configure the dse-sink.json file to include the path to your secure bundle and credentials. For example:
"name": "dse-sink",
    "config": {
        "connector.class": "com.datastax.kafkaconnector.DseSinkConnector",
        "cloud.secureConnectBundle": "/path/to/secure-connect-database-name.zip",
        "auth.username": "username",
        "auth.password": "password"
...
    }
}

For details, see the Streaming data with the DataStax Apache Kafka Connector page on the Astra docs site. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Ryan Quey avatar image Ryan Quey commented ·

Thanks Erick, this is helpful. Can you use Astra with Kafka Connector standalone mode as well, or only distributed mode? The Astra docs seem to only show how to use distributed mode.

0 Likes 0 ·
Tomasz Lelek avatar image Tomasz Lelek Ryan Quey commented ·

You can use it in the standalone mode as well. You need to set the same settings in the .properties file:

cloud.secureConnectBundle = /path/to/secure-connect-database-name.zip
auth.username=username
auth.password=password



2 Likes 2 ·