question

MayuriD avatar image
MayuriD asked Erick Ramirez edited

Accessing AWS Keyspaces with Spark connector returns "Unsupported partitioner 'com.amazonaws.cassandra.DefaultPartitioner', token map will be empty."

We are trying to connect Spark with Keyspaces on AWS using below library and we are facing error "java.lang.IllegalArgumentException: Unable to get Token Metadata".

Library :- com.datastax.spark:spark-cassandra-connector_2.12:3.1.0

As we understand, Keyspaces uses partitioner as "com.amazonaws.cassandra.DefaultPartitioner" and it is not supported by Datastax java driver. Recommended partitioners are "org.apache.cassandra.dht.Murmur3Partitioner", "org.apache.cassandra.dht.RandomPartitioner". As a result, we get below warning while connecting to Keyspaces.

WARN DefaultTokenFactoryRegistry: [s0] Unsupported partitioner 'com.amazonaws.cassandra.DefaultPartitioner', token map will be empty.

It further results in error on Spark implementation.

java.lang.IllegalArgumentException: Unable to get Token Metadata
at com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy.$anonfun$tokenMap$1(LocalNodeFirstLoadBalancingPolicy.scala:86)
at scala.Option.orElse(Option.scala:447)
at com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy.tokenMap(LocalNodeFirstLoadBalancingPolicy.scala:86)

Could you please provide your guidance on this.
We are seeing this https://community.datastax.com/questions/2989/trying-to-connect-to-aws-cassandra-with-datastaxsp.html older post.. Is there any latest update?

spark-cassandra-connectorunsupportedaws keyspaces
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered MayuriD commented

The Spark connector isn't tested against AWS Keyspaces because it isn't supported.

As you already pointed out, AWS Keyspaces uses an unsupported partitioner so you won't be able to connect using the Spark connector. Cheers!

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

MayuriD avatar image MayuriD commented ·
@Erick Ramirez Thank You!!
0 Likes 0 ·