Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

MayuriD avatar image
MayuriD asked MayuriD edited

Cassandra Join where column name is in camel case returns "column not found"

Cassandra join with streaming data is not working when column name is stored in Cassandra as camelCase
Steps to reproduce :

1. create a Cassandra table with column name in camelCase

CREATE TABLE pl_event (
      "tntId" varcha
      "pId" varcha
      type varcha
      PRIMARY KEY ("tntId", "pId
) WITH comment='event mapping records';

2. Read streaming data from kafka

eg :entityDF

3. Read data from Cassandra table. eg :

 Dataset<Row> cassandraDf = spark.read()
    .format("org.apache.spark.sql.cassandra")
    .option("table", "pl_event")
    .option("keyspace", "test")
    .option("directJoinSetting","on")
    .load();

4. Try to join streaming DF with Cassandra read DF

entityDF = entityDF.join(cassandraDF,cassandraDF.col("tntId").equalTo(entityDF.col("tntId")).and(cassandraDF.col("pId").equalTo(entityDF.col("pId"))));

Error :

This fail with error saying "tntid" column is not found

Even we try to escape "tntId", it that case it says unable to find column ""tntId"" from table

Any suggestion or solution to handle direct join with column name stored in camelCase?

spark-cassandra-connector
4 comments
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@MayuriD , could you post the output of running cassandraDF.show ?

0 Likes 0 ·
MayuriD avatar image MayuriD smadhavan ♦ ·

@smadhavan Here is the output of cassandraDF.show

+---------------+-----+------------------+
|tntId       |pId| |timeout                                                              |
+---------------+-----+------------------+
|T1|1001 |10 min           
|T2|1002 |20 min
+---------------+-----+------------------+
0 Likes 0 ·
Hi! Which version of the connector do you use? Could you paste the exact error you're getting (full stacktrace included)?
0 Likes 0 ·
Show more comments

0 Answers