Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

nmelotte_125932 avatar image
nmelotte_125932 asked ·

Kafka Connect to Casandra :: Mapping of JSON nested field

I'm trying to parse a nested field into 1 column in cassandra. For example, the value looks like :

{
    field1: value1,
    field2: value2,
    data{
         field3 : value3
    }
}

I manage to parse field1 & field2 by doing :

topic.topic.keyspace.string.mapping=kafka_key=key , field1=value.field1, field2=value.field2

But when trying :

topic.topic.keyspace.string.mapping=kafka_key=key , field1=value.field1, field2=value.field2, field3=value.data.field3

I have the following error :

[2020-01-10 20:32:49,445] WARN Error inserting/updating row for Kafka record SinkRecord{kafkaOffset=25870785, timestampType=CreateTime} 

(...)

Required field 'value.data.field3' (mapped to column field3) was missing from record. Please remove it from the mapping. (com.datastax.kafkaconnector.DseSinkTask:286)

Is it possible to map this field or is it not supported ?

cassandrakafka-connectorjsonmappingnested field
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

csplinter avatar image
csplinter answered ·

Hi there,


Mapping nested fields is not currently natively supported in the DataStax Kafka Connector though it is on our radar of future built-in functionality.


For now, you can use Single Message Transforms to flatten your message and access the nested field in the connector mapping. To do this, you can add the following to your connector configuration file.

"transforms": "flatten",
"transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value",
"transforms.flatten.delimiter": "_"

and then you should be able to do

topic.topic.keyspace.string.mapping=kafka_key=key, field1=value.field1, field2=value.field2, field3=value.data_field3
1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

awesome it works !

Thanks this will have to do for now !!

0 Likes 0 · ·