question

nmelotte_125932 avatar image
nmelotte_125932 asked Erick Ramirez edited

Kafka Connect to Casandra :: Mapping of JSON nested field

I'm trying to parse a nested field into 1 column in cassandra. For example, the value looks like :

{
    field1: value1,
    field2: value2,
    data{
         field3 : value3
    }
}

I manage to parse field1 & field2 by doing :

topic.topic.keyspace.string.mapping=kafka_key=key , field1=value.field1, field2=value.field2

But when trying :

topic.topic.keyspace.string.mapping=kafka_key=key , field1=value.field1, field2=value.field2, field3=value.data.field3

I have the following error :

[2020-01-10 20:32:49,445] WARN Error inserting/updating row for Kafka record SinkRecord{kafkaOffset=25870785, timestampType=CreateTime} 

(...)

Required field 'value.data.field3' (mapped to column field3) was missing from record. Please remove it from the mapping. (com.datastax.kafkaconnector.DseSinkTask:286)

Is it possible to map this field or is it not supported ?

kafka-connector
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

csplinter avatar image
csplinter answered nmelotte_125932 commented

Hi there,


Mapping nested fields is not currently natively supported in the DataStax Kafka Connector though it is on our radar of future built-in functionality.


For now, you can use Single Message Transforms to flatten your message and access the nested field in the connector mapping. To do this, you can add the following to your connector configuration file.

"transforms": "flatten",
"transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value",
"transforms.flatten.delimiter": "_"

and then you should be able to do

topic.topic.keyspace.string.mapping=kafka_key=key, field1=value.field1, field2=value.field2, field3=value.data_field3
1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

nmelotte_125932 avatar image nmelotte_125932 commented ·

awesome it works !

Thanks this will have to do for now !!

0 Likes 0 ·