PLANNED MAINTENANCE

Hello, DataStax Community!

We want to make you aware of a few operational updates which will be carried out on the site. We are working hard to streamline the login process to integrate with other DataStax resources. As such, you will soon be prompted to update your password. Please note that your username will remain the same.

As we work to improve your user experience, please be aware that login to the DataStax Community will be unavailable for a few hours on:

  • Wednesday, July 15 16:00 PDT | 19:00 EDT | 20:00 BRT
  • Thursday, July 16 00:00 BST | 01:00 CEST | 04:30 IST | 07:00 CST | 09:00 AEST

For more info, check out the FAQ page. Thank you for being a valued member of our community.


question

nmelotte_125932 avatar image
nmelotte_125932 asked ·

Kafka Connect to Casandra :: Mapping of JSON nested field

I'm trying to parse a nested field into 1 column in cassandra. For example, the value looks like :

{
    field1: value1,
    field2: value2,
    data{
         field3 : value3
    }
}

I manage to parse field1 & field2 by doing :

topic.topic.keyspace.string.mapping=kafka_key=key , field1=value.field1, field2=value.field2

But when trying :

topic.topic.keyspace.string.mapping=kafka_key=key , field1=value.field1, field2=value.field2, field3=value.data.field3

I have the following error :

[2020-01-10 20:32:49,445] WARN Error inserting/updating row for Kafka record SinkRecord{kafkaOffset=25870785, timestampType=CreateTime} 

(...)

Required field 'value.data.field3' (mapped to column field3) was missing from record. Please remove it from the mapping. (com.datastax.kafkaconnector.DseSinkTask:286)

Is it possible to map this field or is it not supported ?

cassandrajsonkafka-connectormappingnested field
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

csplinter avatar image
csplinter answered ·

Hi there,


Mapping nested fields is not currently natively supported in the DataStax Kafka Connector though it is on our radar of future built-in functionality.


For now, you can use Single Message Transforms to flatten your message and access the nested field in the connector mapping. To do this, you can add the following to your connector configuration file.

"transforms": "flatten",
"transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value",
"transforms.flatten.delimiter": "_"

and then you should be able to do

topic.topic.keyspace.string.mapping=kafka_key=key, field1=value.field1, field2=value.field2, field3=value.data_field3
1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

awesome it works !

Thanks this will have to do for now !!

0 Likes 0 · ·