Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

flovic.gosselin_187484 avatar image
flovic.gosselin_187484 asked ·

Is it possible to map messages from the same topic that have different Avro schemas?

I have a kafka-topic in which I put messages with different avro schemas, for example Messages1 contain Data0, Data1 and Data2, while Messages2 contain Data0, Data3 and Data4, Data1 and Data2 defaulting to null.

With a "global" mapping, including all DataX, it gives error that Data1 and Data2 are null when receiving Messages2, and that Data3 and Data4 are nulll when receiving Messages1.

Is there a way to handle that ? Kind of a "condional" mapping ?

Thanks in advance.

connectorkafka
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Tomasz Lelek avatar image
Tomasz Lelek answered ·

Hello, sending multiple events with a different schema to one topic is considered an anti-pattern in Kafka. Creating Kafka Topic does not incur substantial overhead so it is advised to create one per Avro schema. It will make the maintainability of your producers and consumers easier. Achieving backward compatibility is also a lot easier in such a setup.

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Thanks Tomasz, I'm pretty new to that indeed, your advice are most welcome !

0 Likes 0 · ·
Erick Ramirez avatar image
Erick Ramirez answered ·

@flovic.gosselin_187484 Off the top of my head, I don't think your configuration is supported. You might instead need to have two Kafka topics, each with their own Avro format.

I'm going to reach out internally and get other engineers at DataStax to respond. Cheers!

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Thanks Erick, I'll go that way then. Cheers.

0 Likes 0 · ·