Can't do update query via Kafka Connector.

Question

question

arsen.gasparyan.05_188257 asked Jun 9, '20 alexandre.dutra edited Jun 10, '20

Can't do update query via Kafka Connector.

I have the following Table

CREATE TABLE test.test ( id int, change_log list<text>, PRIMARY KEY (id) ) ;

Value : {"changeLog":["text"],"id":1234}

When I want to run update query by Kafka connector

topic.Test.test.test.query=UPDATE test.test SET change_log = change_log +  :changeLog WHERE id = :id ;

I get the following issue

ERROR Problem when getting queryFuture. This is likely a bug in the connector, please report. (com.datastax.kafkaconnector.DseSinkTask:153)
java.util.concurrent.ExecutionException: com.datastax.oss.driver.api.core.servererrors.ServerError: java.lang.ClassCastException: org.apache.cassandra.cql3.Constants$Value cannot be cast to org.apache.cassandra.cql3.Lists$Value

cassandra cql connector collections kafka

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-06-09T20:30:16Z

smadhavan answered Jun 9, '20 alexandre.dutra edited Jun 10, '20

@arsen.gasparyan.05_188257, it appears that the table schema expect change_log column value to be a list<text>`` whereas the actual JSON structure has it as text type: {..."changeLog":"text"...}. How about if you fix the JSON to have an actual list of strings something like {..."changeLog":["text"]...}?

Couple other references,

2 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

arsen.gasparyan.05_188257 · Jun 09, 2020 at 09:58 PM

I also try as you mentioned and I have the following

ERROR Problem when getting queryFuture. This is likely a bug in the connector, please report. (com.datastax.kafkaconnector.DseSinkTask:153)

java.util.concurrent.ExecutionException: com.datastax.oss.driver.api.core.servererrors.ServerError: java.lang.ClassCastException: org.apache.cassandra.cql3.Constants$Value cannot be cast to org.apache.cassandra.cql3.Lists$Value

0 ·

alexandre.dutra ♦ arsen.gasparyan.05_188257 · Jun 10, 2020 at 01:19 PM

This error seems strange, can you explain a bit more? It seems that the query you provided does not match the data you sent.

0 ·

Answer 2 · 2020-06-10T08:00:08Z

alex.ott answered Jun 10, '20 alex.ott commented Jun 10, '20

Unfortunately, CQL doesn't support binding inside the collection. If you do similar query inside the Java driver, you can get more detailed error, like:

com.datastax.driver.core.exceptions.InvalidQueryException: Invalid set literal for l: bind variables are not supported inside collection literals

To perform what you need to you need to modify your topic values to wrap that :changeLog value into a list instead of the single value, and then use query like this:

topic.Test.test.test.query=UPDATE test.test SET change_log = change_log + :changeLog WHERE id = :id ;

Transformation of single value into a list/array theoretically could be done with KSQL, or maybe custom transformations inside Kafka Connect framework.

P.S. I answered similar question on StackOverflow yesterday.

2 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

arsen.gasparyan.05_188257 · Jun 10, 2020 at 08:58 AM

Anyway with list also I am reseving the same error

{"changeLog":["Hkf"],"id":178}

topic.Test.test.test.query=UPDATE test.test SET change_log = change_log + :changeLog WHERE id = :id ;

This is likely a bug in the connector, please report. (com.datastax.kafkaconnector.DseSinkTask:153)

java.util.concurrent.ExecutionException: com.datastax.oss.driver.api.core.servererrors.ServerError: java.lang.ClassCastException: org.apache.cassandra.cql3.Constants$Value cannot be cast to org.apache.cassandra.cql3.Lists$Value

0 ·

alex.ott arsen.gasparyan.05_188257 · Jun 10, 2020 at 10:45 AM

You don't have list in your Kafka topic - you have single value that you need to transform to list using one of the methods I described (custom transformations, or KSQL).

0 ·

Answer 3 · 2020-06-10T08:26:37Z

Tomasz Lelek answered Jun 10, '20

Please take a look at the logs that are produced by the Kafka connector.

When the prepared statement is produced you should be able to see the log starting with:

[2020-06-10 10:19:13,879] ERROR WorkerSinkTask{id=dse-sink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)
java.lang.RuntimeException: Prepare failed for statement: UPDATE my_ks.test SET change_log = change_log + [:changeLog] WHERE id = :id ;

When you will scroll down at the end of the stack trace of this log you will see the underlying cause of your problems:

Caused by: com.datastax.oss.driver.api.core.servererrors.InvalidQueryException: Invalid list literal for change_log: bind variables are not supported inside collection literals

The error denotes that you cannot use bind variables (:changeLog) for collections.

It’s a limitation of the CQL itself.

I am enclosing the full exception log for a reference: Invalid_list_literal_error.txt

invalid-list-literal-error.txt (4.2 KiB)

Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

question