dsbulk load json issue

Question

question

ulavapalle.meghamala_184243 asked Jan 17, '20 Erick Ramirez commented Jan 21, '20

dsbulk load json issue

We are facing an issue while loading data from json file into cassandra cluster. Here is the command:

dsbulk-1.4.1/bin/dsbulk load  -url  ex1.json  -h 'x.x.x.x,x.x.x.x,x.x.x.x' -k xxxxxx -t users -u xxxxx -p xxxxxx --driver.advanced.auth-provider.class PlainTextAuthProvider

ex1.json:

{

"uidx": "123456"

}

users table:

CREATE TABLE users (uidx text PRIMARY KEY);

From mapping-errors.log
Resource: file:/home/deploy/ex1.json
Position: 1
Source:         "uidx": "123456"\u000a
com.datastax.dsbulk.engine.internal.schema.InvalidMappingException: Required field uidx (mapped to column uidx) was missing from record. Please remove it from the mapping or set schema.allowMissingFields to true.
        at com.datastax.dsbulk.engine.internal.schema.InvalidMappingException.missingField(InvalidMappingException.java:38)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.ensureAllFieldsPresent(DefaultRecordMapper.java:185)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.map(DefaultRecordMapper.java:106)
        at java.lang.Thread.run(Thread.java:748) [22 skipped]

Resource: file:/home/deploy/ex1.json
Position: 2
Source: }\u000a
com.datastax.dsbulk.engine.internal.schema.InvalidMappingException: Required field uidx (mapped to column uidx) was missing from record. Please remove it from the mapping or set schema.allowMissingFields to true.
        at com.datastax.dsbulk.engine.internal.schema.InvalidMappingException.missingField(InvalidMappingException.java:38)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.ensureAllFieldsPresent(DefaultRecordMapper.java:185)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.map(DefaultRecordMapper.java:106)
        at java.lang.Thread.run(Thread.java:748) [22 skipped]

Can you please let us know what are we doing incorrectly here?

Thanks !

dsbulk json load

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-01-17T10:37:30Z

Erick Ramirez answered Jan 17, '20 Erick Ramirez commented Jan 21, '20

@ulavapalle.meghamala_184243 Similar to the other question you posted, your JSON file contains a character which isn't compatible with Unix.

Please try to convert the file to Unix using a utility like dos2unix to replace the DOS/Mac line breaks with Unix line breaks. That should allow you to bulk load the data. Cheers!

2 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

ulavapalle.meghamala_184243 · Jan 20, 2020 at 09:41 PM

Still having trouble. Looks like the command is stuck at this stage:

$ bin/dsbulk load  --connector.json.urlfile ex1.json  -h '...' -k xxx -t users -u xxx -p xxx --driver.advanced.auth-provider.class PlainTextAuthProvider 
...
total | failed | rows/s | p50ms | p99ms | p999ms | batches
    0 |      0 |      0 |  0.00 |  0.00 |   0.00 |    0.00

0 ·

Erick Ramirez ♦♦ ulavapalle.meghamala_184243 · Jan 21, 2020 at 08:16 AM

You didn't specify the connector name and your use of urlfile is incorrect. Instead try:

$ bin/dsbulk load --connector.name json --connector.json.url /path/to/ex1.json _the_rest_of_your_options_

Cheers!

1 ·

question