question

ulavapalle.meghamala_184243 avatar image
ulavapalle.meghamala_184243 asked Erick Ramirez edited

Loading data in JSON format with DSBulk fails with mapping errors

We are facing an issue while loading data from json file into cassandra cluster. Here is the command:

dsbulk-1.4.1/bin/dsbulk load  -url  ex1.json  -h 'x.x.x.x,x.x.x.x,x.x.x.x' -k xxxxxx -t users -u xxxxx -p xxxxxx --driver.advanced.auth-provider.class PlainTextAuthProvider

ex1.json: { "uidx": "123456" }

users table: CREATE TABLE users (uidx text PRIMARY KEY);

From mapping-errors.log
Resource: file:/home/deploy/ex1.json
Position: 1
Source:         "uidx": "123456"\u000a
com.datastax.dsbulk.engine.internal.schema.InvalidMappingException: Required field uidx (mapped to column uidx) was missing from record. Please remove it from the mapping or set schema.allowMissingFields to true.
        at com.datastax.dsbulk.engine.internal.schema.InvalidMappingException.missingField(InvalidMappingException.java:38)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.ensureAllFieldsPresent(DefaultRecordMapper.java:185)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.map(DefaultRecordMapper.java:106)
        at java.lang.Thread.run(Thread.java:748) [22 skipped]

Resource: file:/home/deploy/ex1.json
Position: 2
Source: }\u000a
com.datastax.dsbulk.engine.internal.schema.InvalidMappingException: Required field uidx (mapped to column uidx) was missing from record. Please remove it from the mapping or set schema.allowMissingFields to true.
        at com.datastax.dsbulk.engine.internal.schema.InvalidMappingException.missingField(InvalidMappingException.java:38)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.ensureAllFieldsPresent(DefaultRecordMapper.java:185)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.map(DefaultRecordMapper.java:106)
        at java.lang.Thread.run(Thread.java:748) [22 skipped]

Can you please let us know what are we doing incorrectly here?

Thanks !

dsbulk
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

@ulavapalle.meghamala_184243 Similar to the other question you posted, your JSON file contains a character which isn't compatible with Unix.

Please try to convert the file to Unix using a utility like dos2unix to replace the DOS/Mac line breaks with Unix line breaks. That should allow you to bulk load the data. Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

ulavapalle.meghamala_184243 avatar image ulavapalle.meghamala_184243 commented ·

Still having trouble. Looks like the command is stuck at this stage:

$ bin/dsbulk load  --connector.json.urlfile ex1.json  -h '...' -k xxx -t users -u xxx -p xxx --driver.advanced.auth-provider.class PlainTextAuthProvider 
...
total | failed | rows/s | p50ms | p99ms | p999ms | batches
    0 |      0 |      0 |  0.00 |  0.00 |   0.00 |    0.00
0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ ulavapalle.meghamala_184243 commented ·

You didn't specify the connector name and your use of urlfile is incorrect. Instead try:

$ bin/dsbulk load --connector.name json --connector.json.url /path/to/ex1.json _the_rest_of_your_options_

Cheers!

1 Like 1 ·