Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

ulavapalle.meghamala_184243 avatar image
ulavapalle.meghamala_184243 asked ·

dsbulk load json issue

We are facing an issue while loading data from json file into cassandra cluster. Here is the command:

dsbulk-1.4.1/bin/dsbulk load  -url  ex1.json  -h 'x.x.x.x,x.x.x.x,x.x.x.x' -k xxxxxx -t users -u xxxxx -p xxxxxx --driver.advanced.auth-provider.class PlainTextAuthProvider

ex1.json:

{

"uidx": "123456"

}

users table:

CREATE TABLE users (uidx text PRIMARY KEY);


From mapping-errors.log
Resource: file:/home/deploy/ex1.json
Position: 1
Source:         "uidx": "123456"\u000a
com.datastax.dsbulk.engine.internal.schema.InvalidMappingException: Required field uidx (mapped to column uidx) was missing from record. Please remove it from the mapping or set schema.allowMissingFields to true.
        at com.datastax.dsbulk.engine.internal.schema.InvalidMappingException.missingField(InvalidMappingException.java:38)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.ensureAllFieldsPresent(DefaultRecordMapper.java:185)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.map(DefaultRecordMapper.java:106)
        at java.lang.Thread.run(Thread.java:748) [22 skipped]

Resource: file:/home/deploy/ex1.json
Position: 2
Source: }\u000a
com.datastax.dsbulk.engine.internal.schema.InvalidMappingException: Required field uidx (mapped to column uidx) was missing from record. Please remove it from the mapping or set schema.allowMissingFields to true.
        at com.datastax.dsbulk.engine.internal.schema.InvalidMappingException.missingField(InvalidMappingException.java:38)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.ensureAllFieldsPresent(DefaultRecordMapper.java:185)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.map(DefaultRecordMapper.java:106)
        at java.lang.Thread.run(Thread.java:748) [22 skipped]


Can you please let us know what are we doing incorrectly here?


Thanks !

dsbulkjsonload
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

@ulavapalle.meghamala_184243 Similar to the other question you posted, your JSON file contains a character which isn't compatible with Unix.

Please try to convert the file to Unix using a utility like dos2unix to replace the DOS/Mac line breaks with Unix line breaks. That should allow you to bulk load the data. Cheers!

2 comments Share
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

ulavapalle.meghamala_184243 avatar image ulavapalle.meghamala_184243 ·

Still having trouble. Looks like the command is stuck at this stage:

$ bin/dsbulk load  --connector.json.urlfile ex1.json  -h '...' -k xxx -t users -u xxx -p xxx --driver.advanced.auth-provider.class PlainTextAuthProvider 
...
total | failed | rows/s | p50ms | p99ms | p999ms | batches
    0 |      0 |      0 |  0.00 |  0.00 |   0.00 |    0.00
0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ ulavapalle.meghamala_184243 ·

You didn't specify the connector name and your use of urlfile is incorrect. Instead try:

$ bin/dsbulk load --connector.name json --connector.json.url /path/to/ex1.json _the_rest_of_your_options_

Cheers!

1 Like 1 ·