dsbulk load json issue

Question

question

ulavapalle.meghamala_184243 asked · Jan 17 at 10:28 AM

dsbulk load json issue

We are facing an issue while loading data from json file into cassandra cluster. Here is the command:

dsbulk-1.4.1/bin/dsbulk load  -url  ex1.json  -h 'x.x.x.x,x.x.x.x,x.x.x.x' -k xxxxxx -t users -u xxxxx -p xxxxxx --driver.advanced.auth-provider.class PlainTextAuthProvider

ex1.json:

{

"uidx": "123456"

}

users table:

CREATE TABLE users (uidx text PRIMARY KEY);

From mapping-errors.log
Resource: file:/home/deploy/ex1.json
Position: 1
Source:         "uidx": "123456"\u000a
com.datastax.dsbulk.engine.internal.schema.InvalidMappingException: Required field uidx (mapped to column uidx) was missing from record. Please remove it from the mapping or set schema.allowMissingFields to true.
        at com.datastax.dsbulk.engine.internal.schema.InvalidMappingException.missingField(InvalidMappingException.java:38)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.ensureAllFieldsPresent(DefaultRecordMapper.java:185)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.map(DefaultRecordMapper.java:106)
        at java.lang.Thread.run(Thread.java:748) [22 skipped]

Resource: file:/home/deploy/ex1.json
Position: 2
Source: }\u000a
com.datastax.dsbulk.engine.internal.schema.InvalidMappingException: Required field uidx (mapped to column uidx) was missing from record. Please remove it from the mapping or set schema.allowMissingFields to true.
        at com.datastax.dsbulk.engine.internal.schema.InvalidMappingException.missingField(InvalidMappingException.java:38)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.ensureAllFieldsPresent(DefaultRecordMapper.java:185)
        at com.datastax.dsbulk.engine.internal.schema.DefaultRecordMapper.map(DefaultRecordMapper.java:106)
        at java.lang.Thread.run(Thread.java:748) [22 skipped]

Can you please let us know what are we doing incorrectly here?

Thanks !

dsbulk json load

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-01-17T10:37:30Z

Erick Ramirez answered · Jan 17 at 10:37 AM

@ulavapalle.meghamala_184243 Similar to the other question you posted, your JSON file contains a character which isn't compatible with Unix.

Please try to convert the file to Unix using a utility like dos2unix to replace the DOS/Mac line breaks with Unix line breaks. That should allow you to bulk load the data. Cheers!

2 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

ulavapalle.meghamala_184243 · Jan 20 at 09:41 PM

Still having trouble. Looks like the command is stuck at this stage:

$ bin/dsbulk load  --connector.json.urlfile ex1.json  -h '...' -k xxx -t users -u xxx -p xxx --driver.advanced.auth-provider.class PlainTextAuthProvider 
...
total | failed | rows/s | p50ms | p99ms | p999ms | batches
    0 |      0 |      0 |  0.00 |  0.00 |   0.00 |    0.00

0 · ·

Erick Ramirez ♦♦ ulavapalle.meghamala_184243 · Jan 21 at 08:16 AM

You didn't specify the connector name and your use of urlfile is incorrect. Instead try:

$ bin/dsbulk load --connector.name json --connector.json.url /path/to/ex1.json _the_rest_of_your_options_

Cheers!

1 · ·

question