My DSBULK load suddenly is running slow. When I turned on debug, i see that max batch is 2, ideally it should be 32. I have not changed anything. I am not sure why the max is changing to 2.
This question was
closed
by
Erick Ramirez for the following reason: The question is answered, right answer was accepted
Why is DSBulk reporting 2 max batches when maxBatchStatements is 32 by default?
Were you observing a better batch performance before on the same data? Because the efficiency of DSBulk's batching mechanism depends very much on the data being loaded. In general it works much better when the data to load is sorted by partition key and the row sizes are small.
Here is what you can try:
Increase
dsbulk.batch.bufferSizeto e.g. 256 or 512. Take care, this could take up all the available heap.If that doesn't help you could try setting
dsbulk.batch.modetoREPLICA_SET. But beware that this will make your latencies much worse, so you also should setdsbulk.batch.maxBatchStatementsto a low value to avoid timeouts or errors (e.g. 5); then, if this works, you could slowly increasemaxBatchStatementsto get the best throughput while keeping latencies acceptable.
Hi Alexandre, you are right on the partition key sorting. I discovered today that the latest files that we have tried to load were not ordered by partition key and thats when we saw DSBULK using less number of batches between 1 and 2. Earlier it was sorted by partition key when we saw the number of batches going upto 14
7 People are following this question.
