PLANNED MAINTENANCE

Hello, DataStax Community!

We want to make you aware of a few operational updates which will be carried out on the site. We are working hard to streamline the login process to integrate with other DataStax resources. As such, you will soon be prompted to update your password. Please note that your username will remain the same.

As we work to improve your user experience, please be aware that login to the DataStax Community will be unavailable for a few hours on:

  • Wednesday, July 15 16:00 PDT | 19:00 EDT | 20:00 BRT
  • Thursday, July 16 00:00 BST | 01:00 CEST | 04:30 IST | 07:00 CST | 09:00 AEST

For more info, check out the FAQ page. Thank you for being a valued member of our community.


question

376752150_179413 avatar image
376752150_179413 asked ·

Is restricting total partitions to a small range helpful for accelorating node repair process?

We perform a repair on a weekly basis.
at the beginning, it took 3 hours to repair
but the more data we had and the more time it took
Now it takes something like 20h to repair.
I found that creating an artificial partition key to restrict the totall range will reduce the repair for each table by at least a factor 2 or 3.


For example,

table A1 {

id

PRIMARY KEY (id)

};

table A2 {

cursor // the artificial partition key, calculated according to id manually, the range is restricted 0-7

id

PRIMARY KEY (cursor, id)

};


Why "creating an artificial partition key to restrict the totall range will reduce the repair for each table by at least a factor 2 or 3"?

Any insights would be much appreciated!

repair
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

RashidSajjad avatar image
RashidSajjad answered ·

You haven't really figured our what is causing a long repair process so artificial partition keys at this point will only be a suggestion, not a fix.

By itself, reducing number of partitions is irrelevant, the other part is what is your replication factor. the more copies of data, the more concurrency is needed.

Look at your metrics and review them against here first to figure out why is repair taking long:
https://academy.datastax.com/support-blog/diagnostic-tarball-gold-mine

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Lewisr650 avatar image
Lewisr650 answered ·

That will be an unintended outcome, but your data model should be driven by your query patterns and then enhanced to optimize partition sizes. It's reasonable that repair takes longer the more data you have. But the critical timeline is that repair is completed in less that gc_grace_seconds which is 10 days by default. You don't want to force repair to consume compute resources that might have a negative impact in read and write performance of the cluster. The important factor here is that repair is completed before tombstones are deleted and validating data consistency within partitions where tombstones will not be written forward.

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Thanks a lot for your reply! Would you please share mor insights about why "creating an artificial partition key to restrict the totall range will reduce the repair for each table by at least a factor 2 or 3"?

0 Likes 0 · ·