Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

debasis.tcs_69445 avatar image
debasis.tcs_69445 asked Erick Ramirez edited

Getting "RuntimeException: Could not create snapshot" when running repairs

Hi Team

I hope you are doing well.

We have 2 DC. One DC is having 26 nodes and once DC has 16 nodes. We recently added 10 Nodes in the DC which has 26 nodes ( previously it was 16). After node addition we see repair is taking huge time (around 1 hr) though we have only 2 GB data . Cassandra version is 3.11.5 .

nohup nodetool -u casmon -pw PjKitkH9oMe repair -seq -j 4 2>&1 >> /root/logs/mslogs/`hostname`_`date +\%Y\%m\%d\%H\%M`.out &

We are getting the following error during repair

WARN [RepairJobTask:3] 2021-03-30 00:16:53,221 RepairJob.java:131 - [repair #e678f190-9127-11eb-aa31-75dbf09fbdd0] feature_limit sync failed
ERROR [Repair#2:1] 2021-03-30 00:16:53,222 CassandraDaemon.java:228 - Exception in thread Thread[Repair#2:1,5,RMI Runtime]
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Could not create snapshot at /X.X.X.X
    at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525) ~[guava-18.0.jar:na]
    at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
    at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:138) ~[apache-cassandra-3.11.5.jar:3.11.5]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_251]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_251]
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) ~[apache-cassandra-3.11.5.jar:3.11.5]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_251]
Caused by: java.lang.RuntimeException: Could not create snapshot at /X.X.X.X
    at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:80) ~[apache-cassandra-3.11.5.jar:3.11.5]
    at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:47) ~[apache-cassandra-3.11.5.jar:3.11.5]
repair
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

debasis.tcs_69445 avatar image debasis.tcs_69445 ·

Hi Experts , any comment please.

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

As the error suggests, the repair job failed to create a snapshot on one of the nodes:

Caused by: java.lang.RuntimeException: Could not create snapshot at /X.X.X.X
    at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:80) ~[apache-cassandra-3.11.5.jar:3.11.5]

A snapshot in Cassandra simply creates hardlinks to the original inodes. This is a filesystem operation and isn't internal to Cassandra. I've explained it in a bit more detail in question #5967.

You need to review the logs on the node in order to determine the cause of the snapshot failure. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.