question

arunkolluri07_188529 avatar image
arunkolluri07_188529 asked Erick Ramirez answered

Streaming issues migrating C* 2.1 cluster to cloud

Hi Team,

I migrating a on prem cluster(Apache Cassandra 2.1.21) to cloud which has 4TB of dataI have build the new dc in cloud but the rebuild is not going well. It is getting struck when it receives large data. Also if i run the rebuild simultaneously i get streaming error. So far in the past week the new dc got about 200gb of data. Is there any configurations needed to increase the rebuild rate. And how can i run the rebuild simultaneously on 2or 3 nodes with out the error.

nodetool netstats
Mode: NORMAL
Rebuild 90d56d90-33bb-11ec-aaf0-551339603e47
  /10.3.17.192
    Receiving 142 files, 157422954482 bytes total. Already received 24 files, 36421731930 bytes total
error: Error while rebuilding node: Stream failed
-- StackTrace --
java.lang.RuntimeException: Error while rebuilding node: Stream failed
 at org.apache.cassandra.service.StorageService.rebuild(StorageService.java:1162)
 at org.apache.cassandra.service.StorageService.rebuild(StorageService.java:1049)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
 at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)


migration
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

With limited information and not knowing the root cause, it's impossible for us to know what to recommend.

The issue you're running into is very involved and is beyond what I can provide in a Q&A forum since what you need is support assistance.

Feel free to log a ticket with DataStax Support if you have a valid subscription so one of our engineers can assist you. Cheers!

Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.