question

Tonny avatar image
Tonny asked Erick Ramirez edited

cassandra 在读请求较大的情况下 jvm STW 时间过长,频繁出现concurrent mode failure

cassandra采用默认配置,在4U16G规格机器上,参数配置如下:

Xms=4G,Xmx=4G,Xmn=400M,-XX:SurvivorRatio=8,-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75

较频繁出现STW持续时间达7s

2020-11-09T11:58:56.491+0000: 30505.639: Total time for which application threads were stopped: 0.0250854 seconds, Stopping threads took: 0.0000859 seconds
{Heap before GC invocations=8849 (full 140):
 par new generation   total 368640K, used 367676K [0x00000006c0000000, 0x00000006d9000000, 0x00000006d9000000)
  eden space 327680K, 100% used [0x00000006c0000000, 0x00000006d4000000, 0x00000006d4000000)
  from space 40960K,  97% used [0x00000006d6800000, 0x00000006d8f0f3a0, 0x00000006d9000000)
  to   space 40960K,   0% used [0x00000006d4000000, 0x00000006d4000000, 0x00000006d6800000)
 concurrent mark-sweep generation total 3784704K, used 3717119K [0x00000006d9000000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43699K, capacity 45400K, committed 48724K, reserved 1091584K
  class space    used 4689K, capacity 5045K, committed 6032K, reserved 1048576K
2020-11-09T11:58:56.567+0000: 30505.715: [GC (Allocation Failure) 2020-11-09T11:58:56.568+0000: 30505.715: [ParNew: 367676K->367676K(368640K), 0.0000289 secs]2020-11-09T11:58:56.568+0000: 30505.715: [CMS2020-11-09T11:58:57.556+0000: 30506.703: [CMS-concurrent-mark: 1.147/1.273 secs] [Times: user=5.09 sys=0.06, real=1.28 secs] 
 (concurrent mode failure): 3717119K->3574572K(3784704K), 7.1465703 secs] 4084796K->3574572K(4153344K), [Metaspace: 43699K->43699K(1091584K)], 7.1468865 secs] [Times: user=12.21 sys=0.06, real=7.15 secs] 
Heap after GC invocations=8850 (full 141):
 par new generation   total 368640K, used 0K [0x00000006c0000000, 0x00000006d9000000, 0x00000006d9000000)
  eden space 327680K,   0% used [0x00000006c0000000, 0x00000006c0000000, 0x00000006d4000000)
  from space 40960K,   0% used [0x00000006d6800000, 0x00000006d6800000, 0x00000006d9000000)
  to   space 40960K,   0% used [0x00000006d4000000, 0x00000006d4000000, 0x00000006d6800000)
 concurrent mark-sweep generation total 3784704K, used 3574572K [0x00000006d9000000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43699K, capacity 45400K, committed 48724K, reserved 1091584K
  class space    used 4689K, capacity 5045K, committed 6032K, reserved 1048576K
}
2020-11-09T11:59:03.714+0000: 30512.862: Total time for which application threads were stopped: 7.1505311 seconds, Stopping threads took: 0.0022926 seconds

有什么比较有效的调优办法,降低STW的时长

jvm
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

4GB对于Cassandra工作负载而言太小了。对于16GB的RAM,我建议将heap max size增大到8GB,将minimum size增大到800MB。

让我们看看它在此配置下的性能。

4GB is too small for C workloads. With 16GB of RAM, I recommend increasing the heap max size to 8GB and minimum size to 800MB.

Let's see how it performs with this configuration. Cheers!

Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.