DS210, Exercise: Add and Remove a Node to/from the Cluster
One of the goal of this exercise is to run a cassandra-stress
on a singe-node cluster. Then add a 2nd node, redo the same stress test and observe that the performance has improved.
I found the opposite, ie a degradation instead of improvement in IO performance. There might be a plausible explanation. Hope you can help to clarify.
My lab setup is different than that of the exercise (separate machines in cloud). In my case, everything (cassandra-stress and all the nodes) runs on the same physical machine (Ubuntu 20.04, 32GB, SSD). Each node is installed like in DS201:
dse-6.8.1-bin.tar.gz is uncompressed in ~/bin/DSE6/nodeX
configured by running script labwork/config_node X
MAX_HEAP_SIZE=1GB, HEAP_NEWSIZE=128MB
I my case, the 2-nodes cluster has slightly lower performance than the single-node cluster. The results are not fluke, repeated runs gave similar trends. I suppose the lower performance of the 2nodes scenario is b/c the 2 nodes, running in the same machine, compete for some common resources (maybe disk IO, memory bandwdth ?)
QUESTION: I am not entirely sure how to interpret the results as why 2-nodes are slower than 1node. Because the machine still have execess CPU/RAM available even when 2 nodes were running. Can you please help for some explanation?
Below is the summary of cassandra-stress:
./node1/resources/cassandra/tools/bin/cassandra-stress \
user profile=./DS210_Operations/labwork/TestProfileEdited.yaml \
"ops(insert=1,get_user=3)" -node 127.0.0.1 -port native=9041
#----single-node cluster
Op rate : 33,071 op/s [get_user: 24,774 op/s, insert: 8,297 op/s]
Partition rate : 32,975 pk/s [get_user: 24,678 pk/s, insert: 8,297 pk/s]
Row rate : 32,975 row/s [get_user: 24,678 row/s, insert: 8,297 row/s]
Latency mean : 12.170 ms [get_user: 12.3 ms, insert: 11.7 ms]
Latency median : 9.486 ms [get_user: 9.6 ms, insert: 9.2 ms]
#---2-nodes cluster
Op rate : 30,747 op/s [get_user: 23,063 op/s, insert: 7,684 op/s]
Partition rate : 30,676 pk/s [get_user: 22,992 pk/s, insert: 7,684 pk/s]
Row rate : 30,676 row/s [get_user: 22,992 row/s, insert: 7,684 row/s]
Latency mean : 29.405 ms [get_user: 30.0 ms, insert: 27.8 ms]
Latency median : 23.446 ms [get_user: 24.1 ms, insert: 21.6 ms]