question

nyjdams_136971 avatar image
nyjdams_136971 asked Erick Ramirez edited

Using the VM image in DS201, why did I lose node1 when I decommissioned node3?

Hi - I hope someone can help

I had three nodes as below, one of them in the wrong datacenter:

ubuntu@ds201-node1:~/node3$ bin/nodetool status
Datacenter: Cassandra
=====================
Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address    Load       Tokens       Owns    Host ID                               Rack
UN  127.0.0.3  233.64 KiB  128          ?       b84ac6a8-2895-4f2e-bfc8-8478df2bcebc  rack1
Datacenter: east-side
=====================
Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address    Load       Tokens       Owns    Host ID                               Rack
UN  127.0.0.2  242.24 KiB  128          ?       99ec4273-7960-40fe-ae3a-574c9981138f  hakuna-matata
Datacenter: west-side
=====================
Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address    Load       Tokens       Owns    Host ID                               Rack
UN  127.0.0.1  205.01 KiB  128          ?       2708096e-7256-469c-8cb9-1d1e43c6e837  hakuna-matata
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

I issued the following command, intending to decommission node 3, from within the node 3 filepath:

ubuntu@ds201-node1:~/node3/bin$ nodetool -h 127.0.0.3 decommission -f

Then immediately after, I ran nodetool status, and as you can see, I've lost node 1.

ubuntu@ds201-node1:~/node3/bin$ nodetool status
Datacenter: Cassandra
=====================
Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address    Load       Tokens       Owns    Host ID                               Rack
UN  127.0.0.3  233.64 KiB  128          ?       b84ac6a8-2895-4f2e-bfc8-8478df2bcebc  rack1
Datacenter: east-side
=====================
Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address    Load       Tokens       Owns    Host ID                               Rack
UN  127.0.0.2  227.68 KiB  128          ?       99ec4273-7960-40fe-ae3a-574c9981138f  hakuna-matata
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

What have I done wrong?

cassandrads201
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered nyjdams_136971 commented

@nyjdams_136971 In my answer in your question #3981, I failed to take into account the differing JMX configurations that allowed the nodes to be deployed on the same VM server. I have since downloaded the VM in the DS201 Foundations course on DataStax Academy so I could check how the VMs are configured.

In my sample installation, each DSE instance is bound to a different JMX port:

java     5287 ubuntu  492u  IPv4  41481      0t0  TCP *:7199 (LISTEN)
java     5287 ubuntu  500u  IPv4  41983      0t0  TCP 127.0.0.1:7000 (LISTEN)
java     5988 ubuntu  506u  IPv4  46306      0t0  TCP *:7299 (LISTEN)
java     5988 ubuntu  554u  IPv4  46959      0t0  TCP 127.0.0.2:7000 (LISTEN)
java     6561 ubuntu  492u  IPv4  52185      0t0  TCP *:7399 (LISTEN)
java     6561 ubuntu  500u  IPv4  52773      0t0  TCP 127.0.0.3:7000 (LISTEN)

The nodetool utility connects to the JMX agent. I failed to note that because of the special configuration of the VM for the DS201 course, it is necessary to specify not just the host IP but the corresponding JMX port as well. So for connecting to the first node, the format of the command is:

$ nodetool -h 127.0.0.1 -p 7199 <command>

For the other 2 nodes:

$ nodetool -h 127.0.0.2 -p 7299 <command>
$ nodetool -h 127.0.0.3 -p 7399 <command>

Therefore to decommission node 3, we needed to run:

$ nodetool -h 127.0.0.3 -p 7399 decommission -f

I apologise that I provided you an answer in #3981 without having checked the configuration of the nodes. Cheers!

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

nyjdams_136971 avatar image nyjdams_136971 commented ·

Thanks Erick!

0 Likes 0 ·