question

jo.pavankumar_110800 avatar image
jo.pavankumar_110800 asked Erick Ramirez edited

Why can I still connect with cqlsh to node1 in the Consistency exercise of the DS201 course?

HI Team as mentioned in DS201 Consistency Exercise i bought one node (node1) in my case down , i have observed the following behavior which deviating from the exercise steps .

Question 1) By bringing node1(127.0.0.1) down , exercise says cqlsh should not connect IP that is (127.0.0.1) ,but am still able to connect on 127.0.0.1 .(is this expected)??

( see bold 127.0.0.1 is down)

ubuntu@ds201-node1:~/node3/bin$ ./dsetool status
Warning: Not able to read Analytics Master data, SparkMaster data may not be correct.
DC: east-side       Workload: Cassandra      Graph: no
======================================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Server ID          Address    Load        Owns VNodes Rack        Health [0,1]
UN  08-00-27-32-1E-DD  127.0.0.2  133.18 KiB  ?    128    hakuna-mata 0.30
DC: west-side       Workload: Cassandra      Graph: no
======================================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Server ID           Address    Load        Owns VNodes Rack        Health [0,1]
DN  08-00-27-32-1E-DD  127.0.0.1  114.67 KiB  ?    128    hakuna-mata 0.30 
UN  08-00-27-32-1E-DD  127.0.0.3  124.36 KiB  ?    128    hakuna-mata 0.30

Step 2: when i am trying to connect cqlsh on node3 am still able to to connect on 127.0.0.1 with port number 9042

ubuntu@ds201-node1:~/node3/bin$ ./cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 4.0.0.2284 | DSE 6.0.0 | CQL spec 3.4.5 | DSE protocol v2]
Use HELP for help.
cqlsh>

Read and Write Consistency

Question 2a (step 11)

Exercise says making Consistency level TWO and execute the following query outcome :insert query should fail.( step 11 in the exercise), but am able to insert the data successfully is that expected ?

INSERT INTO killrvideo.videos_by_tag (tag, added_date, video_id, title) VALUES ('cassandra', '2016-2-8', uuid(), 'Cassandra Write Test');

Actual Result:

insert was successful (Observe Bold)

cqlsh:killrvideo> INSERT INTO killrvideo.videos_by_tag (tag, added_date, video_id, title) ... VALUES ('cassandra', '2016-2-8', uuid(), 'Cassandra Write Test');
cqlsh:killrvideo> select * from videos_by_tag ;
NoHostAvailable: .
cqlsh:killrvideo> CONSISTENCY ONE;
Consistency level set to ONE.
cqlsh:killrvideo> select * from videos_by_tag ;tag | added_date | video_id | title
-----------+---------------------------------+--------------------------------------+------------------------------
datastax | 2013-10-16 00:00:00.000000+0000 | 4845ed97-14bd-11e5-8a40-8338255b7e33 | DataStax Studio
datastax | 2013-04-16 00:00:00.000000+0000 | 5645f8bd-14bd-11e5-af1a-8638355b8e3a | What is DataStax Enterprise?
cassandra | 2016-02-08 00:00:00.000000+0000 | 58c6816d-e21a-4114-8a0e-44add01f1dce | Me Lava Cassandra
cassandra | 2016-02-08 00:00:00.000000+0000 | fbc4498e-2b2b-4a72-aff2-9f208f027f06 | Cassandra Write Test
cassandra | 2014-01-29 00:00:00.000000+0000 | 1645ea59-14bd-11e5-a993-8138354b7e31 | Cassandra History
cassandra | 2013-03-17 00:00:00.000000+0000 | 3452f7de-14bd-11e5-855e-8738355b7e3a | Cassandra Intro
cassandra | 2012-04-03 00:00:00.000000+0000 | 245e8024-14bd-11e5-9743-8238356b7e32 | Cassandra & SSDs

(By changing consistency level i am able to see the inserted record)

Note : Still read was failing with No Host available Exception (for consistency level two which is meeting with Exercise outcome)

cqlsh:killrvideo> CONSISTENCY TWO;
Consistency level set to TWO.
cqlsh:killrvideo> select * from videos_by_tag ;
NoHostAvailable:
ds201
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

Background

When you run the commands dsetool status or nodetool status, the IP addresses in the output are the private interfaces used for internode/gossip communication (listen_address) configured in (cassandra.yaml).

But when you connect to the nodes (cqlsh is just another DB client), clients connect to the publicly accessible CQL client interface (native_transport_address).

Cause

In the DS201 course VM, the nodes are configured with the following loopback addresses for gossip:

$ for _node in node*; do echo "$_node - `grep ^listen_address $_node/resources/cassandra/conf/cassandra.yaml`"; done
node1 - listen_address: 127.0.0.1
node2 - listen_address: 127.0.0.2
node3 - listen_address: 127.0.0.3

But all nodes are listening for client requests on 127.0.0.1:

$ for _node in node*; do echo "$_node - `grep ^native_transport_address $_node/resources/cassandra/conf/cassandra.yaml`"; done
node1 - native_transport_address: localhost
node2 - native_transport_address: localhost
node3 - native_transport_address: localhost

with differing CQL ports configured to differentiate each node:

$ for _node in node*; do echo "$_node - `grep ^native_transport_port $_node/resources/cassandra/conf/cassandra.yaml`"; done
node1 - native_transport_port: 9041
node2 - native_transport_port: 9042
node3 - native_transport_port: 9043

In my sample VM, here is a list of the CQL client IPs and ports for the three DSE nodes:

$ sudo lsof -nPi | grep LISTEN | grep 904
java      1544 ubuntu  481u  IPv4  19709      0t0  TCP 127.0.0.1:9041 (LISTEN)
java      2135 ubuntu  485u  IPv4  24848      0t0  TCP 127.0.0.1:9042 (LISTEN)
java     18820 ubuntu  480u  IPv4  45516      0t0  TCP 127.0.0.1:9043 (LISTEN)

Solution

To respond to your questions directly, you have incorrectly assumed that you are connecting to node1 with cqlsh. You are in fact connected to node2 (127.0.0.1:9042). If you wanted to connect to node1 with cqlsh, you needed to run the following command:

$ cqlsh 127.0.0.1 9041

Specifically, you needed to specify both the client IP 127.0.0.1 and CQL port 9041. Otherwise, it would default to port 9042. I'm sorry that the exercise is not clear about this.

With respect to the CQL queries I suspect that you inserted the data with a consistency of ONE instead of TWO. Please try again. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.