Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

ranjeet_ranjee avatar image
ranjeet_ranjee asked ·

Cassandra not returning consistent results when READ consistency level is LOCAL_ONE

I have a 2 node Cassandra server. The consistency level for reads is LOCAL_ONE. However, when one of the nodes goes down, my read query returns no result. If both are up, then the results return fine as expected

Both the node is in the same DC.


[root@cassandra-7 ~]# nodetool status
Datacenter: singaporedo
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens       Owns    Host ID                               Rack
UN  Node-1-IP   406.66 GiB  1            ?       9159fc01-f08a-4334-bb29-5b3cf7d5727f  rack-1
UN  Node-2-IP  380.15 GiB  1            ?       2aca8bd7-f211-4913-888f-1749d7e000a4  rack-1


Table Description


adsizz@cqlsh:adserv> describe conversion_by_uid;

CREATE TABLE adserv.conversion_by_uid (
    uid uuid PRIMARY KEY,
    android_id text,
    app_version text,
    brand text,
    browser text,
    camp_id int,
    carrier text,
    city text,
    click_id text,
    clk_browser text,
    clk_carrier text,
    clk_city text,
    clk_country text,
    clk_device_type text,
    clk_event_code int,
    clk_gaid text,
    clk_id text,
    clk_idfa text,
    clk_ip text,
    clk_is_mob boolean,
    clk_is_tab boolean,
    clk_platform text,
    clk_platform_mkr text,
    clk_platform_ver text,
    clk_province text,
    clk_redirected_frm_camp int,
    clk_redirected_frm_cr int,
    clk_redirection_valid boolean,
    clk_referer text,
    clk_sub_id1 text,
    clk_sub_id2 text,
    clk_sub_id3 text,
    clk_sub_id4 text,
    clk_sub_id5 text,
    clk_sub_pub text,
    clk_timestamp timestamp,
    clk_uid text,
    clk_unique_id text,
    clk_user_agent text,
    commission decimal,
    conv_city text,
    conv_country text,
    conv_ip text,
    conv_os text,
    conv_os_ver text,
    conv_platform text,
    conv_platform_ver text,
    conv_type int,
    country text,
    creative_id int,
    device_type text,
    event_code int,
    event_name text,
    event_type text,
    gaid text,
    idfa text,
    imei text,
    ip text,
    is_mob boolean,
    is_tab boolean,
    order_id text,
    order_val text,
    partners_trans_id text,
    payout decimal,
    platform text,
    platform_mkr text,
    platform_ver text,
    province text,
    pub_id int,
    revenue text,
    sdk_ver text,
    sub_pub text,
    time_gap_seconds int,
    timestamp timestamp,
    trans_id1 text,
    trans_id2 text,
    trans_id3 text,
    trans_id4 text,
    trans_id5 text,
    user_agent text,
    wifi text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.DeflateCompressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';


cassandra
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image
Erick Ramirez answered ·

This is expected behaviour if you're most likely writing with a consistency level of LOCAL_ONE as well. The issue is that the replicas are out-of-sync and you need to run repairs regularly.

I'm aware that your on the startup program but 2 nodes in production isn't ideal. We recommend you have at least 3 replicas in each of the data centres so your application can tolerate an outage to 1 node and can use a stronger consistency of LOCAL_QUORUM (recommended consistency).

You've posted several questions now about things not working when you bring a node down but your tests are invalid in your current setup. You need to address the underlying issues or your efforts are going to be futile.

At the very least you need to verify that the DSE configuration is identical on both nodes and run repairs regularly at least once a week. Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

saravanan.chinnachamy_185977 avatar image
saravanan.chinnachamy_185977 answered ·

@ranjeet_ranjee

Cassandra stores data replicas on multiple nodes to ensure reliability and fault tolerance. The replication strategy for each keyspace determines the nodes where replicas are placed.

The total number of replicas for a keyspace across a Cassandra cluster is referred to as the keyspace's replication factor.

A replication factor of one means that there is only one copy of each row in the Cassandra cluster. A replication factor of two means there are two copies of each row, where each copy is on a different node.

You can inspect which nodes has the data that you inserted into the table using the following command.

nodetool <options> getendpoints -- <keyspace> <table> key

$ nodetool getendpoints killervideo emp_by_id "1003"
10.142.0.3
10.142.0.2
10.142.0.4

So please make sure to inspect the following and try again.

  1. What is your replication factor for the schema in question?
  2. Where does the data live (nodes)?

In your case, please set the RF=2 and then test again.


4 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

If I check RF its 2.

adsz@cqlsh> describe asderv;

CREATE KEYSPACE adserv WITH replication = {'class': 'NetworkTopologyStrategy', 'singaporedo': '2'}  AND durable_writes = true;


0 Likes 0 · ·

Can you also run the following command and post your response?

 nodetool  getendpoints asderv <table> <partiton_key>

Also please post your table details.

0 Likes 0 · ·
ranjeet_ranjee avatar image ranjeet_ranjee saravanan.chinnachamy_185977 ·

Here you go details.

[root@cassandra-8 ~]# nodetool  getendpoints adserv conversion_by_uid 3b5d83cf-13eb-41c1-93ce-7ccc6db5bdba

Response

Node-1-Public_ip 
Node-2-Public_ip 


Please check my question for the table description.


0 Likes 0 · ·
Show more comments