Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

david.cao avatar image
david.cao asked ·

Why is read latency so high in our cluster?

From our monitor tool, we can see the read latency is very high like 500ms -600ms. the read local quorum is like 5-10 ms. we dont know what caused this. the workload is not very heavy. this is the nodetool cfstats. we think it might be related with the consistency level setting on driver side ( which cause some of the query was not using local quorum). then the coordinate node redirect the read to other cassandra node in anoter dc. then we got a very high read latency. is this possible? Please help analyze it. Thanks a lot.屏幕快照-2020-11-13-上午91028.png屏幕快照-2020-11-13-上午90438.png

Table: clusters_by_id
SSTable count: 2
Space used (live): 10893
Space used (total): 10893
Space used by snapshots (total): 0
Off heap memory used (total): 384
SSTable Compression Ratio: 0.864963503649635
Number of partitions (estimate): 4
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 90016
Local read latency: 0.293 ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 352
Bloom filter off heap memory used: 336
Index summary off heap memory used: 32
Compression metadata off heap memory used: 16
Compacted partition minimum bytes: 18
Compacted partition maximum bytes: 258
Compacted partition mean bytes: 80
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
Dropped Mutations: 0
performance
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

david.cao avatar image
david.cao answered ·

@Erick Ramirez Thanks for the help.

I found that a very strange thing. I enabled consistency local quorum in cqlsh on cassandra node. but the query sometimes goes to another dc. sometimes in local dc. I have no idea why the consistency level doesn;t work even on cassandra node. is this related with snitch? Thanks.

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image
Erick Ramirez answered ·

There isn't enough information in your post that gives meaningful data for analysis.

As a starting point, you need to work out which queries are taking a long time. Once you have the problematic queries, I suggest manually running them in cqlsh with the same consistency as the app and enable TRACING.

The trace output will give you clues as to why the query is slow. Cheers!

3 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

from the bad query log, i can see that the local cassandra nodes responsed correctly. but it still redirects to the nodes in other dc..

0 Likes 0 · ·