From our monitor tool, we can see the read latency is very high like 500ms -600ms. the read local quorum is like 5-10 ms. we dont know what caused this. the workload is not very heavy. this is the nodetool cfstats. we think it might be related with the consistency level setting on driver side ( which cause some of the query was not using local quorum). then the coordinate node redirect the read to other cassandra node in anoter dc. then we got a very high read latency. is this possible? Please help analyze it. Thanks a lot.
Table: clusters_by_id SSTable count: 2 Space used (live): 10893 Space used (total): 10893 Space used by snapshots (total): 0 Off heap memory used (total): 384 SSTable Compression Ratio: 0.864963503649635 Number of partitions (estimate): 4 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 0 Local read count: 90016 Local read latency: 0.293 ms Local write count: 0 Local write latency: NaN ms Pending flushes: 0 Percent repaired: 100.0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 352 Bloom filter off heap memory used: 336 Index summary off heap memory used: 32 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 18 Compacted partition maximum bytes: 258 Compacted partition mean bytes: 80 Average live cells per slice (last five minutes): 1.0 Maximum live cells per slice (last five minutes): 1 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 Dropped Mutations: 0
