Seeing few queries taking more than 500 milliseconds and these queries are logged with the message
slow timeout 500 msec/cross-node"
These queries are fired from a java driver and high latency is for less than 1% of the traffic for the same query.
The same queries take few milliseconds for 99% of time and validated the same by running experiments for multiple days. No bottleneck is seen in CPU/Memory/IO.
How to fix these slow queries?