We recently upgraded Cassandra server from 3.11.3 to 3.11.13. After upgrade we observed timeout issues from debug.log and from application server logs.
DEBUG [Native-Transport-Requests-21] 2022-05-24 04:30:52,151 ReadCallback.java:133 - Timed out; received 1 of 2 responses (including data)
this was not the case when we were on Cassandra 3.11.3 version.
we ran same set of jobs on 3.11.3 and they were successful. No time outs observed in debug.log or application server logs.
One thing I observed in tpstats is that "completed" value of NTR requests is low in 3.11.13 when compared with 3.11.3.
3.11.3 - Native-Transport-Requests 0 0 21079745 0 0
3.11.13 - Native-Transport-Requests 0 0 12364724 0 0
on further reading, I see that from 3.11.5, there were some changes made to NTR requests from JIRA -
Prevent client requests from blocking on executor task queue (CASSANDRA-15013)
Why the number of requests from clients are blocked or not processed due to the above change from Cassandra 3.11.5. How do I tune or which parameter/s to let the Cassandra server accept the requests(from client side) and process it.
Thanks for your help.