question

An AsyncReadTimeoutException isn't due to a corrupt SSTable.

Open-source Apache Cassandra uses the traditional staged event-driven architecture (SEDA) where tasks are executed by threads synchronously. This has worked well for quite some time now but we know that the synchronous nature of the architecture can lead to extremely high context-switching during peak loads which can cause resource contention and degrades the performance of the nodes.

In DSE 6.0, we released a new feature called Advanced Performance. Under the hood is a thread-per-core architecture which significantly improves read and write performance with asynchronous IO -- read and write requests are no longer synchronous to remove thread contention and keeps IO threads processing.

An AsyncReadTimeoutException is thrown when an asynchronous disk read (literally an IO request to the disk) takes too long. Although this can take due to faulty hardware (e.g. unresponsive disk subsystem), it is almost always a result of the data disk being overloaded with read requests.

If you are regularly seeing this exception, I recommend you check the disk IO performance with Linux utilities like iostat. Ideally, your system admin team should be monitoring the servers so you have historical metrics on how the disks are performing over time. It will give you an idea on whether IO requests are maxing out. If this is happening regularly, the solution is to increase the capacity of your cluster by adding more nodes.

For more info, see the blog post DSE Advanced Performance: Apache Cassandra™ Goes Turbo. Cheers!