question

INSERT failures

In Cassandra, writes requests from an application usually fail with the following reasons:

• WriteTimeoutException - the write request timed out at the coordinator while it was waiting for N replicas to respond (N number is based on the write consistency level) within write_request_timeout_in_ms
• WriteFailureException - some or all the replicas contacted by the coordinator replied with an error (not a timeout error) to indicate that the write failed, e.g. commitlog disk is full, disk IO queue is exhausted, etc.
• DriverTimeoutException - the driver request timed out because it didn't get a response from the coordinator within the client-side timeout window (configured in your app)
• AllNodesFailedException - the query failed on all the coordinators tried by the driver (for whatever reason)

All these exceptions are logged client-side meaning the driver reports these exceptions in your application's log files. You then need to correlate the errors on the logs of corresponding node to determine what caused the error but the query itself.

For example, if the driver reported in the application log a write failure on node 10.1.2.3 at 13:24, you need to review the log entries on that node around 13:24 to determine why the writes failed. It will usually be a GC pause and/or a dropped mutation message (usually indicates that the commitlog disk is overloaded and cannot keep up with the IO load).

Other items to consider

We don't recommend increasing the batch size thresholds this high:

batch_size_warn_threshold_in_kb: 100
batch_size_fail_threshold_in_kb :1000

Batches are not an optimisation -- they don't make your queries run faster. It can be counter-intuitive because large batch sizes can overload your cluster and make the queries run slower.

We also don't recommend increasing the write timeouts:

write_request_timeout_in_ms: 30000
counter_write_request_timeout_in_ms: 30000

The timeouts are in place to prevent long-running writes from overloading the nodes. If the writes are timing out because the disks cannot keep up with the IO, increasing the timeout threshold only overloads the nodes further since the requests stay queued up for a long time instead of getting rejected by Cassandra.

It looks like you got this statement wrong too:

Cassandra Instance type :- EC2 Instance M5.Xlarge( 8CPU 64 GB RAM)

m5.xlarge instances only come with 4 vCPUs + 16GB of RAM. There isn't an m5 instance that comes with 8 vCPUs + 64GB of RAM. Is it possible your cluster is running with r5.2xlarge EC2 instances?

As a side note, you previously indicated in question #5145 that your cluster is running with Apache Cassandra 3.11.6 but the nodetool getlogginglevels output you posted indicates your cluster is running with a version of DSE:

[cassandra@uatrrcass05 commitlog]$ nodetool getlogginglevels
Logger Name                                        Log Level
ROOT                                                    INFO
com.thinkaurelius.thrift                               ERROR
org.apache.cassandra                                   DEBUG

I only wanted to point this out because the version of Cassandra or DSE your cluster is running is relevant to answering your questions. The recommendation we give depends on the specific version you're running. Cheers!