Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

ranjeet_ranjee avatar image
ranjeet_ranjee asked ·

Getting error "RequestHandler : Host /host_ip:9042 is overloaded."

Hi,

I am getting an error when one of our microservice wanted export data from Cassandra which consist of apx 1K row data.

com.datastax.driver.core.RequestHandler  : Host /Host_ip:9042 is overloaded.
2021-02-02 22:49:34.214  WARN 25005 --- [r1-nio-worker-3] com.datastax.driver.core.RequestHandler  : Host /Host_ip:9042 is overloaded.
2021-02-02 22:49:34.215 ERROR 25005 --- [r1-nio-worker-3] c.g.c.util.concurrent.AggregateFuture    : Got more than one input Future failure. Logging failures after the first

com.datastax.driver.core.exceptions.OverloadedException: Queried host (/Host_ip:9042) was overloaded: Request dropped due to backpressure overload: timeout exceeded.
    at com.datastax.driver.core.Responses$Error.asException(Responses.java:128) ~[cassandra-driver-core-3.2.0.jar!/:na]
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:507) ~[cassandra-driver-core-3.2.0.jar!/:na]
    at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1075) [cassandra-driver-core-3.2.0.jar!/:na]
    at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:998) [cassandra-driver-core-3.2.0.jar!/:na]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [netty-codec-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [netty-codec-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.8.Final.jar!/:4.1.8.Final]
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) [netty-common-4.1.8.Final.jar!/:4.1.8.Final]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]


Table Schema Details


poolingOptions
                .setMaxConnectionsPerHost(HostDistance.REMOTE, 5)
                .setMaxConnectionsPerHost(HostDistance.LOCAL, 5)

                .setMaxRequestsPerConnection(HostDistance.LOCAL, 32768)
                .setMaxRequestsPerConnection(HostDistance.REMOTE, 32768)
                .setMaxQueueSize(1000000)
                .setPoolTimeoutMillis(1000000) 


CREATE TABLE test.conversion_index (
    ids int,
    hour timestamp,
    timestamp timestamp,
    uid uuid,
    PRIMARY KEY ((ids, hour), timestamp, uid)
) WITH CLUSTERING ORDER BY (timestamp DESC, uid ASC)
    AND additional_write_policy = '99PERCENTILE'
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND nodesync = {'enabled': 'true', 'incremental': 'true'}
    AND read_repair = 'BLOCKING'
    AND speculative_retry = '99PERCENTILE';


dse
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

It sounds like you are running a large query that is overloading the nodes and eventually times out.

A common cause of this is when you're running a query that's performing a full table scan. As an example, an unbounded (no filter) read request like:

SELECT * FROM table_name;

This will overload the nodes as it needs to retrieve all the partition from all the nodes in the cluster. Cheers!

6 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Hi Erick,

We are not making read request like

  
                    
  1. SELECT * FROM table_name;

We are making a query to export the data with a filter Campaign ID and publisher id and a date range.

Results also shows something like 800 rows to export but getting error as shown in the question.

0 Likes 0 ·

@ranjeet_ranjee, please update your original question with the complete table schema and the query that you're performing along with any other helpful informationfor the triage. Thanks. I see that you are using cassandra-driver-core:3.2.0 and the latest in that series is https://mvnrepository.com/artifact/com.datastax.cassandra/cassandra-driver-core/3.10.2. It wouldn't hurt to try with that as well as an additional test.

0 Likes 0 ·

@smadhavan So adding additional details about my issue, I am doing Asynchronous Programming in Vertx and using Future object to do File write operation.

after getting data from Cassandra with the below table schema, we are simply querying "SELECT uid FROM test.conversion_index WHERE idx = ? AND hour = ? "


Also, I have updated the question. Please check.

0 Likes 0 ·
Show more comments