question

nitin30_178226 avatar image
nitin30_178226 asked Erick Ramirez answered

I am facing three errors in system.log which are causing Application time outs , I need to find the Root cause of the issue

INFO [ScheduledTasks:1] 2020-01-07 11:11:06,090 MessagingService.java:1238 - RANGE_SLICE messages were dropped in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 21356 ms

INFO [ScheduledTasks:1] 2020-01-07 11:11:11,112 MessagingService.java:1238 - RANGE_SLICE messages were dropped in last 5000 ms: 0 internal and 7 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 23492 ms

INFO [ScheduledTasks:1] 2020-01-07 11:11:16,130 MessagingService.java:1238 - RANGE_SLICE messages were dropped in last 5000 ms: 0 internal and 2 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 23919 ms


WARN [CompactionExecutor:956] 2020-01-07 11:12:33,667 BigTableWriter.java:211 - Writing large partition otk_db/oauth_access_token_pagination.oauth_access_token_pagination_client_key_idx:74a02010fc1c48f59b3b72ad9345fc00 (102.821MiB) to sstable /data/data/otk_db/oauth_access_token_pagination-00880490f3be11e886a07ffa29ea568c/.oauth_access_token_pagination_client_key_idx/mc-16840-big-Data.db

WARN [CompactionExecutor:957] 2020-01-07 11:12:41,350 BigTableWriter.java:211 - Writing large partition otk_db/oauth_access_token_pagination.oauth_access_token_pagination_status_idx:ENABLED (102.821MiB) to sstable /data/data/otk_db/oauth_access_token_pagination-00880490f3be11e886a07ffa29ea568c/.oauth_access_token_pagination_status_idx/mc-16856-big-Data.db

WARN [CompactionExecutor:968] 2020-01-07 11:47:01,007 BigTableWriter.java:211 - Writing large partition otk_db/oauth_access_token_view_resource_owner.oauth_access_token_view_resource_owner_status_idx:ENABLED (301.527MiB) to sstable /data/data/otk_db/oauth_access_token_view_resource_owner-fe166850f3bd11e886a07ffa29ea568c/.oauth_access_token_view_resource_owner_status_idx/mc-16776-big-Data.db


cross node latencys and mean cross-node dropped latencytime outwriting large partition
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

@nitin30_178226 the dropped messages mean that the node is overloaded and cannot keep up with range reads. You need to either throttle the traffic to the cluster or add more nodes to increase the capacity of the cluster.

The large partition warnings are being logged because you are writing partitions which are larger than the recommended 100MB. Depending on the use case and cluster access patterns, partitions which are a few hundred megabytes in size might be OK for your application but larger partitions will have a significant impact on the performance of your cluster. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.