I am facing three errors in system.log which are causing Application time outs , I need to find the Root cause of the issue

Question

question

nitin30_178226 asked · Jan 07 at 08:25 AM

I am facing three errors in system.log which are causing Application time outs , I need to find the Root cause of the issue

INFO [ScheduledTasks:1] 2020-01-07 11:11:06,090 MessagingService.java:1238 - RANGE_SLICE messages were dropped in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 21356 ms

INFO [ScheduledTasks:1] 2020-01-07 11:11:11,112 MessagingService.java:1238 - RANGE_SLICE messages were dropped in last 5000 ms: 0 internal and 7 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 23492 ms

INFO [ScheduledTasks:1] 2020-01-07 11:11:16,130 MessagingService.java:1238 - RANGE_SLICE messages were dropped in last 5000 ms: 0 internal and 2 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 23919 ms

WARN [CompactionExecutor:956] 2020-01-07 11:12:33,667 BigTableWriter.java:211 - Writing large partition otk_db/oauth_access_token_pagination.oauth_access_token_pagination_client_key_idx:74a02010fc1c48f59b3b72ad9345fc00 (102.821MiB) to sstable /data/data/otk_db/oauth_access_token_pagination-00880490f3be11e886a07ffa29ea568c/.oauth_access_token_pagination_client_key_idx/mc-16840-big-Data.db

WARN [CompactionExecutor:957] 2020-01-07 11:12:41,350 BigTableWriter.java:211 - Writing large partition otk_db/oauth_access_token_pagination.oauth_access_token_pagination_status_idx:ENABLED (102.821MiB) to sstable /data/data/otk_db/oauth_access_token_pagination-00880490f3be11e886a07ffa29ea568c/.oauth_access_token_pagination_status_idx/mc-16856-big-Data.db

WARN [CompactionExecutor:968] 2020-01-07 11:47:01,007 BigTableWriter.java:211 - Writing large partition otk_db/oauth_access_token_view_resource_owner.oauth_access_token_view_resource_owner_status_idx:ENABLED (301.527MiB) to sstable /data/data/otk_db/oauth_access_token_view_resource_owner-fe166850f3bd11e886a07ffa29ea568c/.oauth_access_token_view_resource_owner_status_idx/mc-16776-big-Data.db

cross node latency s and mean cross-node dropped latency time out writing large partition

10 |400 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-01-07T08:29:06Z

Erick Ramirez answered · Jan 07 at 08:29 AM

@nitin30_178226 the dropped messages mean that the node is overloaded and cannot keep up with range reads. You need to either throttle the traffic to the cluster or add more nodes to increase the capacity of the cluster.

The large partition warnings are being logged because you are writing partitions which are larger than the recommended 100MB. Depending on the use case and cluster access patterns, partitions which are a few hundred megabytes in size might be OK for your application but larger partitions will have a significant impact on the performance of your cluster. Cheers!

· Share

10 |400 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

I am facing three errors in system.log which are causing Application time outs , I need to find the Root cause of the issue

question details

OTHER RESOURCES