DataStax Academy FAQ

DataStax Academy migrated to a new learning management system (LMS) in July 2020. We are also moving to a new Cassandra Certification process so there are changes to exam bookings, voucher system and issuing of certificates.

Check out the Academy FAQ pages for answers to your questions:

question

nitin30_178226 avatar image
nitin30_178226 asked ·

I am facing three errors in system.log which are causing Application time outs , I need to find the Root cause of the issue

INFO [ScheduledTasks:1] 2020-01-07 11:11:06,090 MessagingService.java:1238 - RANGE_SLICE messages were dropped in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 21356 ms

INFO [ScheduledTasks:1] 2020-01-07 11:11:11,112 MessagingService.java:1238 - RANGE_SLICE messages were dropped in last 5000 ms: 0 internal and 7 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 23492 ms

INFO [ScheduledTasks:1] 2020-01-07 11:11:16,130 MessagingService.java:1238 - RANGE_SLICE messages were dropped in last 5000 ms: 0 internal and 2 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 23919 ms


WARN [CompactionExecutor:956] 2020-01-07 11:12:33,667 BigTableWriter.java:211 - Writing large partition otk_db/oauth_access_token_pagination.oauth_access_token_pagination_client_key_idx:74a02010fc1c48f59b3b72ad9345fc00 (102.821MiB) to sstable /data/data/otk_db/oauth_access_token_pagination-00880490f3be11e886a07ffa29ea568c/.oauth_access_token_pagination_client_key_idx/mc-16840-big-Data.db

WARN [CompactionExecutor:957] 2020-01-07 11:12:41,350 BigTableWriter.java:211 - Writing large partition otk_db/oauth_access_token_pagination.oauth_access_token_pagination_status_idx:ENABLED (102.821MiB) to sstable /data/data/otk_db/oauth_access_token_pagination-00880490f3be11e886a07ffa29ea568c/.oauth_access_token_pagination_status_idx/mc-16856-big-Data.db

WARN [CompactionExecutor:968] 2020-01-07 11:47:01,007 BigTableWriter.java:211 - Writing large partition otk_db/oauth_access_token_view_resource_owner.oauth_access_token_view_resource_owner_status_idx:ENABLED (301.527MiB) to sstable /data/data/otk_db/oauth_access_token_view_resource_owner-fe166850f3bd11e886a07ffa29ea568c/.oauth_access_token_view_resource_owner_status_idx/mc-16776-big-Data.db


cross node latencys and mean cross-node dropped latencytime outwriting large partition
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

@nitin30_178226 the dropped messages mean that the node is overloaded and cannot keep up with range reads. You need to either throttle the traffic to the cluster or add more nodes to increase the capacity of the cluster.

The large partition warnings are being logged because you are writing partitions which are larger than the recommended 100MB. Depending on the use case and cluster access patterns, partitions which are a few hundred megabytes in size might be OK for your application but larger partitions will have a significant impact on the performance of your cluster. Cheers!

Share
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.