question

Beck avatar image
Beck asked Erick Ramirez edited

Gossip stage has pending tasks.How to refresh the Gossip?

Hi all,

(DSE version: 6.7.3)

Sometimes I got the message in the "debug.log" when I start a node:

WARN  [GossipTasks:1] 2019-07-02 09:10:31,276  Gossiper.java:1007 - Gossip stage has 633 pending tasks; skipping status check (no nodes will be marked down)
WARN  [GossipTasks:1] 2019-07-02 09:10:32,377  Gossiper.java:1007 - Gossip stage has 635 pending tasks; skipping status check (no nodes will be marked down)
WARN  [GossipTasks:1] 2019-07-02 09:10:33,477  Gossiper.java:1007 - Gossip stage has 636 pending tasks; skipping status check (no nodes will be marked down)
WARN  [GossipTasks:1] 2019-07-02 09:10:34,578  Gossiper.java:1007 - Gossip stage has 639 pending tasks; skipping status check (no nodes will be marked down)
WARN  [GossipTasks:1] 2019-07-02 09:10:35,678  Gossiper.java:1007 - Gossip stage has 641 pending tasks; skipping status check (no nodes will be marked down)
WARN  [GossipTasks:1] 2019-07-02 09:10:36,778  Gossiper.java:1007 - Gossip stage has 644 pending tasks; skipping status check (no nodes will be marked down)

I found this article, and I checked the permission of my commit logs.

What can I debug this issue?

How to refresh the Gossip?

Thanks!

---

update:

I seen the DataStax Enterprise 6.7 release notes:

  
                 
    
                  
  1. Fixed an issue by incrementing pending echos when sending gossip echo requests. (DB-3187)
    2.

Is that probably about this issue?

gossip
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Beck edited

@Beck the symptoms you described here and here point to the nodes being overloaded so you'll run into lots of issues. Specifically for gossip, it's also possible that the commitlog directory has not run out of disk space. Cheers!

14 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Beck avatar image Beck ·

Hi @Erick Ramirez ,


That issue has sometimes happened when a node starts.

Maybe It's not the same as this.


I also checked disk usage, everything was fine.


Thanks for your reply.

1 Like 1 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Beck ·

Can you also confirm if you're seeing dropped mutations on the node? If the commitlog disk cannot keep up with the writes, that is another symptom of an overloaded node. In particular if it has issues flushing memtables then that would also cause gossip tasks to queue for the same reason that I explained in the KB article you referenced. Cheers!

1 Like 1 ·
Beck avatar image Beck Erick Ramirez ♦♦ ·

> Can you also confirm if you're seeing dropped mutations on the node? If the commitlog disk cannot keep up with the writes


Ok,but I already upgrade to DSE 6.7.4.

If It happened again,I'll concern the dropped messages.


Thanks for your reply!

1 Like 1 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Beck ·

I just wanted to confirm that it's not because the node is overloaded. How much CPUs/RAM does the server have and how much memory is allocated to the heap? Cheers!

0 Likes 0 ·
Beck avatar image Beck Erick Ramirez ♦♦ ·

> How much CPUs/RAM does the server have and how much memory is allocated to the heap?


It's CPU: 24 core RAM: 128 GB per node and my cluster use "default setting".


Thanks!

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Beck ·

But how much memory is allocated to the heap?

0 Likes 0 ·
Beck avatar image Beck Erick Ramirez ♦♦ ·

Sorry for I am late.


That is my cluster info: 

-XX:MaxDirectMemorySize=48289M -Xms32193M -Xmx32193M
1 Like 1 ·
Show more comments
Erick Ramirez avatar image Erick Ramirez ♦♦ ·

@Beck just a friendly note that you posted a "comment" as an "answer". Just letting you know since it can be confusing for others coming across this post in the future. Cheers!

0 Likes 0 ·
Beck avatar image Beck Erick Ramirez ♦♦ ·

Sorry about that.


I can't push a comment under that answer yesterday.


Nothing have happened when I push it.

1 Like 1 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Beck ·

Cool, cheers! :)

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ ·

As it states in CASSANDRA-9279, the issue is that the commitlog disk cannot keep up with the writes. When the write requests are queued up, it stops gossip from working. Gossip is just the symptom of the problem -- it's not the cause of the issue.
You'll need to review the disk performance for this timeperiod -- 2019-07-02 09:10:31. Cheers!

0 Likes 0 ·
Beck avatar image Beck Erick Ramirez ♦♦ ·

I have checked again but it seem normal.

My issue was only happened when I start node.


But most of all metrics explain that the node was overloaded as this question(What could be the cause of stuck with "Remote TPC backpressure" increasing?).

I will keep concern it if this issue happened.


Thanks for your reply again!

1 Like 1 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Beck ·

Awesome, cheers!

0 Likes 0 ·