Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

whpw avatar image
whpw asked ·

Cassandra node won't startup after reboot

Hi, we have a Cassandra cluster running with 5 nodes. Cassandra version 3.10 / CentOS 7

After I have restarted the Cassandra service without changing any configuration, the node does not come up anymore. The problem is that the port 7000 (gossip) and 9042 ist not up (not available) -> cannot connect with other nodes in the DC.

I have no idea why.

Here some logs

INFO  [main] 2021-05-31 14:18:50,344 ColumnFamilyStore.java:406 - Initializing system.schema_aggregates
INFO  [main] 2021-05-31 14:18:50,344 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO  [main] 2021-05-31 14:18:50,458 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
INFO  [main] 2021-05-31 14:18:50,839 StorageService.java:599 - Populating token metadata from system tables
INFO  [main] 2021-05-31 14:18:51,730 ColumnFamilyStore.java:406 - Initializing system_distributed.view_build_status
INFO  [main] 2021-05-31 14:18:51,742 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized

then I get a lot of following logs.

WARN  [ScheduledTasks:1] 2021-05-31 14:42:48,790 QueryProcessor.java:105 - 62506 prepared statements discarded in the last minute because cache limit reached (11 MB)
WARN  [ScheduledTasks:1] 2021-05-31 14:43:48,437 QueryProcessor.java:105 - 35963 prepared statements discarded in the last minute because cache limit reached (11 MB)

Many thanks in advance

cassandra
2 comments
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

whpw avatar image whpw ·

and in the debug.log i find a lot of following logs:

'DEBUG [main] 2021-05-31 17:01:56,128 SystemKeyspace.java:1474 - stored prepared statement for logged keyspace 'bds': 'SELECT * FROM tableA WHERE ad_id=xxxxxxx AND user_id=yyyyyyyy AND event_type_id IN (1,21)'
'


0 Likes 0 ·
whpw avatar image whpw ·

Hi,

OK, the node is back.

It took a long time(whatever it was) until the node was back.

WARN  [ScheduledTasks:1] 2021-05-31 17:04:52,887 QueryProcessor.java:105 - 50773 prepared statements discarded in the last minute because cache limit reached (11 MB)
INFO  [main] 2021-05-31 17:05:15,469 QueryProcessor.java:162 - Preloaded 3607012 prepared statements 

Maybe someone can explain me the logs above. Thx

0 Likes 0 ·

2 Answers

Erick Ramirez avatar image
Erick Ramirez answered ·

The log entries you posted do not indicate issues with the node starting up. In fact, they show that the Cassandra process is in fact operating.

If you post error messages (not info or warning messages) that indicate a problem with the node starting, we'd be happy to review them. Cheers!

3 comments Share
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

whpw avatar image whpw ·

Hi,

ok. I have narrowed down the strange behaviour (for me) of cassandra.

I found out, that the cassandra java driver for some applications have been updated to the version 4.11.1. Since that we get a lot WARNs

in the system.log. 

 WARN  [ScheduledTasks:1] 2021-06-01 10:20:47,662 QueryProcessor.java:105 - 18120 prepared statements discarded in the last minute because cache limit reached (78 M

and in the debug.log

DEBUG [Native-Transport-Requests-1] 2021-06-01 10:48:13,977 SystemKeyspace.java:1474 - stored prepared statement for logged keyspace 'bds': 'SELECT * FROM table_x WHERE ad_id IN (fc15410f-4e4a-4f00-8c02-f28c8cc150da) AND type IN (0,21)'

it looks like, that a logging has been activated. Aftet a restart of the cassandra service it takes really long until the node is recovered and joined to the cluster again. Do you have any idea and can explain this behaviour. It makes a little bit nervous.

Many thanks in advance....

cheers

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ whpw ·

These don't have anything to do with startup. Again, the relevant messages are errors and not warning/debug messages. Cheers!

0 Likes 0 ·
whpw avatar image whpw Erick Ramirez ♦♦ ·

ok. we have no errors at the startup. I just wanted to know what the WARNs mean and why I see stored prepared statement for logged keyspace in the debug.log. We did not have this before using the driver.

0 Likes 0 ·
whpw avatar image
whpw answered ·

I don't know if anyone cares. We have disabled the prepared_statements on the application side and truncated table system.prepared_statements. After a restart of the Cassandra nodes, the node connected to the existing cluster immediately. All is well again.


Share
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.