Cassandra node won't startup after reboot

Question

question

whpw asked · May 31 at 02:52 PM

Cassandra node won't startup after reboot

Hi, we have a Cassandra cluster running with 5 nodes. Cassandra version 3.10 / CentOS 7

After I have restarted the Cassandra service without changing any configuration, the node does not come up anymore. The problem is that the port 7000 (gossip) and 9042 ist not up (not available) -> cannot connect with other nodes in the DC.

I have no idea why.

Here some logs

INFO  [main] 2021-05-31 14:18:50,344 ColumnFamilyStore.java:406 - Initializing system.schema_aggregates
INFO  [main] 2021-05-31 14:18:50,344 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO  [main] 2021-05-31 14:18:50,458 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
INFO  [main] 2021-05-31 14:18:50,839 StorageService.java:599 - Populating token metadata from system tables
INFO  [main] 2021-05-31 14:18:51,730 ColumnFamilyStore.java:406 - Initializing system_distributed.view_build_status
INFO  [main] 2021-05-31 14:18:51,742 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized

then I get a lot of following logs.

WARN  [ScheduledTasks:1] 2021-05-31 14:42:48,790 QueryProcessor.java:105 - 62506 prepared statements discarded in the last minute because cache limit reached (11 MB)
WARN  [ScheduledTasks:1] 2021-05-31 14:43:48,437 QueryProcessor.java:105 - 35963 prepared statements discarded in the last minute because cache limit reached (11 MB)

Many thanks in advance

cassandra

2

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

whpw · May 31 at 03:04 PM

and in the debug.log i find a lot of following logs:

'DEBUG [main] 2021-05-31 17:01:56,128 SystemKeyspace.java:1474 - stored prepared statement for logged keyspace 'bds': 'SELECT * FROM tableA WHERE ad_id=xxxxxxx AND user_id=yyyyyyyy AND event_type_id IN (1,21)'
'

0 ·

whpw · May 31 at 03:15 PM

Hi,

OK, the node is back.

It took a long time(whatever it was) until the node was back.

WARN  [ScheduledTasks:1] 2021-05-31 17:04:52,887 QueryProcessor.java:105 - 50773 prepared statements discarded in the last minute because cache limit reached (11 MB)
INFO  [main] 2021-05-31 17:05:15,469 QueryProcessor.java:162 - Preloaded 3607012 prepared statements

Maybe someone can explain me the logs above. Thx

0 ·

Answer 1 · 2021-06-02T12:14:02Z

whpw answered · Jun 02 at 12:14 PM

I don't know if anyone cares. We have disabled the prepared_statements on the application side and truncated table system.prepared_statements. After a restart of the Cassandra nodes, the node connected to the existing cluster immediately. All is well again.

Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 2 · 2021-06-01T04:09:48Z

Erick Ramirez answered · Jun 01 at 04:09 AM

The log entries you posted do not indicate issues with the node starting up. In fact, they show that the Cassandra process is in fact operating.

If you post error messages (not info or warning messages) that indicate a problem with the node starting, we'd be happy to review them. Cheers!

3 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

whpw · Jun 01 at 08:55 AM

Hi,

ok. I have narrowed down the strange behaviour (for me) of cassandra.

I found out, that the cassandra java driver for some applications have been updated to the version 4.11.1. Since that we get a lot WARNs

in the system.log.

 WARN  [ScheduledTasks:1] 2021-06-01 10:20:47,662 QueryProcessor.java:105 - 18120 prepared statements discarded in the last minute because cache limit reached (78 M

and in the debug.log

DEBUG [Native-Transport-Requests-1] 2021-06-01 10:48:13,977 SystemKeyspace.java:1474 - stored prepared statement for logged keyspace 'bds': 'SELECT * FROM table_x WHERE ad_id IN (fc15410f-4e4a-4f00-8c02-f28c8cc150da) AND type IN (0,21)'

it looks like, that a logging has been activated. Aftet a restart of the cassandra service it takes really long until the node is recovered and joined to the cluster again. Do you have any idea and can explain this behaviour. It makes a little bit nervous.

Many thanks in advance....

cheers

0 ·

Erick Ramirez ♦♦ whpw · Jun 01 at 10:35 AM

These don't have anything to do with startup. Again, the relevant messages are errors and not warning/debug messages. Cheers!

0 ·

whpw Erick Ramirez ♦♦ · Jun 01 at 11:02 AM

ok. we have no errors at the startup. I just wanted to know what the WARNs mean and why I see stored prepared statement for logged keyspace in the debug.log. We did not have this before using the driver.

0 ·

question