Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

gappa avatar image
gappa asked ·

How can I determine why my keyspace is missing?

hello,

This morning, Monday!!!, I was checking on Cassandra cluster and I noticed that all data was missing.

I noticed the following error on each node (9 nodes in the cluster):

2021-03-01 09:05:52,984 WARN  [MessagingService-Incoming-/x.x.x.x] IncomingTcpConnection.java:103 run UnknownColumnFamilyException reading from socket; closingorg.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId cba90a70-5c46-11e9-9e36-f54fe3235e69. If a table was just created, this is likely due to the schema not being fully propagated. Please wait for schema agreement on table creation. at org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1533) at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:758) at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:697) at org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50) at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123) at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:183) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)

I tried to query the keyspace and got this:

node1# cqlshConnected to Cassandra Cluster at x.x.x.x:9042.[cqlsh 5.0.1 | Cassandra 3.11.5.1 | CQL spec 3.4.4 | Native protocol v4]Use HELP for help.cqlsh> select * from mykeyspace.mytable where id = 123935;
InvalidRequest: Error from server: code=2200 [Invalid query] message="Keyspace mykeyspace  does not exist"

Investigating on each node I found that all the SStables exist, so I think data is still there but the keyspace vanished, "magically".

Other facts I can tell you are:

  • I have been getting Anticompaction errors from 2 nodes due to the fact the disk was almost full.
  • the cluster was online Friday
  • this morning, Monday, the whole cluster was offline and I noticed the problem of "missing keyspace"
  • During the weekend the cluster has been subject to inserts and deletes
  • I have a 9 node (HDD) Cassandra 3.11 cluster.

I really need help on this, how can I restore the cluster?

Thank you very much

schema
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

As previously discussed on the Apache Cassandra users' mailing list, the keyspace is missing because an operator manually deleted it with the DROP KEYSPACE command:

2021-03-01 06:08:08,864 INFO  [Native-Transport-Requests-1] MigrationManager.java:542 announceKeyspaceDrop Drop Keyspace <keyspace_name>

Cheers!

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.