question

gappa avatar image
gappa asked Erick Ramirez answered

How can I determine why my keyspace is missing?

hello,

This morning, Monday!!!, I was checking on Cassandra cluster and I noticed that all data was missing.

I noticed the following error on each node (9 nodes in the cluster):

2021-03-01 09:05:52,984 WARN  [MessagingService-Incoming-/x.x.x.x] IncomingTcpConnection.java:103 run UnknownColumnFamilyException reading from socket; closingorg.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId cba90a70-5c46-11e9-9e36-f54fe3235e69. If a table was just created, this is likely due to the schema not being fully propagated. Please wait for schema agreement on table creation. at org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1533) at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:758) at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:697) at org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50) at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123) at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:183) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)

I tried to query the keyspace and got this:

node1# cqlshConnected to Cassandra Cluster at x.x.x.x:9042.[cqlsh 5.0.1 | Cassandra 3.11.5.1 | CQL spec 3.4.4 | Native protocol v4]Use HELP for help.cqlsh> select * from mykeyspace.mytable where id = 123935;
InvalidRequest: Error from server: code=2200 [Invalid query] message="Keyspace mykeyspace  does not exist"

Investigating on each node I found that all the SStables exist, so I think data is still there but the keyspace vanished, "magically".

Other facts I can tell you are:

  • I have been getting Anticompaction errors from 2 nodes due to the fact the disk was almost full.
  • the cluster was online Friday
  • this morning, Monday, the whole cluster was offline and I noticed the problem of "missing keyspace"
  • During the weekend the cluster has been subject to inserts and deletes
  • I have a 9 node (HDD) Cassandra 3.11 cluster.

I really need help on this, how can I restore the cluster?

Thank you very much

schema
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

As previously discussed on the Apache Cassandra users' mailing list, the keyspace is missing because an operator manually deleted it with the DROP KEYSPACE command:

2021-03-01 06:08:08,864 INFO  [Native-Transport-Requests-1] MigrationManager.java:542 announceKeyspaceDrop Drop Keyspace <keyspace_name>

Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.