Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started



sridhar.addanki_188870 avatar image
sridhar.addanki_188870 asked Erick Ramirez edited

Why does the AlwaysOn SQL Service (AOSS) keep restarting?

We had issue with one of our cassandra node due to hardware issue. We have removed that from ring and restarted DSE in other nodes. From then Alwayson-sql service keeps restarting. Found below error from service.log under /var/log/spark/always_on

Because of this we are unable use spark-sql from datastax sudio for JOIN queries. Please help.

Earlier this used to work.


ERROR 2020-05-01 20:31:17,230 org.apache.spark.deploy.DseSparkSubmitBootstrapper: Spark application failed
org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: com.datastax.bdp.fs.model.InternalServerException: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (1 required but only 0 alive);
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:53)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:96)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

@sridhar.addanki_188870 Based on the stack trace you provided, it looks like the ThriftServer which serves up Hive to AOSS is having issues during initialisation.

Check for the existence of /tmp/hive/cassandra on the node. Is it owned by root or cassandra?

You can either (a) grant global (all users) write access to the directory or (b) delete the directory itself. Then check the AOSS service.log again to see if the service is attempting to restart. If it isn't, you can do a clean stop then restart by running these commands on the node:

$ dse client-tool alwayson-sql stop
$ dse client-tool alwayson-sql start

Let me know how you go. Cheers!

10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.