Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

scano_183208 avatar image
scano_183208 asked ·

Spark Worker/Master State in cassandra logs

When I do a tail /var/log/cassandra/debug.log on a spark worker node it shows the following logs:

DEBUG [RMI TCP Connection(657)-127.0.0.1] 2020-03-24 12:16:28,816 ServiceRunner.java:197 - Service SPARK-MASTER state is: NOT_STARTED

DEBUG [RMI TCP Connection(657)-127.0.0.1] 2020-03-24 12:16:28,816 ServiceRunner.java:197 - Service SPARK-WORKER state is: RUNNING

DEBUG [RMI TCP Connection(657)-127.0.0.1] 2020-03-24 12:16:28,817 ServiceRunner.java:197 - Service ALWAYSON-SQL state is: RUNNING


I notice that it always shows the spark Master state as not_started even though my spark master node is running.

On my second node (spark master node) It shows that the SPARK MASTER/WOKER State is running but ALWAYSON-SQL NOT_STARTED, again, always on sql is configured to be on, on all 3 nodes.

Lastly, my 3rd node just shows a SPARK-WORKER STATE as running and SPARK-MASTER and ALWAYSON-SQL States as NOT_RUNNING.

Any insight to this would be appreciated.

dsesparklogs
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Russell Spitzer avatar image
Russell Spitzer answered ·

While there is a service to make sure that Always on and the Spark Master are alway running, these services still only run on a single node. When the service is detected as down on all nodes, the service is restarted on a running load as long as consensus can be reached.


DSE includes routing mechanisms to take requests for DSE Spark Master from any node and sends them to the actual node running the service. This is why a Master URL like dse://10.10.10.10 will work as long as 10.10.10.10 is in the same datacenter as the actual master.

The Always On Sql Server is similar, the service always is only running on a single node. For automatic routing in this case you need to use the Simba Driver. https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/spark/simbaOdbcDriver.html

6 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

@Russell Spitzer

Hi Russell,

Thank you for getting back to me. I guess what is the service called to make sure that Always on and Spark Master are running?

On the other hand, it sounds like these logs are not critically and are most common, correct?

Thanks again!

0 Likes 0 · ·

@scano_183208 They are logged at DEBUG level because they're not critical, not even logged at INFO level. Cheers!

0 Likes 0 · ·
scano_183208 avatar image scano_183208 Erick Ramirez ♦♦ ·

@Erick Ramirez

Hi Erick,

Looking for the service that russell mentioned which keeps track of the status of the services?

0 Likes 0 · ·
Show more comments