question

scano_183208 avatar image
scano_183208 asked Erick Ramirez edited

Logs show "Service SPARK-MASTER state is: NOT_STARTED"

When I do a tail /var/log/cassandra/debug.log on a spark worker node it shows the following logs:

DEBUG [RMI TCP Connection(657)-127.0.0.1] 2020-03-24 12:16:28,816 ServiceRunner.java:197 - Service SPARK-MASTER state is: NOT_STARTED
DEBUG [RMI TCP Connection(657)-127.0.0.1] 2020-03-24 12:16:28,816 ServiceRunner.java:197 - Service SPARK-WORKER state is: RUNNING
DEBUG [RMI TCP Connection(657)-127.0.0.1] 2020-03-24 12:16:28,817 ServiceRunner.java:197 - Service ALWAYSON-SQL state is: RUNNING

I notice that it always shows the spark Master state as not_started even though my spark master node is running.

On my second node (spark master node) It shows that the SPARK MASTER/WOKER State is running but ALWAYSON-SQL NOT_STARTED, again, always on sql is configured to be on, on all 3 nodes.

Lastly, my 3rd node just shows a SPARK-WORKER STATE as running and SPARK-MASTER and ALWAYSON-SQL States as NOT_RUNNING.

Any insight to this would be appreciated.

dsespark
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Russell Spitzer avatar image
Russell Spitzer answered Russell Spitzer commented

While there is a service to make sure that Always on and the Spark Master are alway running, these services still only run on a single node. When the service is detected as down on all nodes, the service is restarted on a running load as long as consensus can be reached.


DSE includes routing mechanisms to take requests for DSE Spark Master from any node and sends them to the actual node running the service. This is why a Master URL like dse://10.10.10.10 will work as long as 10.10.10.10 is in the same datacenter as the actual master.

The Always On Sql Server is similar, the service always is only running on a single node. For automatic routing in this case you need to use the Simba Driver. https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/spark/simbaOdbcDriver.html

6 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

scano_183208 avatar image scano_183208 commented ·

@Russell Spitzer

Hi Russell,

Thank you for getting back to me. I guess what is the service called to make sure that Always on and Spark Master are running?

On the other hand, it sounds like these logs are not critically and are most common, correct?

Thanks again!

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ scano_183208 commented ·

@scano_183208 They are logged at DEBUG level because they're not critical, not even logged at INFO level. Cheers!

0 Likes 0 ·
scano_183208 avatar image scano_183208 Erick Ramirez ♦♦ commented ·

@Erick Ramirez

Hi Erick,

Looking for the service that russell mentioned which keeps track of the status of the services?

0 Likes 0 ·
Show more comments