Attempting to reengineer several client mode dse spark jobs as cluster mode. What method is most foolproof to tell when the cluster mode job has completed from a shell script running locally on the data node?
- dse spark-submit --status driver-id ... loop
- rest api call such as ... curl -L http://nodename:18080/api/v1/applications?status=running
Any other recommendations? Thank you!