question

@ratna53_110276 It looks like you were making the correct OpsCenter API call but perhaps just didn't know how to interpret the output.

Here's an example where I attempt to retrieve information about one of the nodes in my cluster:

$ curl -L http://10.1.2.3:8888/community/nodes/10.1.2.45
{
  "node_ip": "10.1.2.45",
  "last_seen": 0,
  "dse_config_encryption": false,
  "mode": "normal",
  ...
}

Note that I have formatted the output to make it easier to read. The 2 important properties in the output above are last_seen and mode. In the example where the node is up and operational:

• last_seen is 0 meaning it's been zero seconds since the node was detected as being down
• mode is normal

When DSE is shutdown on the node, we get a different output:

$ curl -L http://10.1.2.3:8888/community/nodes/10.1.2.45
{
  "node_ip": "10.1.2.45",
  "last_seen": 1588056113,
  "dse_config_encryption": false,
  "mode": null,
  ...
}

The last_seen property is a Unix timestamp indicating the number of seconds since epoch (January 1, 1970). In the case above, 1588056113 is equivalent to April 28 6:41am GMT. The mode is null because DSE is down and the agent can't get DSE's state.

For your purposes, what you need to check for is a non-zero value for the last_seen property to determine if a node is down. Cheers!

UPDATE - The reason that the status is not immediate is because several things have to happen when you are restarting DSE on the nodes:

• DSE has to go through the shutdown procedure (usually takes several seconds depending on the size of the node)
• a 15-second window when the agent reconnects to the DSE/Cassandra instance
• the time it takes for the agent waiting for the JMX connection to timeout (10 seconds)
• the time it takes for the DSE connection attempt to timeout (30 seconds)
• DSE startup sequence -- this can vary widely depending on what workloads are configured, node density, size of Search cores, etc
• the time it takes for the agent to report back to OpsCenter

All this time is cumulative but note that it does _NOT_ mean that all the steps I noted above comes into play -- it depends on what the agent is running at the time. But the 2 biggest contributors to the turnaround are the DSE shutdown and startup sequence. But know that no two nodes will have identical times.

It's important to note that scripting a shutdown/startup of DSE can be problematic. Because you cannot assume that a node has cleanly shutdown not that it has started up cleanly either. You need to monitor/watch the logs while you're working on a node to make sure nothing goes wrong. Cheers!