Unable to add existing cluster to opscenter

Question

question

zico_178040 asked Oct 15, '19 Erick Ramirez commented Oct 18, '19

Unable to add existing cluster to opscenter

When I attempt to add a cluster (DSE 6.0.5, single node) to opscenter (6.7.4) (with client-to-node encryption enabled) via the web url (opscenter-ip:8888). I get the below error:

OpsCenter was not able to add the cluster: Unable to connect to cluster. Error is: Unable to connect to any seed nodes, tried [u'XX.XXX.XXX.XX']

(XX.XXX.XXX.XX = ip where dse is installed)

Looking at the opscenterd.log file, I see the following:

2019-10-10 20:28:22,209 [opscenterd] WARN: ProcessingError while calling CreateClusterConfController: Unable to connect to cluster. Error is: Unable to connect to any seed nodes, tried [u'XX.XXX.XXX.XX'] (MainThread)
2019-10-10 20:32:28,475 [opscenterd] ERROR: Error connecting to the cluster: Traceback (most recent call last):
 File "/opscenter/opscenter-6.7.4/lib/py/twisted/internet/defer.py", line 1122, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g)
 File "/opscenter/opscenter-6.7.4/lib/py/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb)
 File "/usr/share/opscenter/Lib/site-packages/opscenterd/WrappedDriver.py", line 113, in _defer_to_pool NoHostAvailable: All host(s) tried for query failed (tried: /XX.XXX.XXX.XX:9042 (com.datastax.driver.core.exceptions.TransportException: [/XX.XXX.XXX.XX:9042] Error writing)) (MainThread)

(Wondering what "Error writing" means)

I also followed the guide to install the datastax-agent manually (via tarball) and when I check the "agent.log" file, I see the following:

ERROR [async-dispatch-8] 2019-10-10 19:47:02,213 Can't connect to Cassandra (All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.TransportException: [/127.0.0.1:9042] Cannot connect))), retrying soon.

Not entirely sure why its showing 127.0.0.1, when I check "nodetool status" I see the correct IP address and I can connect to cassandra via cqlsh while specifying the correct IP address and username/password. In the cassandra.yaml file; listen_address and rpc_address are pointing the the IP address where dse is installed.

Also I am able to telnet from my opscenter machine to the DSE cluster (vice versa as well) both on port 8888 and 9042. I cannot tell if this is a networking or authentication issue or something else.

I would appreciate any inputs!

EDIT - Follow up questions:

Hello Erick, Thanks a lot for your response! Below is an update based on your suggestions

Server A (where opscenter is installed: 172.13.11.91):

Server B (where dse cluster is installed: 172.81.18.321):

Below are the results of the telnet tests I performed from Server A to Server B on both port 9042 and 8888, which were both successful

[capstone@ip-172-13-11-91 conf]$ telnet 172.81.18.321 9042
Trying 172.81.18.321...
Connected to 172.81.18.321.
Escape character is '^]'.

[capstone@ip-ip-172-13-11-91 conf]$ telnet 172.81.18.321 8888
Trying 172.81.18.321...
Connected to 172.81.18.321.
Escape character is '^]'.

As suggested, I updated the address.yaml (on Server B) file with the "hosts" options. I see the following in the opscenterd.log file, when I attempt to connect (with client-node encryption enabled):

2019-10-16 14:47:08,188 [opscenterd] ERROR: Error connecting to the cluster: Traceback (most recent call last):
File "/opt/apps/opscenter/opscenter-6.7.4/lib/py/twisted/internet/defer.py", line 1122, in _inlineCallbacks
...
NoHostAvailable: All host(s) tried for query failed (tried: /172.81.18.321:9042 (com.datastax.driver.core.exceptions.TransportException: [/172.81.18.321:9042] Error writing))
(MainThread)
2019-10-16 14:47:08,192 [opscenterd] WARN: ProcessingError while calling CreateClusterConfController: Unable to connect to cluster. Error is: Unable to connect to any seed nodes, tried [u'172.81.18.321'] (MainThread)

In the agent.log file in Server B, I see the following

INFO [async-dispatch-5] 2019-10-16 13:53:58,643 Discovered Solr HTTP address(es): ["/172.81.18.321:8983"]
INFO [async-dispatch-5] 2019-10-16 13:53:58,643 Selected Solr HTTP address: /172.81.18.321:8983
INFO [async-dispatch-5] 2019-10-16 13:53:58,643 Starting monitored database connection.
ERROR [async-dispatch-5] 2019-10-16 13:54:00,872 Can't connect to Cassandra (All host(s) tried for query failed (tried: /172.81.18.321:9042 (com.datastax.driver.core.exceptions.TransportException: [/172.81.18.321:9042] Connection has been closed))), retrying soon.

In the above error, the datastax-agent trying to connect to 172.81.18.321 (not 127.0.0.1, so that issues seems to be resolved), which is were DSE installed.

Below are my settings in the address.yaml file:

# stomp_interface: 127.0.0.1
# use_ssl: 0
stomp_interface: 172.13.11.91
hosts: ["172.81.18.321"]
cassandra_user: capstone_user
cassandra_pass: capstone12345

I am able to connect to cassandra using the above credentials:

[jenkins@ip-172-81-18-321 log]$ cqlsh -u capstone_user -p capstone12345 172.81.18.321
Connected to TEST_DSE at 172.81.18.321:9042.
[cqlsh 5.0.1 | DSE 6.0.7 | CQL spec 3.4.5 | DSE protocol v2]
Use HELP for help.

Also, I am wondering if this could be an authentication issue or if its still a networking issue?

opscenter connector python

10 |1000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

DataStax Enterprise is powered by the best distribution of Apache Cassandra ™

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Privacy Policy Terms of Use

Answer 1 · 2019-10-16T02:06:38Z

Erick Ramirez answered Oct 16, '19 Erick Ramirez commented Oct 18, '19

@zico_178040 the clues to the issues are in the log entries. This indicates that OpsCenter doesn't have network connectivity to the node:

2019-10-10 20:28:22,209 [opscenterd] WARN: ProcessingError while calling CreateClusterConfController: Unable to connect to cluster. Error is: Unable to connect to any seed nodes, tried [u'XX.XXX.XXX.XX']

You mentioned that you were able to telnet to the DSE node -- was it to the same IP that is mentioned in the warning? If there is a firewall between OpsCenter and the DSE node, make sure that the required ports are open. See OpsCenter ports reference for details.

In this error, the agent tries to connect to the node on localhost because it doesn't know that it needs to connect to a different IP:

ERROR [async-dispatch-8] 2019-10-10 19:47:02,213 Can't connect to Cassandra (All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.TransportException: [/127.0.0.1:9042] Cannot connect))), retrying soon.

OpsCenter tells the agent which IP to connect to but until you resolve the connectivity issue, you will need to configure the address.yaml and set hosts to the same IP as the native_transport_address (or whatever port 9042 is bound to). The format is:

hosts: ["x.x.x.x"]

See DataStax Agent configuration for more info. You might also be interested in the diagram on the OpsCenter architecture overview page. Cheers!

2 Share

10 |1000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

zico_178040 commented · Oct 16, 2019 at 03:29 PM

Hi Erick, I added a follow up in the answer.

0 ·

Erick Ramirez ♦♦ zico_178040 commented · Oct 18, 2019 at 05:10 AM

It doesn't look like you've imported the certificates with client-to-node encryption enabled. The steps are documented here. Cheers!

0 ·

question