Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

danny.chow338_161019 avatar image
danny.chow338_161019 asked ·

How do I configure OpsCenter to monitor a SSL enabled cluster?

I have opscenter/datastax-agent/DSE cassandra v6.8 installed on a virtualbox VM. I am able to start monitor non-SSL enabled cluster. When I enable SSL, opscenterd could not connect. (error in log file below). I suspect something is wrong with my keystore between the key and the cert. This is why the monitored cluster is refusing login.

I have tried all kinds of things using the DSE docs I could find during the past 4 days...

I have setup SSL on cluster and it is working with cqlsh on both 1-way and 2-way SSL with a PEM key and cert obtained from client-keystore.jks.

[connection]
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = /root/.cassandra/client_cert.pem
validate = false
userkey = /root/.cassandra/client_key.pem
usercert = /root/.cassandra/client_cert.pem
# cqlsh 10.0.2.50 --ssl
Connected to OPS_Cluster at 10.0.2.50:9042.
client_encryption_options:
    enabled: true
    keystore: /etc/dse/cassandra/conf/client-keystore.jks
    keystore_password: myKeyPass
    truststore: /etc/dse/cassandra/conf/client-truststore.jks
    truststore_password: myKeyPass
    require_client_auth: false

To start, I reduced it back to 1-way SSL and edit OPS_Cluster.conf with this:

[cassandra]
cql_port = 9042
seed_hosts = 10.0.2.50
ssl_keystore_password = myKeyPass
ssl_keystore = /etc/dse/cassandra/conf/client-keystore.jks
ssl_truststore_password = myKeyPass
ssl_truststore = /etc/dse/cassandra/conf/client-truststore.jks

Verified the cert and key fingerprints also.

keytool -list -keystore /etc/dse/cassandra/conf/client-truststore.jks -storepass myKeyPass
dse_cluster_client, Jul 30, 2020, trustedCertEntry,
Certificate fingerprint (SHA1): 55:95:A2:37:11:94:FC:DD:79:42:15:57:D4:BB:41:13:55:FB:EB:25
node_cert, Aug 24, 2020, trustedCertEntry,
Certificate fingerprint (SHA1): AB:6C:55:EC:0E:A0:3F:50:69:8D:5E:60:9C:64:13:1D:0C:68:56:B9
keytool -list -keystore /etc/dse/cassandra/conf/client-keystore.jks -storepass myKeyPass
dse_cluster_client, Jul 30, 2020, PrivateKeyEntry,
Certificate fingerprint (SHA1): 55:95:A2:37:11:94:FC:DD:79:42:15:57:D4:BB:41:13:55:FB:EB:25

Opscenterd.log said:

[OPS_Cluster] ERROR: Error connecting to the cluster: Traceback (most recent call last):
NoHostAvailable: All host(s) tried for query failed (tried: /10.0.2.50:9042 (com.datastax.driver.core.exceptions.TransportException: [/10.0.2.50:9042] Connection has been closed))

Agent.log said:

ERROR [async-dispatch-5] 2020-08-24 22:11:11,076Z Can't connect to Cassandra (All host(s) tried for query failed (tried: /10.0.2.50:9042 (com.datastax.driver.core.exceptions.TransportException: [/10.0.2.50:9042] Connection has been closed))), retrying soon.

system.log said:

INFO [CoreThread-0] 2020-08-24 15:12:29,986 NoSpamLogger.java:95 - Unexpected exception during request; channel = [id: 0xa6cae760, L:/10.0.2.50:9042 ! R:/10.0.2.50:55344]
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
opscenter
1 comment
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Adding info:

for the node-to-node SSL, the keystore/trustore use CaRoot (self-signed).

for client-to-node, there is no CA. I created a keypair into keystore, extract the cert into truststore. It worked for cqlsh.


0 Likes 0 · ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

It's not clear to me whether you followed the correct instructions to configure OpsCenter to connect to a DSE cluster with client-to-node encryption.

High-level steps

As part of the process, you are supposed to:

  • create a keystore (preferably named opscenter.jks for simplicity) on the opscenterd server
  • export the opscenterd certificate
  • create a truststore on the opscenterd server and import the each DSE nodes' certificates

Using the OpsCenter keystore and truststore files you created, you were then supposed to configure the cluster connection settings in the cluster_name.conf file (with cluster_name being the name of your DSE cluster).

Your configuration

However based on the information you supplied, it doesn't look like you've followed the instructions correctly. It looks like you are using the keystore/truststore of the DSE nodes instead of the OpsCenter keystore/truststore files:

ssl_keystore = /etc/dse/cassandra/conf/client-keystore.jks
ssl_truststore = /etc/dse/cassandra/conf/client-truststore.jks

Recommendation

Please follow the steps in Connect to DSE with client-to-node encryption in OpsCenter and the DataStax Agents.

It's very important that you follow the steps exactly as documented. If you miss a step or don't follow it correctly, the whole setup won't work because that's the nature of SSL encryption.

It is very difficult to troubleshoot encryption-related issues in the Q&A format of this forum so if you require further assistance, I would suggest you log a ticket with DataStax Support if your organisation has a current DSE subscription so one of our engineers can work with you directly. Cheers!

6 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Thanks for responding to my question. I was actually using the same link last week (step 1 to 3) but I may have missed something or confused.

Question: "Create a truststore on the opscenterd machine and import each node's public certificate. ", that means the public cert from server-truststore.jks (node-to-node). Or is it the cert from client-truststore.jks? (client-to-node)

keytool -import -v -trustcacerts -alias trusted_DSE_Cluster_node -file /root/ca/cluster_node.crt_signed -keystore /root/ca/opscenter-truststore.jks -noprompt -keypass OpsKeyPass -storepass OpsKeyPass

0 Likes 0 · ·
Erick Ramirez avatar image Erick Ramirez ♦♦ danny.chow338_161019 ·

The node (server) certificate. Cheers!

0 Likes 0 · ·

opscenterd.log:

[OPS_Cluster] ERROR: Error connecting to the cluster

NoHostAvailable: All host(s) tried for query failed (tried: /10.0.2.50:9042 (com.datastax.driver.core.exceptions.TransportException: [/10.0.2.50:9042] Connection has been closed))

system.log:

io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record


keytool -exportcert -keystore /root/ca/server-keystore.jks -storepass myKeyPass -alias cluster_node -file /root/ca/cluster_node.cert

keytool -import -v -trustcacerts -alias cluster_node_cert -file /root/ca/cluster_node.cert -keystore /root/ca/opscenter-truststore.jks -noprompt -keypass OpsKeyPass -storepass OpsKeyPass

keytool -list -keystore /etc/opscenter/opscenter-truststore.jks -storepass OpsKeyPass

cluster_node_cert, Aug 25, 2020, trustedCertEntry,

Certificate fingerprint (SHA1): AB:6C:55:EC:0E:A0:3F:50:69:8D:5E:60:9C:64:13:1D:0C:68:56:B9

0 Likes 0 · ·
Show more comments
[agents]
ssl_truststore = /etc/opscenter/opscenter-truststore.jks
ssl_truststore_password = OpsKeyPass
storage_ssl_keystore = /etc/opscenter/opscenter-keystore.jks
storage_ssl_keystore_password = OpsKeyPass
storage_ssl_truststore = /etc/opscenter/opscenter-truststore.jks
storage_ssl_truststore_password = OpsKeyPass

[jmx]
password =
port = 7199
username =

[cassandra]
cql_port = 9042
seed_hosts = 10.0.2.50
ssl_keystore_password = OpsKeyPass
ssl_keystore = /etc/opscenter/opscenter-keystore.jks
ssl_truststore_password = OpsKeyPass
ssl_truststore = /etc/opscenter/opscenter-truststore.jks
username = cassandra
password = cassandra

[storage_cassandra]
username = cassandra
password = cassandra
seed_hosts = 10.0.2.50
cql_port = 9042
keyspace = ops_cluster_metric
0 Likes 0 · ·

keytool -genkeypair -keyalg RSA -validity 3650 -keysize 2048 -alias opscenter_key -keypass OpsKeyPass -storepass OpsKeyPass -keystore /root/ca/opscenter-keystore.jks -dname "CN=opscenter-keystore, OU= OPS_Cluster, O=VBox, C=US"

keytool -exportcert -keystore /root/ca/server-keystore.jks -storepass myKeyPass -alias cluster_node -file /root/ca/cluster_node.cert

keytool -import -v -trustcacerts -alias cluster_node_cert -file /root/ca/cluster_node.cert -keystore /root/ca/opscenter-truststore.jks -noprompt -keypass OpsKeyPass -storepass OpsKeyPass

keytool -list -keystore /root/ca/opscenter-truststore.jks -storepass OpsKeyPass

cluster_node_cert, Aug 25, 2020, trustedCertEntry,

Certificate fingerprint (SHA1): AB:6C:55:EC:0E:A0:3F:50:69:8D:5E:60:9C:64:13:1D:0C:68:56:B9


system.log

io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record

0 Likes 0 · ·