question

cmouse avatar image
cmouse asked Erick Ramirez edited

Cannot connect to TLS enabled Cassandra

DataStax CPP driver version: 2.15.2

Cassandra version: Apache Cassandra 3.11.6

Environment: Debian GNU/Linux 10 (buster)

I am trying to use the datastax cpp driver to connect to a Cassandra instance that has been configured to use SSL. However, when trying with this sample code:

int main(void)
{
        cass_log_set_level(CASS_LOG_TRACE);

        CassCluster *cluster = cass_cluster_new();
        CassSsl *ssl = cass_ssl_new();
        cass_ssl_add_trusted_cert(ssl, server_ca_cert);
        cass_ssl_set_verify_flags(ssl, CASS_SSL_VERIFY_PEER_CERT);
        cass_cluster_set_ssl(cluster, ssl);
        cass_ssl_free(ssl);

        cass_cluster_set_contact_points(cluster, "127.0.0.1");
        cass_cluster_set_protocol_version(cluster, CASS_PROTOCOL_VERSION_V4);

        CassSession *session = cass_session_new();
        cass_session_connect_keyspace(session, cluster, "test");

        cass_session_free(session);
        cass_cluster_free(cluster);
}

I get following Trace output:

1598257812.684 [INFO] (session_base.cpp:86:datastax::internal::core::Future::Ptr datastax::internal::core::SessionBase::connect(const datastax::internal::core::Config&, const String&)): Client id is bbfda964-ddca-4e3c-84fb-cd84d1bbffb1
1598257812.684 [INFO] (session_base.cpp:87:datastax::internal::core::Future::Ptr datastax::internal::core::SessionBase::connect(const datastax::internal::core::Config&, const String&)): Session id is b9e80f52-9a90-4667-8272-77a3211488a3
1598257812.685 [DEBUG] (socket_connector.cpp:226:void datastax::internal::core::SocketConnector::on_connect(datastax::internal::core::TcpConnector*)): Connected to host 127.0.0.1 on socket(0x7f4ea00010c0)
1598257812.699 [TRACE] (connection.cpp:172:int32_t datastax::internal::core::Connection::write(const Ptr&)): Sending message type CQL_OPCODE_STARTUP with stream 0 on host 127.0.0.1
1598257812.699 [TRACE] (socket.cpp:148:void SslSocketWrite::encrypt()): Copying 2 bufs
1598257812.699 [TRACE] (socket.cpp:186:void SslSocketWrite::encrypt()): Copied 157 bytes for encryption
1598257812.699 [TRACE] (socket.cpp:129:virtual size_t SslSocketWrite::flush()): Sending 179 encrypted bytes
1598257812.701 [TRACE] (connection.cpp:277:void datastax::internal::core::Connection::on_read(const char*, size_t)): Consumed message type CQL_OPCODE_READY with stream 0, input 9, remaining 9 on host 127.0.0.1
1598257812.701 [TRACE] (connection.cpp:172:int32_t datastax::internal::core::Connection::write(const Ptr&)): Sending message type CQL_OPCODE_REGISTER with stream 64 on host 127.0.0.1
1598257812.701 [TRACE] (socket.cpp:148:void SslSocketWrite::encrypt()): Copying 2 bufs
1598257812.701 [TRACE] (socket.cpp:186:void SslSocketWrite::encrypt()): Copied 58 bytes for encryption
1598257812.701 [TRACE] (socket.cpp:129:virtual size_t SslSocketWrite::flush()): Sending 80 encrypted bytes
1598257812.702 [TRACE] (connection.cpp:277:void datastax::internal::core::Connection::on_read(const char*, size_t)): Consumed message type CQL_OPCODE_READY with stream 64, input 9, remaining 9 on host 127.0.0.1
1598257812.702 [TRACE] (connection.cpp:172:int32_t datastax::internal::core::Connection::write(const Ptr&)): Sending message type CQL_OPCODE_QUERY with stream 192 on host 127.0.0.1
1598257812.702 [TRACE] (socket.cpp:148:void SslSocketWrite::encrypt()): Copying 3 bufs
1598257812.702 [TRACE] (socket.cpp:186:void SslSocketWrite::encrypt()): Copied 60 bytes for encryption
1598257812.702 [TRACE] (socket.cpp:129:virtual size_t SslSocketWrite::flush()): Sending 82 encrypted bytes
1598257812.702 [TRACE] (connection.cpp:172:int32_t datastax::internal::core::Connection::write(const Ptr&)): Sending message type CQL_OPCODE_QUERY with stream 128 on host 127.0.0.1
1598257812.703 [ERROR] (socket.cpp:216:virtual void datastax::internal::core::SslSocketHandler::on_read(datastax::internal::core::Socket*, ssize_t, const uv_buf_t*)): Unable to decrypt data: 
1598257812.703 [DEBUG] (socket.cpp:373:void datastax::internal::core::Socket::handle_close()): Socket(0x7f4ea0001c30) to host 127.0.0.1 closed
1598257812.703 [ERROR] (cluster_connector.cpp:234:void datastax::internal::core::ClusterConnector::on_connect(datastax::internal::core::ControlConnector*)): Unable to establish a control connection to host 127.0.0.1 because of the following error: Error running host queries on control connection: Request timed out

and on cassandra logs I can see

io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0400008007000000210000001a53454c454354202a2046524f4d2073797374656d2e7065657273000a00

which indicates that SSL is not being attempted. Any suggestion as to what I am doing wrong here?


Additional information at https://gist.github.com/cmouse/264034469d4327fe1cbe08f2a6a920ed

cpp driver
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

cmouse avatar image cmouse commented ·

Turns out this problem went away by reinitializing the entire database server. Not a problem in datastax cpp after all.

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez edited

We'll need additional information about your environment in order to diagnose your problem:

  • the client_encryption_options in cassandra.yaml -- the configured options will dictate how you configure the driver
  • how you generated the certificates on the cluster
  • which certificate you've configured for server_ca_cert
  • whether you've exported the certificate into PEM format
  • your code for loading the certificate

Cheers!

[EDIT] It's really strange that a database restart made the problem go away. That isn't an issue I have come across before. The only hypothesis I could come up with is that you made configuration changes to cassandra.yaml which only took effect when you restarted Cassandra.

I'm glad to hear that you got it sorted in the end.

4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

cmouse avatar image cmouse commented ·

First of all, this is a test setup, so the configured data here is not used in production, and will be gone when I get this working.

cassandra.yaml

client_encryption_options:
  enabled: true
  keystore: /etc/cassandra/keystore.p12
  keystore_password: password
  truststore: /etc/cassandra/truststore.p12
  truststore_password: password
  protocol: TLS
  algorithm: SunX509
  store_type: PKCS12

proof that it actually works

# env SSL_CERTFILE=/tmp/testrun-200824_080933/dict-sql-ssl/etc/dovecot/ec-server-ca.pem cqlsh --ssl 
Connected to ci-cluster at localhost:9042.
[cqlsh 5.0.1 | Cassandra 3.11.6 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> DESCRIBE KEYSPACES;

system_traces  system_schema  system_auth  system  system_distributed

cqlsh>
0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ commented ·

@cmouse It looks like you've tried to post a comment as an "answer" and it isn't allowed since comments are not answers. Instead, I would suggest that you edit your original question if you need to provide additional information.

It also appears that you've tried to post large amounts of text which triggers the spam filter on the platform. We recommend that you upload log excerpts and command outputs to file-sharing sites such as https://gist.github.com/ and post the URL here. Cheers!

0 Likes 0 ·
cmouse avatar image cmouse commented ·

Deleted the answer and moved the stuff into gist. Thank you for pointing out.

0 Likes 0 ·
cmouse avatar image cmouse commented ·

I took a pcap from the traffic, and interestingly it seems that there is TLS traffic going on. So This seems to indicate that there must be something else wrong. But I cannot figure out why cqlsh works just fine, but datastax cpp driver fails with this.

0 Likes 0 ·