Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

michael.guissine_30999 avatar image
michael.guissine_30999 asked Erick Ramirez commented

DSE 6.7.x fails to start on Centos 7.7 (latest)

DSE (6.7.4) fails to start after CentOS was upgraded from 7.6 to 7.7 (latest). We tried upgrading DSE to 6.7.6 but still seeing the same issue, downgrading CentOS back to 7.6 resolves the issue.

This is the error we are seeing in system.log file

ERROR [main] 2019-10-25 20:37:34,728  DseDaemon.java:550 - Unable to start DSE server.                                              
java.lang.RuntimeException: Failed to setup secure pipeline                                                                         
        at org.apache.cassandra.transport.Server$AbstractSecureIntializer.<init>(Server.java:435)                                   
        at org.apache.cassandra.transport.Server$SecureInitializer.<init>(Server.java:493)                                          
        at org.apache.cassandra.transport.Server.start(Server.java:144)                                                             
        at java.util.ArrayList.forEach(ArrayList.java:1257)                                                                         
        at org.apache.cassandra.service.NativeTransportService.start(NativeTransportService.java:131)                               
        at org.apache.cassandra.service.CassandraDaemon.startNativeTransport(CassandraDaemon.java:795)                              
        at com.datastax.bdp.server.DseDaemon.startNativeTransport(DseDaemon.java:580)                                               
        at org.apache.cassandra.service.CassandraDaemon.lambda$start$8(CassandraDaemon.java:589)                                    
        at org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:601)                                             
        at com.datastax.bdp.server.DseDaemon.start(DseDaemon.java:545)                                                              
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:751)                                          
        at com.datastax.bdp.DseModule.main(DseModule.java:96)                                                                       
Caused by: java.io.IOException: Error creating the initializing the SSL Context                                                     
        at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:207)                                           
        at org.apache.cassandra.transport.Server$AbstractSecureIntializer.<init>(Server.java:431)                                   
        ... 11 common frames omitted                                                                                                
Caused by: java.security.NoSuchAlgorithmException: DseClientReloadableTrustManager KeyManagerFactory not available                  
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)                                                           
        at javax.net.ssl.KeyManagerFactory.getInstance(KeyManagerFactory.java:137)                                                  
        at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:183)                                           
        ... 12 common frames omitted                                                                                                
ERROR [main] 2019-10-25 20:37:34,730  CassandraDaemon.java:862 - Exception encountered during startup                               
java.lang.RuntimeException: java.lang.RuntimeException: Failed to setup secure pipeline                                             
        at com.datastax.bdp.server.DseDaemon.start(DseDaemon.java:553)                                                              
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:751)                                          
        at com.datastax.bdp.DseModule.main(DseModule.java:96)                                                                       
Caused by: java.lang.RuntimeException: Failed to setup secure pipeline                                                              
        at org.apache.cassandra.transport.Server$AbstractSecureIntializer.<init>(Server.java:435)                                   
        at org.apache.cassandra.transport.Server$SecureInitializer.<init>(Server.java:493)                                          
        at org.apache.cassandra.transport.Server.start(Server.java:144)                                                             
        at java.util.ArrayList.forEach(ArrayList.java:1257)                                                                         
        at org.apache.cassandra.service.NativeTransportService.start(NativeTransportService.java:131)                               
        at org.apache.cassandra.service.CassandraDaemon.startNativeTransport(CassandraDaemon.java:795)                              
        at com.datastax.bdp.server.DseDaemon.startNativeTransport(DseDaemon.java:580)                                               
        at org.apache.cassandra.service.CassandraDaemon.lambda$start$8(CassandraDaemon.java:589)                                    
        at org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:601)                                             
        at com.datastax.bdp.server.DseDaemon.start(DseDaemon.java:545)                                                              
        ... 2 common frames omitted                                                                                                 
Caused by: java.io.IOException: Error creating the initializing the SSL Context                                                     
        at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:207)                                           
        at org.apache.cassandra.transport.Server$AbstractSecureIntializer.<init>(Server.java:431)                                   
        ... 11 common frames omitted                                                                                                
Caused by: java.security.NoSuchAlgorithmException: DseClientReloadableTrustManager KeyManagerFactory not available                  
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)                                                           
        at javax.net.ssl.KeyManagerFactory.getInstance(KeyManagerFactory.java:137)                                                  
        at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:183)                                           
        ... 12 common frames omitted


dsedse 6.7.4
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

2 Answers

Lewisr650 avatar image
Lewisr650 answered Erick Ramirez commented

DSE is not currently supported on CentOS 7.7. You can verify supported platforms here:

https://docs.datastax.com/en/landing_page/doc/landing_page/supportedPlatforms.html

8 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

michael.guissine_30999 avatar image michael.guissine_30999 ·

Thanks Lewisr650! Any word on when DSE will support Centos 7.7? Centos 7.7 includes numerous critical security updates which we would like to install as soon as possible.

0 Likes 0 ·
mscrivo_94376 avatar image mscrivo_94376 ·

@Lewisr650 CentOS 7.7 was released on 2019-08-06. DSE 7.6 still does not work on it. Is there a good reason for that? Seems overly long to me.

0 Likes 0 ·
Lewisr650 avatar image Lewisr650 mscrivo_94376 ·

CentOS 7.7 is not currently something we test with based on customer requests. You can submit a a request for CentOS 7.7 to be supported, but then out engineering and testing processes have to go through their cycles. Is there something specific in CentOS 7.7 you are trying to take advantage of?

0 Likes 0 ·
mscrivo_94376 avatar image mscrivo_94376 Lewisr650 ·

Yes, security updates. Which are likely what broke DSE in CentOS 7.7.


I'm not sure I understand your logic. I would understand if you did not yet support CentOS 8 because it's a fairly new major release. But if you support CentOS 7.x, then you need to support the latest version because everyone who is on CentOS 7.6 for example, will get 7.7 when they do a simple `yum update`

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ ·

@michael.guissine_30999, @mscrivo_94376 The evaluation of CentOS 7.7 was completed but there were some issues unrelated to DSE that we uncovered.

We have received reports that upgrading from CentOS 7.6 to 7.7 has some performance degradation irrespective of the software/application/database that is running on the server.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Erick Ramirez ♦♦ ·

@michael.guissine_30999 & @mscrivo_94376 I forgot to mention that I maintain this FAQ which lists new OS releases being tested. Cheers!

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ ·

Furthermore, users who have installed CentOS 7.7 from the start (not upgraded from 7.6) do not run into this performance problem. You will need to conduct your own testing to see how this issue affects you. In the meantime, we will see how this is resolved in RHEL/CentOS. Cheers!

0 Likes 0 ·
mscrivo_94376 avatar image mscrivo_94376 Erick Ramirez ♦♦ ·

Thanks @Erick Ramirez .. that's helpful information. We'll do some experiments on our end and then open a ticket if we can't figure it out.

0 Likes 0 ·
Erick Ramirez avatar image
Erick Ramirez answered

@michael.guissine_30999 specifically for this exception:

Caused by: java.security.NoSuchAlgorithmException: DseClientReloadableTrustManager KeyManagerFactory not available

Do you have the steps to reproduce it? We haven't had this reported before and it appears as though one of the CentOS 7.7 security patches breaks JCE.

FWIW DseClientReloadableTrustManager is just an extension of X509ExtendedTrustManager and the NoSuchAlgorithmException gets thrown when it tries to load the trustStore.

We need to have a look at your cluster's configuration so we can try and replicate the problem. The best way to get the information we require is for you (or someone in your organisation) to log a DataStax Support ticket and attach a Diagnostics Report from Opscenter. Let me know the ticket details and I will work with the Support engineers internally. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.