Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

Shannu avatar image
Shannu asked Erick Ramirez answered

Unable to configure Reaper for Apache Cassandra 3.11.11

I am facing below issue adding a cluster in reaper UI.

INFO [2021-08-17 08:27:06,471] [dw-39 - POST /cluster/auth] i.c.r.ClusterResource - POST addOrUpdateCluster called with seedHost: 172.31.22.173
ERROR [2021-08-17 08:27:06,635] [dw-39 - POST /cluster/auth] i.c.j.JmxConnectionFactory - Failed creating a new JMX connection to 172.31.22.173:7199
java.lang.RuntimeException: io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 172.31.22.173:7199
    at io.cassandrareaper.jmx.JmxConnectionFactory$JmxConnectionProvider.apply(JmxConnectionFactory.java:250)
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
    at io.cassandrareaper.jmx.JmxConnectionFactory.connectImpl(JmxConnectionFactory.java:111)
    at io.cassandrareaper.jmx.JmxConnectionFactory.connectAny(JmxConnectionFactory.java:148)
    at io.cassandrareaper.jmx.ClusterFacade.connectImpl(ClusterFacade.java:889)
    at io.cassandrareaper.jmx.ClusterFacade.connect(ClusterFacade.java:873)
    at io.cassandrareaper.jmx.ClusterFacade.getClusterName(ClusterFacade.java:142)
    at io.cassandrareaper.resources.ClusterResource.findClusterWithSeedHost(ClusterResource.java:315)
    at io.cassandrareaper.resources.ClusterResource.addOrUpdateCluster(ClusterResource.java:239)
    at io.cassandrareaper.resources.ClusterResource.addOrUpdateCluster(ClusterResource.java:175)
    ...
Caused by: io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 172.31.22.173:7199
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:286)
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:179)
    at io.cassandrareaper.jmx.JmxConnectionFactory$JmxConnectionProvider.apply(JmxConnectionFactory.java:246)
    ... 98 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
 javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:206)
    at io.cassandrareaper.jmx.JmxProxyImpl.connectWithTimeout(JmxProxyImpl.java:304)
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:242)
    ... 100 common frames omitted
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
 javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:307)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132)
    ...
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
    ...
INFO [2021-08-17 08:27:06,635] [dw-39 - POST /cluster/auth] i.c.j.JmxConnectionFactory - Unreachable host:
io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 172.31.22.173:7199
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:286)
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:179)
    at io.cassandrareaper.jmx.JmxConnectionFactory$JmxConnectionProvider.apply(JmxConnectionFactory.java:246)
    ...
Caused by: java.util.concurrent.ExecutionException: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
 javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:206)
    at io.cassandrareaper.jmx.JmxProxyImpl.connectWithTimeout(JmxProxyImpl.java:304)
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:242)
    ... 100 common frames omitted
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
 javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:307)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
    ...
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
    ...
ERROR [2021-08-17 08:27:06,635] [dw-39 - POST /cluster/auth] i.c.r.ClusterResource - failed to find cluster with seed hosts: [172.31.22.173]
io.cassandrareaper.ReaperException: no host could be reached through JMX
    at io.cassandrareaper.jmx.JmxConnectionFactory.connectAny(JmxConnectionFactory.java:163)
    at io.cassandrareaper.jmx.ClusterFacade.connectImpl(ClusterFacade.java:889)
    at io.cassandrareaper.jmx.ClusterFacade.connect(ClusterFacade.java:873)
    at io.cassandrareaper.jmx.ClusterFacade.getClusterName(ClusterFacade.java:142)
    at io.cassandrareaper.resources.ClusterResource.findClusterWithSeedHost(ClusterResource.java:315)
    at io.cassandrareaper.resources.ClusterResource.addOrUpdateCluster(ClusterResource.java:239)
    at io.cassandrareaper.resources.ClusterResource.addOrUpdateCluster(ClusterResource.java:175)
    ...

cassandra-env.sh file.

JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=172.**.**.173"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=7199"
if [ "x$LOCAL_JMX" = "x" ]; then
    LOCAL_JMX=yes
fi
JMX_PORT="7199"
if [ "$LOCAL_JMX" = "yes" ]; then
JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
else
JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.remote.port=$JMX_PORT"
# if ssl is enabled the same port cannot be used for both jmx and rmi so either
# pick another value for this property or comment out to use a random port (though see CASSANDRA-7087 for origins)
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
# turn on JMX authentication. See below for further options
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
# jmx ssl options
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=true"
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true"
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=<enabled-protocols>"
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=<enabled-cipher-suites>"
#JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore"
#JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>"
#JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/path/to/truststore"
#JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>"
fi
# jmx authentication and authorization options. By default, auth is only
# activated for remote connections but they can also be enabled for local only JMX
## Basic file based authn & authz
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.access.file=/etc/cassandra/jmxremote.access"
## Custom auth settings which can be used as alternatives to JMX's out of the box auth utilities.
## JAAS login modules can be used for authentication by uncommenting these two properties.

@alex.dejanovski_191715 @Erick Ramirez Could you help me out with this

Thanks in advance

reaper
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

By default, JMX access is only accessible on localhost so Reaper cannot access the nodes remotely.

To enable remote access in cassandra-env.sh:

LOCAL_JMX=no                    <----- add this line
if [ "x$LOCAL_JMX" = "x" ]; then
    LOCAL_JMX=yes
fi
Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.