question

Shannu avatar image
Shannu asked Erick Ramirez answered

Unable to configure Reaper for Apache Cassandra 3.11.11

I am facing below issue adding a cluster in reaper UI.

INFO [2021-08-17 08:27:06,471] [dw-39 - POST /cluster/auth] i.c.r.ClusterResource - POST addOrUpdateCluster called with seedHost: 172.31.22.173
ERROR [2021-08-17 08:27:06,635] [dw-39 - POST /cluster/auth] i.c.j.JmxConnectionFactory - Failed creating a new JMX connection to 172.31.22.173:7199
java.lang.RuntimeException: io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 172.31.22.173:7199
    at io.cassandrareaper.jmx.JmxConnectionFactory$JmxConnectionProvider.apply(JmxConnectionFactory.java:250)
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
    at io.cassandrareaper.jmx.JmxConnectionFactory.connectImpl(JmxConnectionFactory.java:111)
    at io.cassandrareaper.jmx.JmxConnectionFactory.connectAny(JmxConnectionFactory.java:148)
    at io.cassandrareaper.jmx.ClusterFacade.connectImpl(ClusterFacade.java:889)
    at io.cassandrareaper.jmx.ClusterFacade.connect(ClusterFacade.java:873)
    at io.cassandrareaper.jmx.ClusterFacade.getClusterName(ClusterFacade.java:142)
    at io.cassandrareaper.resources.ClusterResource.findClusterWithSeedHost(ClusterResource.java:315)
    at io.cassandrareaper.resources.ClusterResource.addOrUpdateCluster(ClusterResource.java:239)
    at io.cassandrareaper.resources.ClusterResource.addOrUpdateCluster(ClusterResource.java:175)
    ...
Caused by: io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 172.31.22.173:7199
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:286)
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:179)
    at io.cassandrareaper.jmx.JmxConnectionFactory$JmxConnectionProvider.apply(JmxConnectionFactory.java:246)
    ... 98 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
 javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:206)
    at io.cassandrareaper.jmx.JmxProxyImpl.connectWithTimeout(JmxProxyImpl.java:304)
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:242)
    ... 100 common frames omitted
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
 javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:307)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132)
    ...
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
    ...
INFO [2021-08-17 08:27:06,635] [dw-39 - POST /cluster/auth] i.c.j.JmxConnectionFactory - Unreachable host:
io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 172.31.22.173:7199
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:286)
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:179)
    at io.cassandrareaper.jmx.JmxConnectionFactory$JmxConnectionProvider.apply(JmxConnectionFactory.java:246)
    ...
Caused by: java.util.concurrent.ExecutionException: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
 javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:206)
    at io.cassandrareaper.jmx.JmxProxyImpl.connectWithTimeout(JmxProxyImpl.java:304)
    at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:242)
    ... 100 common frames omitted
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
 javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:307)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
    ...
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
    ...
ERROR [2021-08-17 08:27:06,635] [dw-39 - POST /cluster/auth] i.c.r.ClusterResource - failed to find cluster with seed hosts: [172.31.22.173]
io.cassandrareaper.ReaperException: no host could be reached through JMX
    at io.cassandrareaper.jmx.JmxConnectionFactory.connectAny(JmxConnectionFactory.java:163)
    at io.cassandrareaper.jmx.ClusterFacade.connectImpl(ClusterFacade.java:889)
    at io.cassandrareaper.jmx.ClusterFacade.connect(ClusterFacade.java:873)
    at io.cassandrareaper.jmx.ClusterFacade.getClusterName(ClusterFacade.java:142)
    at io.cassandrareaper.resources.ClusterResource.findClusterWithSeedHost(ClusterResource.java:315)
    at io.cassandrareaper.resources.ClusterResource.addOrUpdateCluster(ClusterResource.java:239)
    at io.cassandrareaper.resources.ClusterResource.addOrUpdateCluster(ClusterResource.java:175)
    ...

cassandra-env.sh file.

JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=172.**.**.173"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=7199"
if [ "x$LOCAL_JMX" = "x" ]; then
    LOCAL_JMX=yes
fi
JMX_PORT="7199"
if [ "$LOCAL_JMX" = "yes" ]; then
JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
else
JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.remote.port=$JMX_PORT"
# if ssl is enabled the same port cannot be used for both jmx and rmi so either
# pick another value for this property or comment out to use a random port (though see CASSANDRA-7087 for origins)
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
# turn on JMX authentication. See below for further options
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
# jmx ssl options
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=true"
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true"
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=<enabled-protocols>"
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=<enabled-cipher-suites>"
#JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore"
#JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>"
#JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/path/to/truststore"
#JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>"
fi
# jmx authentication and authorization options. By default, auth is only
# activated for remote connections but they can also be enabled for local only JMX
## Basic file based authn & authz
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.access.file=/etc/cassandra/jmxremote.access"
## Custom auth settings which can be used as alternatives to JMX's out of the box auth utilities.
## JAAS login modules can be used for authentication by uncommenting these two properties.

@alex.dejanovski_191715 @Erick Ramirez Could you help me out with this

Thanks in advance

reaper
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

By default, JMX access is only accessible on localhost so Reaper cannot access the nodes remotely.

To enable remote access in cassandra-env.sh:

LOCAL_JMX=no                    <----- add this line
if [ "x$LOCAL_JMX" = "x" ]; then
    LOCAL_JMX=yes
fi
Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.