question

francisdomoney@aol.com avatar image
francisdomoney@aol.com asked Cedrick Lunven commented

Why am I getting vast amounts of log messages in my k8s cluster from the workshop?

While trying to run

$ kubectl -n cass-operator logs cluster2-dc2-default-sts-0 -c cassandra

I get vast amounts of

INFO [nioEventLoopGroup-2-2] 2020-07-30 09:09:56,453 Cli.java:555 - address=/10.244.4.1:57790 url=/api/v0/probes/readiness status=500 Internal Server Error
INFO [nioEventLoopGroup-2-1] 2020-07-30 09:10:02,990 Cli.java:555 - address=/10.244.4.1:57876 url=/api/v0/probes/liveness status=200 OK
INFO [epollEventLoopGroup-404-1] 2020-07-30 09:10:06,445 Clock.java:35 - Could not access native clock (see debug logs for details), falling back to Java system clock
WARN [epollEventLoopGroup-404-2] 2020-07-30 09:10:06,447 Slf4JLogger.java:146 - Unknown channel option 'TCP_NODELAY' for channel '[id: 0x292fd5c8]'
WARN [epollEventLoopGroup-404-2] 2020-07-30 09:10:06,450 Loggers.java:28 - [s399] Error connecting to /tmp/dse.sock, trying next node
java.io.FileNotFoundException: null
    at io.netty.channel.unix.Errors.throwConnectException(Errors.java:110)
    at io.netty.channel.unix.Socket.connect(Socket.java:257)
    at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:732)
    at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:717)
    at io.netty.channel.epoll.EpollDomainSocketChannel.doConnect(EpollDomainSocketChannel.java:87)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:559)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366)
    at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
    at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
    at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
    at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
    at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
    at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
    at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
    at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
    at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
    at com.datastax.oss.driver.internal.core.channel.ConnectInitHandler.connect(ConnectInitHandler.java:57)
    at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
    at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
    at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512)
    at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024)
    at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:276)
    at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:375)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
INFO [nioEventLoopGroup-2-2] 2020-07-30 09:10:06,453 Cli.java:555 - address=/10.244.4.1:57920 url=/api/v0/probes/readiness status=500 Internal Server Error

I don't thinks this is log output. It looks more like a series of warnings and error messages

workshopcass-operator
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Cedrick Lunven avatar image Cedrick Lunven ♦ commented ·

Hello,

I only see a WARN and INFO logs in your stack logs (even with a stack trace)

`WARN:` Not having `/tmp/dse.sock` mean you don't have the native clock but you will use some java one (probably docker constraint)

`INFO:` The server is still initializing and the readyness probe does not return 200 code but 500. This will be the case until the node finish initializing after a few minutes


no Crashing.

0 Likes 0 ·

1 Answer

john.sanda_194109 avatar image
john.sanda_194109 answered Cedrick Lunven commented

Those logs are from the management api. To get Cassandra's logs use,

kubectl -n cass-operator logs cluster2-dc2-default-sts-0 -c server-system-logger

The cassandra container starts the management api which in turns creates Cassandra as a child process in the same container.

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

francisdomoney@aol.com avatar image francisdomoney@aol.com commented ·

My problem with no example output was knowing id the thing had crashed


0 Likes 0 ·
Cedrick Lunven avatar image Cedrick Lunven ♦ francisdomoney@aol.com commented ·

You should rather go with `get pods` command

to see if the pod is still initializing, ImagePullBackoff or ready.


0 Likes 0 ·