question

Author - Greg Smith.

Since DSE 4.8, DataStax Enterprise has leveraged logback for all logging. Logback has many advantages over the older log4j logging project, one of the main benefits is the ability to change the logging configuration dynamically, without the need to restart the DSE software. This is particularly useful in a production environment where obtaining permission to restart production nodes may not be possible.

Starting with DSE 5.0 the output has been split between the traditional system.log and a new debug.log. This change which is discussed in CASSANDRA-10241 was aimed at separating lower level information designed for troubleshooting, from the normal day-to-day logging. Note, this change has resulted in numerous messages that were originally logged at INFO level being moved to DEBUG level, such as compaction and flush information. The system.log now contains a subset of information that is available in the debug.log. For this reason it is recommended that the new debug log is not disabled as it contains information that is useful for troubleshooting production issues.

logback configuration

The logback configuration controls what is logged into what log files and how those files are handled. The what and the where is all controlled by elements within the logback.xml configuration file. The file is large but is split into various elements which we will go through in this section.

Automatic reload of logging configuration

If instructed to do so, logback will scan for changes in its configuration file and automatically reconfigure itself with the configuration file changes. By default the configuration file will be scanned for changes every minute. To enable this behaviour ensure that the scan attribute is set correctly.

<configuration scan="true">

Appenders

The logback configuration that is supplied with DSE 5.0+ contains several different appenders for system, debug, gremlin, solr and audit logs. The basic syntax of these appenders is the same and for simplicity we will focus on the system log appender.

  <appender name="SYSTEMLOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
      <level>INFO</level>
    </filter>
    <file>${cassandra.logdir}/system.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
      <fileNamePattern>${cassandra.logdir}/system.log.%i.zip</fileNamePattern>
      <minIndex>1</minIndex>
      <maxIndex>20</maxIndex>
    </rollingPolicy>
    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
      <maxFileSize>20MB</maxFileSize>
    </triggeringPolicy>
    <encoder>
      <pattern>%-5level [%thread] %date{ISO8601} %X{service} %F:%L - %msg%n</pattern>
      <!-- old-style log format
      <pattern>%5level [%thread] %date{ISO8601} %F (line %L) %msg%n</pattern>
      -->
    </encoder>
  </appender>

This appender is used to write filtered information to the system.log. Each individual file will be a maximum of 20MB:

      <maxFileSize>20MB</maxFileSize>

Note that to save space each of the saved files will also be compressed. The SYSTEMLOG appender also contains a ThresholdFilter clause, unlike the DEBUGLOG appender. This clause will only log information provided it is at a level of INFO or higher, so any DEBUG messages will not appear in the system.log.

These appenders are then referenced in the root element:

  <root level="${logback.root.level:-INFO}">
    <appender-ref ref="SYSTEMLOG" />
    <appender-ref ref="ASYNCDEBUGLOG" /> 
    ...
  </root>

It is possible to define additional appenders with different file formats, retention policies or locations. For example, below is a custom log appender and a centralised syslog appender:

  <appender name="CUSTOM" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${cassandra.logdir}/custom.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
      <fileNamePattern>${cassandra.logdir}/custom.log.%i.zip</fileNamePattern>
      <minIndex>1</minIndex>
      <maxIndex>20</maxIndex>
    </rollingPolicy>
    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
      <maxFileSize>20MB</maxFileSize>
    </triggeringPolicy>
    <encoder>
      <pattern>%-5level [%thread] %date{ISO8601} %X{service} %F:%L - %msg%n</pattern>
    </encoder>
  </appender>
 
<appender name="SYSLOG" class="ch.qos.logback.classic.net.SyslogAppender">
    <syslogHost>10.200.180.166</syslogHost>
    <port>514</port>
    <facility>USER</facility>
    <suffixPattern>%-5level [%thread] %logger  %date{ISO8601} %X{service} %F:%L - %msg%n</suffixPattern>
  </appender>

The logback documentation contains a full list of the various appenders that are supported.

Loggers

The loggers determine what is logged to the various appenders. For example check out the following logger:

<logger name="org.apache.cassandra" level="DEBUG"/>

This logger will ensure that any messages from org.apache.cassandra at DEBUG level or higher will be sent to all the appenders defined in the root level appender. If the appenders are not referenced in the root level appender it is possible to specify which appenders to use.

  <!-- Custom loggers -->
  <logger name="org.apache.cassandra" level="WARN" additivity="true">
    <appender-ref ref="SYSLOG"/>
    <appender-ref ref="CUSTOM"/>
  </logger>

In this case, any WARN or ERROR messages will be sent to the custom.log and also the centralised syslog servers. The additivity attribute is important and ensures that messages are logged to these appenders in addition to any other defined appenders. In this case system.log and debug.log.

For additional information on logback and logging check out the links below - logback is a very powerful and flexible logging facility. Finally, to assist support on diagnosing issues, please do not disable the debug.log. Using the information in this article it should be possible to add additional appenders and loggers to meet your logging requirements.

Support KB - Configuring Logging in Apache Cassandra

DataStax Docs - Configuring logging