Logging

The ASKAPsoft HPC applications are written in C/C++ and use the Apache log4cxx logging framework. This allows flexible logging to stdout, file, or other sinks.

Here is an example of a typical log output:

INFO  synthesis.parallel (0, nid6) [2013-12-12 17:32:41] - ASKAP cimager (parallel) running on 913 nodes
INFO  synthesis.parallel (0, nid6) [2013-12-12 17:32:41] - askapparallel; ASKAPsoft==trunk; r20475; 2013-11-25
INFO  synthesis.parallel (0, nid6) [2013-12-12 17:32:41] - Compiled without OpenMP support

The information, from left to right includes:

• Log level
• Origin of the log message
• MPI rank
• Hostname of the node the process is running on
• Date and time log message was created
• Log message

The log levels are enumerated and described below:

Level	Description
FATAL	Indicates a problem which will result in an error condition under which process cannot continue
ERROR	Indicates a problem which results in seriously degraded functionality
WARN	Indicates a problem which results in only slightly degraded functionality
INFO	General information useful for a user monitoring the process
DEBUG	Log messages useful to hardware or software engineers for debugging purposes
TRACE	Very high level of debugging needed for debugging. E.g. Entry/Exit of a function call

All programs have a default logging configuration, essentially writing to STDOUT. You can customise this using a custom log configuration file. For example create the following file, call it askap.log_cfg:

# Configure the rootLogger
log4j.rootLogger=INFO,STDOUT

log4j.appender.STDOUT=org.apache.log4j.ConsoleAppender
log4j.appender.STDOUT.layout=org.apache.log4j.PatternLayout
log4j.appender.STDOUT.layout.ConversionPattern=%-5p %c{2} (%X{mpirank}, %X{hostname}) [%d] - %m%n

Then use the “-l” parameter of the program to tell the program to use your log configuration file instead of the default. E.g:

cimager -c config.in -l askap.log_cfg

The log configuration file above says to log to the console (i.e. STDOUT) however only log messages with level INFO or above (i.e. between INFO and FATAL).

MPI Rank Based Filtering

Many of the data processing programs are MPI based and use a master/worker approach to distributing work. The result can be very verbose logging, with the workers logging somewhat redundant information. It is possible to trivially use grep to search for messages from the master process only (MPI rank zero) however the logging framework can also be setup to do this. For this the RankFilter cab be used. However filtering can only be specified via the XML logging configuration, and this is introduced below.

Here is an example of logging to the console with only MPI rank 0 accepted. It is otherwise identical to the console based logging described above:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
  <appender name="console" class="org.apache.log4j.ConsoleAppender">
    <param name="Target" value="System.out"/>
    <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern" value="%-5p %c{2} (%X{mpirank}, %X{hostname}) [%d] - %m%n"/>
    </layout>
    <filter class="RankFilter">
        <param name="rankToMatch" value="0" />
        <param name="acceptOnMatch" value="true" />
    </filter>
  </appender>

  <root>
    <priority value ="info" />
    <appender-ref ref="console" />
  </root>

</log4j:configuration>

If you have saved this configuration file to a file named logconfig.xml then the program (in this case the cimager) can be configured to use it like so:

cimager -c config.in -l logconfig.xml

In this second example two log appenders are used:

• A ConsoleAppender that writes only messages with rank equal to 0 (i.e. the master process) and level equal to or higher than INFO to STDOUT
• A FileAppender that writes all messages to a file called debug.out

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
    <appender name="console" class="org.apache.log4j.ConsoleAppender">
        <param name="Target" value="System.out"/>
        <param name="Threshold" value="INFO"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%-5p %c{2} (%X{mpirank}, %X{hostname}) [%d] - %m%n"/>
        </layout>
        <filter class="RankFilter">
            <param name="rankToMatch" value="0" />
            <param name="acceptOnMatch" value="true" />
        </filter>
    </appender>

    <appender name="file" class="org.apache.log4j.FileAppender">
        <param name="File" value="debug.log"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%-5p %c{2} (%X{mpirank}, %X{hostname}) [%d] - %m%n"/>
        </layout>
    </appender>

    <root>
        <priority value ="debug" />
        <appender-ref ref="console" />
        <appender-ref ref="file" />
    </root>

</log4j:configuration>