Monitoring the logs in real-time on the command line, can also be very useful when debugging either the rules themselves or when analyzing certain issues. Rather than just looking at the logs, in some situations it might be useful to track the rate of the log messages. A self-written Perl script can be useful as it allows to be flexible when it comes to:

• parsing logs
• formatting the output (with colors or tables or …)
• calculating statistics
• …

The following Perl script uses a few modules which need to be present:

use IO::Async::Timer::Periodic;
use IO::Async::Loop;
use Time::HiRes qw/time/;
use Term::ANSIColor qw(:constants);
use Getopt::Long;

The first two modules can be installed on Debian systems with:

apt install libio-async-perl

The others are part of the normal Perl packages and do not require any extra installation.

Next the script will use a polling mechanism to read from standard output at fixed intervals, to calculate the rate of the unique log lines. The default polling rate is set to 2 seconds but it can be managed through command line parameters:

my $last_poll_time = time;

my $poll_rate = 2;
GetOptions (
    'p|pollrate=i' => \$poll_rate,
);

my $loop = IO::Async::Loop->new;
my $timer = IO::Async::Timer::Periodic->new(
   interval => $poll_rate,
   on_tick  => \&log_rate
);

$timer->start;
$loop->add( $timer );
$loop->run;

Finally, the script will define a subroutine called log_rate, which will read from standard output (or even a file) at each poll interval. Important is of course that the log lines from standard output do not contain unique data such as timestamps. The output must be as generic as possible.

Example:

tail -qf /var/log/ulog/blocked_detailed.json /var/log/ulog/blocked.json /var/log/ulog/passed.json  | jq -r --unbuffered '."oob.prefix"' 
blocked: invalid state
blocked: invalid state
blocked: invalid state
blocked: invalid state
blocked: invalid state
action=blocked
action=blocked
action=blocked
action=blocked
action=blocked
action=passed
action=passed
action=passed
action=passed

The code snippit for log_rate could contain:

sub log_rate {
    local $SIG{ALRM} = sub { die time, " time exceeded to read STDIN\n" };

    alarm($poll_rate);
    my $h;
    eval {
        local $| = 1;
        while (my $line = <>) {
            chomp($line);
            $h->{$line}++;
        }
    };
    alarm(0);

    return unless keys %$h;

    my $delta_time = time - $last_poll_time;
    print DARK WHITE . sprintf("%d: ", time) . RESET;
    print( BOLD WHITE . $_ ." [" . GREEN . sprintf("%.2f/s", $h->{$_}/$delta_time) . BOLD WHITE "] | " . RESET) foreach keys %$h; 
    print "\n";

    $last_poll_time = time;
}

Line 2 will start with declaring the “ALARM” signal. This signal is called when the alarm timeout has been reached (see further below).

Line 4 defines the alarm timeout in seconds: meaning: if everything below line 4 (until the next alarm line) takes longer than the defined timeout in seconds, the “ALRM” signal handler defined at line 2 will be called, which basically stops the code execution with a die (which in theory should stop the script with an exit 1).

Line 5 defines a hash reference which is required down below, to temporarily store unique log lines.

Line 6 until 12 define an eval block. The eval block will catch the ALRM signal die (once reached) without stopping the script with an exit 1. Inside the eval block, the standard output will be read with the diamond operator (<>) and unique lines will be counted and stored in the $h hash reference.

Line 13, right after the eval block, sets to alarm timeout to 0 again, which means it is disabled. This allows that only execution of the eval block will be evaluated for timeout.

Line 15 ensures that only when log lines were discovered and stored in the temporary hash-ref $h, that rates will be printed to the screen.

The rest of the code will take care of printing the discovered log lines with their rates to the screen. Colors from Term::ANSIColor are used to make the output more vivid.

Example output:

The full version of the script can be found at: https://github.com/insani4c/perl_tools/tree/master/log_rate

First, install the ulogd2 package (example is based on Debian/ Ubuntu):

apt install ulogd2

Example: log and drop packets which have an invalid state

# Log and drop all invalid packets                                                                                                                                                                                         
iptables -A INPUT -m conntrack --ctstate INVALID -j NFLOG --nflog-group 123 --nflog-prefix "packet with invalid state"
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

To log all those packets to a file in JSON format, ulogd2 requires the following configuration at /etc/ulogd.conf

[global]                                                                                                                                                                                                           
logfile="syslog"
loglevel=3                                                                                                                                                                                                                   
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_inppkt_NFLOG.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IFINDEX.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IP2STR.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IP2BIN.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_PRINTPKT.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_HWHDR.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_raw2packet_BASE.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_JSON.so"

stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,mac2str1:HWHDR,json1:JSON

[log1]
group=123

[json1]
sync=1
file="/var/log/ulog/netfilter_log.json"

After creating the configuration file, ensure that ulogd2 is restarted and that the directory /var/log/ulog exists

mkdir /var/log/ulog
chown ulog /var/log/ulog
systemctl restart ulogd2.service

Once the above created rule matches, a JSON log line will be written to disk:

tail -1 /var/log/ulog/netfilter_log.json | jq
{
  "timestamp": "2022-03-30T14:46:20.527282+0200",
  "dvc": "Netfilter",
  "raw.pktlen": 52,
  "raw.pktcount": 1,
  "oob.prefix": "packet with invalid state",
  "oob.time.sec": 1648644380,
  "oob.time.usec": 527282,
  "oob.mark": 0,
  "oob.ifindex_in": 2,
  "oob.hook": 1,
  "raw.mac_len": 14,
  "oob.family": 2,
  "oob.protocol": 2048,
  "raw.label": 0,
  "raw.type": 1,
  "raw.mac.addrlen": 6,
  "ip.protocol": 6,
  "ip.tos": 0,
  "ip.ttl": 116,
  "ip.totlen": 52,
  "ip.ihl": 5,
  "ip.csum": 41779,
  "ip.id": 16049,
  "ip.fragoff": 16384,
  "src_port": 58662,
  "dest_port": 445,
  "tcp.seq": 3872158206,
  "tcp.ackseq": 0,
  "tcp.window": 8192,
  "tcp.offset": 0,
  "tcp.reserved": 0,
  "tcp.urg": 0,
  "tcp.ack": 0,
  "tcp.psh": 0,
  "tcp.rst": 0,
  "tcp.syn": 1,
  "tcp.fin": 0,
  "tcp.res1": 0,
  "tcp.res2": 0,
  "tcp.csum": 60039,
  "oob.in": "eth0",
  "oob.out": "",
  "src_ip": "181.122.165.177",
  "dest_ip": "1.1.1.1",
  "mac.saddr.str": "94:f7:ad:4f:81:fc",
  "mac.daddr.str": "aa:aa:aa:aa:aa:aa",
  "mac.str": "aa:aa:aa:aa:aa:aa:94:f7:ad:4f:81:fc:08:00"
}