To prevent this issue, three steps are required:

• change the order kernel modules get loaded by initrd
• disable the mdadm auto-assembly of disks when initrd is loaded
• generate a new initrd image

The initrd configuration files can be found at /etc/initramfs-tools.

First define the order of module loading during boot (ensure that multipath is loaded before mdadm). Edit the file /etc/initramfs-tools/modules to match:

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax: module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
libfc
megaraid_sas
scsi_dh_alua
scsi_transport_fc
dm_multipath
dm_service_time
multipath
md_mod

This starts by loading the “fiber channel libs” (libfc) first, followed by the MegaRAID modules, SCSI over fiber channel, multipath and mdadm.

Now copy the mdadm initramfs hook to /etc to override the default one:

cp /usr/share/initramfs-tools/hooks/mdadm /etc/initramfs-tools/hooks/

Edit the hook file in /etc/initramfs-tools/hooks/mdadm and add an ‘exit 0‘ (see line 10):

#!/bin/sh
#
# Copyright © 2006-2008 Martin F. Krafft <madduck@debian.org>,
#
2012 Michael Tokarev <mjt@tls.msk.ru>
# based on the scripts in the initramfs-tools package.
# released under the terms of the Artistic Licence.
#
set -eu
exit 0
PREREQ="udev"
prereqs()
{
echo "$PREREQ"
}

Disable automatic startup of the mdadm service:

rm /lib/systemd/system/mdadm.service
systemctl daemon-reload
systemctl disable mdadm.service

Finally, update the initrd images for all kernel versions installed:

update-initramfs -u -k all

Reboot the server to see if modules are now loaded in the correct order, and multipath is loaded and started before mdadm.

Many graphical interfaces are available for managing user accounts in OpenLDAP like PHPLDAPAdmin (http://phpldapadmin.sourceforge.net/wiki/index.php/Main_Page) or LAM (https://www.ldap-account-manager.org/lamcms/).

When generating a bulk amount of accounts with automation or just managing user details with a simple script, allows much more flexibility and can be even quicker.

LDAP passwords can be stored or changed by using an LDIF file. This LDIF file needs 3 required lines:

1. The “dn” you are about to change
2. the “changetype” set to “modify“
3. A “replace” line containing the field you want to change (in our case, since we are changing the password, this will be “userPassword“)

Your LDAP password can be stored either in clear-text (which is not advisable) or by sending a SHA-hash. The SHA-hash must include the salt at the end and must be base64 encoded.

The code snippit below will call a subroutine called generate_password() which comes from a previous article (Secure Password Generator in Perl).

At the end of the script, it will print out the LDIF file content, which needs to be saved to change.ldif. As last, it will print the ldapmodify command to make the actual change. You will need to know the admin password for this. Alternatively, you could also make this change using your own dn for authentication.

use Digest::SHA;
use MIME::Base64;

my $random_password = generate_password(24);
my $random_salt     = generate_password(3);

my $ctx = Digest::SHA->new;
$ctx->add($random_password);
$ctx->add($random_salt);
my $hashedPasswd = encode_base64($ctx->digest . $random_salt, '');

print "password: $random_password\n";
print "salt: $random_salt\n";
print <<EOF;
# LDIF
dn: uid=user1,ou=users,dc=shihai-corp,dc=at
changetype: modify
replace: userPassword
userPassword: {SSHA}$hashedPasswd
EOF

print "\n";
print q{LDAP cmd: ldapmodify -H "ldap://ldap_server01" -Z -x -W -D "cn=ldapadmin,ou=admins,dc=shihai-corp,dc=at" -f change.ldif} . "\n\n"

AppArmor will write messages to the kernel log (visible with either the dmesg command or in kernel.log if available) regarding its actions.

If your libvirt guests are not starting up or failing, have a look at dmesg. Example:

In the above example AppArmor has denied (apparmor="DENIED") read access (requested_mask=r) to the file /data/vms/cluster_storage/base-os-ubuntu-focal.qcow2. This blocks of course the startup guest machines we have previously created in the article: Terraform and libvirtd nodes.

To fix the issue, edit the file: /etc/apparmor.d/libvirt/TEMPLATE.qemu

By default it has the following content:

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
}

In order to allow libvirt to use the guest image files, change the content to (or add a similar line if your file path is different):

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  /data/vms/cluster_storage/**.qcow2 rwk,
}

The added line (line 9) will allow read (r), write (w) and lock (k) access to all qcow2 files in the directory /data/vms/cluster_storage.

Once added, all libvirt guests will start up again without any (AppArmor) issues.

First, install the ulogd2 package (example is based on Debian/ Ubuntu):

apt install ulogd2

Example: log and drop packets which have an invalid state

# Log and drop all invalid packets                                                                                                                                                                                         
iptables -A INPUT -m conntrack --ctstate INVALID -j NFLOG --nflog-group 123 --nflog-prefix "packet with invalid state"
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

To log all those packets to a file in JSON format, ulogd2 requires the following configuration at /etc/ulogd.conf

[global]                                                                                                                                                                                                           
logfile="syslog"
loglevel=3                                                                                                                                                                                                                   
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_inppkt_NFLOG.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IFINDEX.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IP2STR.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IP2BIN.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_PRINTPKT.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_HWHDR.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_raw2packet_BASE.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_JSON.so"

stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,mac2str1:HWHDR,json1:JSON

[log1]
group=123

[json1]
sync=1
file="/var/log/ulog/netfilter_log.json"

After creating the configuration file, ensure that ulogd2 is restarted and that the directory /var/log/ulog exists

mkdir /var/log/ulog
chown ulog /var/log/ulog
systemctl restart ulogd2.service

Once the above created rule matches, a JSON log line will be written to disk:

tail -1 /var/log/ulog/netfilter_log.json | jq
{
  "timestamp": "2022-03-30T14:46:20.527282+0200",
  "dvc": "Netfilter",
  "raw.pktlen": 52,
  "raw.pktcount": 1,
  "oob.prefix": "packet with invalid state",
  "oob.time.sec": 1648644380,
  "oob.time.usec": 527282,
  "oob.mark": 0,
  "oob.ifindex_in": 2,
  "oob.hook": 1,
  "raw.mac_len": 14,
  "oob.family": 2,
  "oob.protocol": 2048,
  "raw.label": 0,
  "raw.type": 1,
  "raw.mac.addrlen": 6,
  "ip.protocol": 6,
  "ip.tos": 0,
  "ip.ttl": 116,
  "ip.totlen": 52,
  "ip.ihl": 5,
  "ip.csum": 41779,
  "ip.id": 16049,
  "ip.fragoff": 16384,
  "src_port": 58662,
  "dest_port": 445,
  "tcp.seq": 3872158206,
  "tcp.ackseq": 0,
  "tcp.window": 8192,
  "tcp.offset": 0,
  "tcp.reserved": 0,
  "tcp.urg": 0,
  "tcp.ack": 0,
  "tcp.psh": 0,
  "tcp.rst": 0,
  "tcp.syn": 1,
  "tcp.fin": 0,
  "tcp.res1": 0,
  "tcp.res2": 0,
  "tcp.csum": 60039,
  "oob.in": "eth0",
  "oob.out": "",
  "src_ip": "181.122.165.177",
  "dest_ip": "1.1.1.1",
  "mac.saddr.str": "94:f7:ad:4f:81:fc",
  "mac.daddr.str": "aa:aa:aa:aa:aa:aa",
  "mac.str": "aa:aa:aa:aa:aa:aa:94:f7:ad:4f:81:fc:08:00"
}

First, let’s set up the PF table which will control the traffic. Add the following to your /etc/pf.conf :

# add time block table
table <time_block> { } persist

Next, create a PF rule which block traffic for all entries in the time_block table:

# block all CIDR addresses in the time block table
block in quick log from <time_block> to any

Since the time_block table is still empty, no traffic is actually blocked.

The last thing to implement, is periodically manipulating the time_block table. This could be done by creating two cronjobs:

1. allow traffic at the beginning of “business hours”
2. block traffic at the end of “business hours”

crontab -e
# Allow traffic
0 7 * * * /usr/local/scripts/allow_employees.sh > /dev/null 2>&1
# Block traffic
0 17 * * * /usr/local/scripts/block_employees.sh > /dev/null 2>&1


The allow_employees.sh script will allow certain network ranges by ensuring those are removed from the time_block table:

#!/bin/sh

/sbin/pfctl -Td -t time_block 10.1.0.0/24
/sbin/pfctl -Td -t time_block 10.2.0.0/24

The block_employees.sh script will do the exact opposite: it will add ranges to the time_block table so that their network access will be blocked by the firewall:

#!/bin/sh

/sbin/pfctl -Ta -t time_block 10.1.0.0/24
/sbin/pfctl -Ta -t time_block 10.2.0.0/24

Finally deploy your new PF rules (first test them!)

pfctl -nf /etc/pf.conf
pfctl -f /etc/pf.conf

The zone files provided by https://ipdeny.com/ need to be stored locally. A simple way to achieve this is by having a cronjob downloading those periodically (for instance once per day):

#!/bin/sh

# download the latest country zone files
curl -s https://www.ipdeny.com/ipblocks/data/countries/ru.zone > /etc/ru.zone
curl -s https://www.ipdeny.com/ipblocks/data/countries/cn.zone > /etc/cn.zone


We store them directly to /etc in the above example.

In the /etc/pf.conf, first add a table based on the files you have generated with the above statements:

# add a bad hosts table based on local disk text files
# one CIDR per line
table <badhosts> persist file "/etc/ru.zone" file "/etc/cn.zone"

In the above example, we have created a table called badhosts based on two local files.

Finally we need some rules which actually blocks from and to these network ranges, an example PF block rule could be:

# block bad IP addresses
block from <badhosts> to any
block from any to <badhosts>

One simple (not full bullet-proof) way of doing this, is by setting up block rules on firewall level, which can be achieved on Linux servers with iptables and zone files of https://www.ipdeny.com/. These zone files contain the network ranges assigned to a specific country.

The network ranges are fed to a tool called ipset, which sets up of hash map that can be easily used within iptables rules.

On Debian/Ubuntu systems, ipset can be installed with the apt command:

apt install ipset

Next, create an iptables chain called “blocked_countries“, to which we will add rules afterwards. Add this chain to the beginning of the INPUT and FORWARD chain, to have early blocking in your ruleset.

iptables -N blocked_countries
iptables -I INPUT -j blocked_countries -m comment --comment "Blocked countries"
iptables -I FORWARD -j blocked_countries -m comment --comment "Blocked countries"

Finally, create a shell script which will download the required zone files from https://ipdeny.com/ and which feeds them to ipset. The example script will try to block the countries China, Russia and Belarus:

#!/bin/bash

COUNTRIES=('cn' 'ru' 'by')

for COUNTRY in "${COUNTRIES[@]}"; do
    ipset create "countries_${COUNTRY}" hash:net
done

iptables -v -F blocked_countries

for i in "${COUNTRIES[@]}"; do
    echo "Ban IP of country ${i}"
    ipset flush "countries_${i}"


    for IP in $(wget -O - https://www.ipdeny.com/ipblocks/data/countries/${i}.zone)
    do
        ipset add "countries_${i}" $IP
    done
    iptables -I blocked_countries   -m set --match-set "countries_${i}" src  -j LOGDROP -m comment   --comment "Block .${i}"
done


You can check the blocked_countries chain if packets are being blocked by your new rules:

iptables -v -n -L blocked_countries 
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain blocked_countries (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   104 LOGDROP    all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set countries_by src /* Block .by */
 2182  155K LOGDROP    all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set countries_ru src /* Block .ru */
  344 21370 LOGDROP    all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set countries_cn src /* Block .cn */


• USB / mSSD drive to install OpenBSD on
• USB drive to boot from
• Serial cable for the initial installation

First go to a mirror and download the amd64 OpenBSD install image, which ends on the .fs extension. The one for OpenBSD 6.0 is called ‘install60.fs’.

The installation procedure will use an USB stick to boot the installation process.
The USB stick itself was created on a Linux Desktop system, using dd. There are many ways to get the install image on an USB stick. Google if you are not using a Linux desktop.

I downloaded the install image to my ‘Downloads’ folder, and the USB stick was on /dev/sdb

dd if=./Downloads/install60.fs of=/dev/sdb bs=1024k

Once the USB stick/ disk is ready, insert it in your Soekris net6501.

Now prepare your terminal console. I’ve used Minicom for this purpose, but any kind of terminal emulation program can be used.

Execute the following on the command line:

minicom -D /dev/ttyUSB0 -b 19200 # no software or hardware flow control

Important to note is:
– The baudrate must be set to 19200 baud
– You must disable software and/ or hardware flow control

Once connected with the serial console cable, plug in the power cable to your Soekris so that it can be boot up.

At the boot prompt, configure the installation image to send its output to the serial console.

Using drive 1, partition 3.
Loading.........
probing: pc0 com0 pci mem[620K 1022M a20=on]
disk: hd0+* hd1+ hd2*
>> OpenBSD/i386 BOOT 3.29
boot> stty com0 19200
boot> set tty com0
switching console to com0
>> OpenBSD/i386 BOOT 3.29
boot> bsd 

There are two commands that must be executed at the boot prompt:
– stty com0 19200
– set tty com0

After the serial console options have been set, boot the kernel by typing bsd and hiting ENTER

The first steps are to configure the network interfaces.
Please also make sure you have enabled sshd, so that after the installation you can access the firewall without using the serial console cable.

The rest of the installation won’t be covered here.

It runs on most operating systems, including Linux, MacOS, Solaris, HP-UX, AIX and OpenBSD.

There is no OSSEC package available on the OSSEC website or in the OpenBSDs ports repositry, so I’ve decided to create an OpenBSD on my own.
OpenBSD packages are pretty easy to create and are very useful when installing, upgrading or deleting software on a server.

One of the disadvantages when creating an OpenBSD package, is that you will need to have X11 installed on your OpenBSD system.
In the following example I have used OpenBSD 5.8 to create a package for OSSEC 2.8.2 (OSSEC 2.8.3 doesn’t compile on OpenBSD 5.8)

Step 1: Prerequisites

cd /tmp
wget http://ftp.eu.openbsd.org/pub/OpenBSD/5.8/amd64/xbase58.tgz 
wget http://ftp.eu.openbsd.org/pub/OpenBSD/5.8/amd64/xshare58.tgz
wget http://ftp.eu.openbsd.org/pub/OpenBSD/5.8/amd64/comp58.tgz
tar -C / -xzvphf xbase58.tgz
tar -C / -xzvphf xshare58.tgz
tar -C / -xzvphf comp58.tgz

cd /tmp
ftp http://ftp.eu.openbsd.org/pub/OpenBSD/$(uname -r)/ports.tar.gz
ftp http://ftp.eu.openbsd.org/pub/OpenBSD/$(uname -r)/SHA256.sig
signify -Cp /etc/signify/openbsd-$(uname -r | cut -c 1,3)-base.pub -x SHA256.sig ports.tar.gz

cd /usr
tar xzf /tmp/ports.tar.gz

You will also need a compiler:

pkg_add gcc

Step 2: Download and repack the source

ossec_version="2.8.2"
cd /usr/src
wget https://github.com/ossec/ossec-hids/archive/${ossec_version}.tar.gz 
mv ${ossec_version}.tar.gz ossec-hids-${ossec_version}.tar.gz 
tar xfz ossec-hids-${ossec_version}.tar.gz
cd ossec-hids-${ossec_version}

Since the Makefile for OSSEC is in the src/ sub directory, we will create a proxy Makefile in /usr/src/ossec-hids-2.8.2

cd ossec-hids-2.8.2
vim Makefile

I have actually taken the original Makefile from src/ and narrowed it down to the following:

# Makefile
# http://www.ossec.net/hids/

none:
 @echo "Nothing selected ..."
 @echo "\"make all\" to compile everything."
 @echo "\"make server\" to build the server."
 @echo "\"make local\" to build the local."
 @echo "\"make agent\" to build the agent."
 @echo "\"make clean\" to clean anything built."

clean:
 cd src/ ; $(MAKE) clean

all:
 cd src/ ; $(MAKE) all
 
test:
 cd src/ ; $(MAKE) test

server:
 cd src/ ; $(MAKE) server

local:
 cd src/ ; $(MAKE) local

agent:
 cd src/ ; $(MAKE) agent

We will also edit the ‘ossec-clients.sh‘ script, because we will use this script as a start/stop script. We will have to set the path name in this script.

vim src/init/ossec-client.sh
# LOCAL=/var/ossec
# cd ${LOCAL}
# PWD=`pwd`
DIR=/var/ossec
cd ${DIR}

And that’s the only thing we will need to change in the sources, we can now repackage it.

cd ..
tar czf /usr/ports/distfiles/ossec-hids-2.8.2.tar.gz ossec-hids-2.8.2/

Step 3: Prepare the ports directory

The following steps explain how to set up a ports directory in /usr/ports for OSSEC, in order to build the package.
Custom made packages are built in /usr/ports/mystuff. In there, we will have to one sub directory for the package category (we will use security) and in there the package name, which in our case will be ossec-hids.

cd /usr/ports/mystuff
mkdir -p security/ossec-hids

The configuration file for building an OpenBSD package is a Makefile. There is a template file in /usr/ports/infrastructure/templates/Makefile.template which can be used.

cd security/ossec-hids
cp /usr/ports/infrastructure/templates/Makefile.template Makefile

This file of course needs editing. Not everything is required in this file, so I have narrowed the Makefile down to what I need it for:

# $OpenBSD: Makefile.template,v 1.68 2013/10/02 07:34:45 ajacoutot Exp $
# $FreeBSD/NetBSD: credit FreeBSD/NetBSD if thats where the port came from $
# Original from: credit the original author here
COMMENT =               OSSEC is an Open Source HIDS
DISTNAME =              ossec-hids-2.8.2
CATEGORIES =            security
HOMEPAGE =              http://www.ossec.net/
MAINTAINER =            Johnny Morano <jmorano@moretrix.com>;
MASTER_SITES =          https://github.com/ossec/ossec-hids/

PERMIT_PACKAGE_CDROM =  Yes
PERMIT_PACKAGE_FTP =    Yes
PERMIT_DISTFILES_FTP =  Yes

PKG_ARCH =              *
PREFIX = /var/ossec

do-install:
        mkdir -p ${PREFIX}/bin
        mkdir -p ${PREFIX}/logs
        mkdir -p ${PREFIX}/var/run
        mkdir -p ${PREFIX}/queue
        mkdir -p ${PREFIX}/active-response/bin
        mkdir -p ${PREFIX}/agentless
        mkdir -p ${PREFIX}/etc/orig/shared
        mkdir -p ${PREFIX}/doc
        ${INSTALL_SCRIPT} ${WRKSRC}/active-response/firewalls/pf.sh ${PREFIX}/active-response/bin/
        ${INSTALL_SCRIPT} ${WRKSRC}/src/agentlessd/scripts/* ${PREFIX}/agentless
        ${INSTALL_SCRIPT} ${WRKSRC}/src/os_execd/ossec-execd ${PREFIX}/bin/
        ${INSTALL_SCRIPT} ${WRKSRC}/src/logcollector/ossec-logcollector ${PREFIX}/bin/
        ${INSTALL_SCRIPT} ${WRKSRC}/src/client-agent/ossec-agentd ${PREFIX}/bin/
        ${INSTALL_SCRIPT} ${WRKSRC}/src/addagent/manage_agents ${PREFIX}/bin/
        ${INSTALL_SCRIPT} ${WRKSRC}/src/syscheckd/ossec-syscheckd ${PREFIX}/bin/
        ${INSTALL_SCRIPT} ${WRKSRC}/src/os_auth/agent-auth ${PREFIX}/bin/
        ${INSTALL_SCRIPT} ${WRKSRC}/src/init/ossec-client.sh ${PREFIX}/bin/
        ${INSTALL_SCRIPT} ${WRKSRC}/doc/*.txt ${PREFIX}/doc/
        ${INSTALL_SCRIPT} ${WRKSRC}/doc/README.config ${PREFIX}/doc/
        ${INSTALL_SCRIPT} ${WRKSRC}/etc/*.conf ${PREFIX}/etc/orig/
        ${INSTALL_SCRIPT} ${WRKSRC}/etc/*.xml ${PREFIX}/etc/orig/
        ${INSTALL_SCRIPT} ${WRKSRC}/src/rootcheck/db/* ${PREFIX}/etc/orig/shared/

.include <bsd.port.mk>;

The above Makefile will install OSSEC in /var/ossec and will only install the agent files. It does not install the server files.

Step 4: Test the settings

First we will make a checksum and then we will start a fake compile run, to see if everything compiles nicely.

make makesum 
===>  Checking files for ossec-hids-2.8.2
`/usr/ports/distfiles/ossec-hids-2.8.2.tar.gz' is up to date.

make fake
===>  Checking files for ossec-hids-2.8.2
`/usr/ports/distfiles/ossec-hids-2.8.2.tar.gz' is up to date.
>> (SHA256) ossec-hids-2.8.2.tar.gz: OK
===>  Extracting for ossec-hids-2.8.2
===>  Patching for ossec-hids-2.8.2
===>  Configuring for ossec-hids-2.8.2
===>  Building for ossec-hids-2.8.2
***snip***

If there were no errors, then we are ready to create the actual package.

Step 5: Create the OpenBSD package

mkdir pkg
echo "OSSEC is an Open Source HIDS" > pkg/DESCR
make plist
vim pkg/PLIST

Normally we do not need to edit the PLIST file, but I wanted to create an ossec user upon installation and chown the /var/ossec directory to that user.
So I have added the following lines to the top of pkg/PLIST:

@comment $OpenBSD$
@newgroup ossec:1002
@newuser ossec:1005:ossec:daemon:OSSEC User:/var/ossec:/bin/sh

And these to the bottom:

@exec-add mkdir -p /var/ossec
@exec-add chown -R ossec.ossec /var/ossec
@exec-add cp %D/bin/ossec-client.sh /etc/rc.d/ossec

Afterwards you will need to run:

make plist

Now we are ready to build the package:

make package
`/usr/ports/pobj/ossec-hids-2.8.2/fake-amd64/.fake_done' is up to date.
===>  Building package for ossec-hids-2.8.2
Create /usr/ports/packages/amd64/no-arch/ossec-hids-2.8.2.tgz
Link to /usr/ports/packages/amd64/all/ossec-hids-2.8.2.tgz
Link to /usr/ports/packages/amd64/ftp/ossec-hids-2.8.2.tgz
Link to /usr/ports/packages/amd64/cdrom/ossec-hids-2.8.2.tgz

That’s it! This package can now be installed with the pkg_add command.

pkg_add ./ossec-hids-2.8.2.tgz 
quirks-2.114 signed on 2015-08-09T11:57:52Z
UNSIGNED PACKAGE file:./ossec-hids-2.8.2.tgz: install anyway ? [y/N/a] y
ossec-hids-2.8.2: ok
UNSIGNED PACKAGES: ossec-hids-2.8.2

The below script is an example how to clean and archive e-mails. The script was written in Perl and tested on a Courier IMAP server. Remember the Perl devise: there are million ways to write Perl scripts.

The following modules were used:

use Net::IMAP::Simple::SSL;
use Email::Simple;
use Getopt::Long qw/:config bundling/;
use DateTime;
use YAML qw/LoadFile/;
use Log::Log4perl;
use Pod::Usage;
use Data::Dumper;

The configuration file is written in YAML and the logging of the script is handled by Log4Perl.

log4perl.logger.shihai= DEBUG, shihaiLogfile
log4perl.logger.hostlistdb= DEBUG, shihaiLogfile

log4perl.appender.shihaiLogfile          = Log::Log4perl::Appender::File
log4perl.appender.shihaiLogfile.filename = /var/tmp/shihai.log
log4perl.appender.shihaiLogfile.layout   = Log::Log4perl::Layout::PatternLayout
log4perl.appender.shihaiLogfile.layout.ConversionPattern = %d %p [%x][%r millis][%c][%F{1}:%L][%M] %m%n

The configuration file contains the login credentials and the threshold values:

imap:
    host: 'mail.example.com'
    user: 'bob@example.com'
    pass: 'myultrasecretpassword'

threshold:
    archive:
        years: 3
    delete:
        years: 8

The threshold values are in fact the same parameters that can be used in the DateTime method ‘subtract()’;

The main loop of the program will:

• Connect to the IMAP server
• Retrieve all IMAP folders and loop through them
• Loop through all messages in each mailbox

# Connect to the IMAP server
my $imap = connect_imap( $config->{imap} );

# get all mailboxes and loop through them
my @mailboxes = get_mailboxes($imap);
foreach my $mailbox_name (@mailboxes){

    # Skip all Archive boxes 
    next if $mailbox_name =~ /Archive/;

    # select the mailbox and get the number of messages
    my $mb = $imap->select($mailbox_name);
    unless(defined $mb){
        $logger->error("Mailbox [$mailbox_name] doesn't exist: ", $imap->errstr());
        next;
    }

    $logger->info("Scanning $mailbox_name");

    # loop through the messages
    foreach my $i (1 .. $mb){
        my ($from, $subject, $date, $year) = get_mail_header($imap, $i);

        if(defined $date){
            if($date < $delete_date ){
                delete_mail($imap, $i);
            }
            elsif($date < $archive_date){
                $logger->info("Archiving [$i][$from][$subject][$date]");
                my $archive_box = get_archive_box($imap, $mailbox_name);
                move_mail($imap, $i, $archive_box)
            }
        }
    }

    $imap->expunge_mailbox($mailbox_name);
}

$imap->quit;

Let’s go through the different subroutines called in the mainloop.

connect_imap()

sub connect_imap {
    my ($cfg) = @_;
    my $logger = Log::Log4perl->get_logger('shihai.archive_mail');

    my $imap = Net::IMAP::Simple::SSL->new($cfg->{host})
        or $logger->logdie("Unable to connect to IMAP server: $Net::IMAP::Simple::errstr");
    $logger->info("Connected to IMAP host $cfg->{host}");

    unless( $imap->login($cfg->{user}, $cfg->{pass}) ){
        $logger->logdie("Login failed: ", $imap->errstr);

    }
    $logger->info("Logged to IMAP host $cfg->{host} as user '$cfg->{user}'");

    return $imap;
}

The parameters expected in the ‘$cfg’ hash are:

• host
• user
• pass

get_mailboxes()

sub get_mailboxes {
    my ($imap) = @_;
    my @mailboxes = $imap->mailboxes;
    my $logger = Log::Log4perl->get_logger('shihai.archive_mail');

    return @mailboxes;
}

Ok, I admit, I shouldn’t have created an extra subroutine for it… but i was kind of in a flow!

get_mail_header()

sub get_mail_header {
    my ($imap, $i) = @_;
    my $logger = Log::Log4perl->get_logger('shihai.archive_mail');

    my $header = $imap->top($i);
    unless( $header ){
        $logger->error("No header found for message $i in ", $imap->current_box);
    }
    
    my $email = Email::Simple->new(join '', @{ $header });
    unless( $email ){
        $logger->error("No Email::Simple object, skipping...");
        return
    }

    my ($subject) = $email->header('Subject');
    my ($date)    = $email->header('Date');
    my ($from)    = $email->header('From');

    # $logger->debug("Got e-mail [$from] [$subject] [$date]");

    unless(defined $date){
        $logger->error("No date found: ", $email->header_obj->as_string);
        delete_mail($imap, $i);
        return;
    }
    
    my($junk, $day, $month, $year) = ( $date =~ m/(...,\s+)?([0-9]{1,2})\s+(...)\s+(\d{4})/ );

    my $date_obj;
    if(defined $year && defined $month && defined $day){
        $date_obj = DateTime->new(
            year  => $year,
            month => $months{$month},
            day   => $day,
        );
    }    

    return ($from, $subject, $date_obj, $year, $month, $day);
}

This subroutine takes the ‘$imap’ object and the message number as input parameters. It will then retrieve the mail header and convert it to an ‘Email::Simple’ object. I’ve chosen this module so I can easily extract header fields.
If no ‘Date:’ field was found in the e-mail, then the tool will just delete the email. I don’t like e-mails with wrong or missing headers (they’re usually spam anyway).

delete_mail()

sub delete_mail {
    my($imap, $i) = @_;
    my $logger = Log::Log4perl->get_logger('shihai.archive_mail');

    if( $imap->delete($i) ){
        $logger->info("Deleted message number $i from ", $imap->current_box);
    }
}

Pretty straightforward.

get_archive_box()

sub get_archive_box{
    my($imap, $mailbox_name) = @_;
    my $logger = Log::Log4perl->get_logger('shihai.archive_mail');

    my ($archive_box) = ($mailbox_name);
    $archive_box =~ s/INBOX/INBOX.Archive/;

    if( not grep /^$archive_box$/, @mailboxes) {
        create_mailbox($imap, $archive_box);
        subscribe($imap, $archive_box);
    }

    return $archive_box;       
}

This subroutine will assemble the archive mailbox name. It will then check if the mailbox already exists and otherwise create it and subscribe to it.

create_mailbox()

sub create_mailbox {
    my($imap, $mb) = @_;
    my $logger = Log::Log4perl->get_logger('shihai.archive_mail');

    $imap->create_mailbox($mb) or $logger->logdie("Mailbox creation '$mb' failed: ", $imap->errstr());
    $logger->info("Created mailbox $mb");
}

It will basically just create the mailbox and log about it.

subscribe()

sub subscribe {
    my($imap, $box) =@_;
    my $logger = Log::Log4perl->get_logger('shihai.archive_mail');

    $imap->folder_subscribe($box);
    $logger->info("Subscribed to mailbox $box");
}

move_mail()

sub move_mail {
    my($imap, $i, $new_box) = @_;
    my $logger = Log::Log4perl->get_logger('shihai.archive_mail');

    if( $imap->copy($i, $new_box) ){
        $logger->info("Copied message number [$i] from ", $imap->current_box ," to [$new_box]");
        delete_mail($imap, $i);
    }
}

Moving an e-mail consists of copying it first to the new mailbox and then afterwards removing it from the old mailbox.

And that’s basically it!