To prevent this issue, three steps are required:

• change the order kernel modules get loaded by initrd
• disable the mdadm auto-assembly of disks when initrd is loaded
• generate a new initrd image

The initrd configuration files can be found at /etc/initramfs-tools.

First define the order of module loading during boot (ensure that multipath is loaded before mdadm). Edit the file /etc/initramfs-tools/modules to match:

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax: module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
libfc
megaraid_sas
scsi_dh_alua
scsi_transport_fc
dm_multipath
dm_service_time
multipath
md_mod

This starts by loading the “fiber channel libs” (libfc) first, followed by the MegaRAID modules, SCSI over fiber channel, multipath and mdadm.

Now copy the mdadm initramfs hook to /etc to override the default one:

cp /usr/share/initramfs-tools/hooks/mdadm /etc/initramfs-tools/hooks/

Edit the hook file in /etc/initramfs-tools/hooks/mdadm and add an ‘exit 0‘ (see line 10):

#!/bin/sh
#
# Copyright © 2006-2008 Martin F. Krafft <madduck@debian.org>,
#
2012 Michael Tokarev <mjt@tls.msk.ru>
# based on the scripts in the initramfs-tools package.
# released under the terms of the Artistic Licence.
#
set -eu
exit 0
PREREQ="udev"
prereqs()
{
echo "$PREREQ"
}

Disable automatic startup of the mdadm service:

rm /lib/systemd/system/mdadm.service
systemctl daemon-reload
systemctl disable mdadm.service

Finally, update the initrd images for all kernel versions installed:

update-initramfs -u -k all

Reboot the server to see if modules are now loaded in the correct order, and multipath is loaded and started before mdadm.

The statistics part contains quite some interesting information for monitoring and alerting.

The below Perl code snippit will loop over a glob of socket files (for instance when you have multiple HAProxy configurations running as separate processes) and print out the values returned by the “show info” command.

use IO::Socket::UNIX;

foreach my $socket_file (glob("/run/haproxy/*.sock")){
    print "- Reading socket: $socket_file\n";
    my $client = IO::Socket::UNIX->new(
        Type => SOCK_STREAM(),
        Peer => $socket_file,
    );

    print "- show info\n";
    print $client "show info\n";
    my $header = <$client>;
    chomp($header);

    $header =~ s/^#\s+//;
    my @keys = split ',', $header;
    print "- header:$header\n";

    while (my $line = <$client>){
        next unless $line =~ /^.+/;

        chomp($line);
        my @values = split ',', $line;
        print " - Got $line\n";
        print "   $keys[$_]: ".($values[$_]//'')."\n" foreach 0..$#keys;
    }

    close $client;
}

Many graphical interfaces are available for managing user accounts in OpenLDAP like PHPLDAPAdmin (http://phpldapadmin.sourceforge.net/wiki/index.php/Main_Page) or LAM (https://www.ldap-account-manager.org/lamcms/).

When generating a bulk amount of accounts with automation or just managing user details with a simple script, allows much more flexibility and can be even quicker.

LDAP passwords can be stored or changed by using an LDIF file. This LDIF file needs 3 required lines:

1. The “dn” you are about to change
2. the “changetype” set to “modify“
3. A “replace” line containing the field you want to change (in our case, since we are changing the password, this will be “userPassword“)

Your LDAP password can be stored either in clear-text (which is not advisable) or by sending a SHA-hash. The SHA-hash must include the salt at the end and must be base64 encoded.

The code snippit below will call a subroutine called generate_password() which comes from a previous article (Secure Password Generator in Perl).

At the end of the script, it will print out the LDIF file content, which needs to be saved to change.ldif. As last, it will print the ldapmodify command to make the actual change. You will need to know the admin password for this. Alternatively, you could also make this change using your own dn for authentication.

use Digest::SHA;
use MIME::Base64;

my $random_password = generate_password(24);
my $random_salt     = generate_password(3);

my $ctx = Digest::SHA->new;
$ctx->add($random_password);
$ctx->add($random_salt);
my $hashedPasswd = encode_base64($ctx->digest . $random_salt, '');

print "password: $random_password\n";
print "salt: $random_salt\n";
print <<EOF;
# LDIF
dn: uid=user1,ou=users,dc=shihai-corp,dc=at
changetype: modify
replace: userPassword
userPassword: {SSHA}$hashedPasswd
EOF

print "\n";
print q{LDAP cmd: ldapmodify -H "ldap://ldap_server01" -Z -x -W -D "cn=ldapadmin,ou=admins,dc=shihai-corp,dc=at" -f change.ldif} . "\n\n"

The YAML configuration hierarchy could be defined as the following file structure:

• common.yaml: Default settings no matter which role or host. These can be overridden in all the below.
• my_environment01.yaml: Environment specific configuration (example: development, staging, production, amsterdam, az01, az04, …). These can be overridden in all the below.
• common/roles/some_server_role.yaml: A server role, or type definition, which contains role specific configuration parameters. The roles could implement an extra hierarchy as for instance:
  
  • debian::databases::postgres
  • debian::databases::postgres::timescale
  • debian::databases::postgres::timescale::prometheus
  • debian::loadbalancer::internal
  • debian::application::request_processor
    
    The hierarchy steps are divided by :: in the above example, and need to be inherited accordingly, each with their own YAML file.
    These can be overridden in all the below.
• my_environment01/roles/some_server_role.yaml: override role configuration parameters per environment.
  These can even be overridden on host level below.
• my_environment01/hosts/my_hostname01.yaml: set host specific configuration parameters. This file is actually always required and should contain at least the IP address of the node and the server role string.

Let’s take the following example: The host vmazdbprm01 has the role debian::databases::postgres::timescale::prometheus and is deployed in the environment in my_cool_location03. The configuration management should search for parameters in the following file locations (and first verify if the file path exists):

1. common.yaml
2. my_cool_location01.yaml
3. common/roles/debian.yaml
4. common/roles/debian::databases.yaml
5. common/roles/debian::databases::postgres.yaml
6. common/roles/debian::databases::postgres::timescale.yaml
7. common/roles/debian::databases::postgres::timescale::prometheus.yaml
8. my_cool_location01/roles/debian.yaml
9. my_cool_location01/roles/debian::databases.yaml
10. my_cool_location01/roles/debian::databases::postgres.yaml
11. my_cool_location01/roles/debian::databases::postgres::timescale.yaml
12. my_cool_location01/roles/debian::databases::postgres::timescale::prometheus.yaml
13. my_cool_location01/hosts/vmazdbprm01.yaml

This means that any code which wants to implement the above configuration management, needs to verify if the above 13 files exists from top to bottom, and if yes loads the YAML file accordingly.

Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services.

Implementing the above YAML hierarchy in Terraform, could be done as follows:

locals {
  host_cfg             = yamldecode(fileexists("cfgmgmt/${var.environment}/hosts/${var.node}.yaml") ? file("cfgmgmt/${var.environment}/hosts/${var.node}.yaml") : "{server_role: debian}")

  roles_list           = split("::", local.host_cfg.server_role)
  all_roles_list       = [ for index in range(length(local.roles_list)): join("::",slice(local.roles_list, 0, index + 1))  ]

  common_cfg           = yamldecode(fileexists("cfgmgmt/common.yaml") ? file("cfgmgmt/common.yaml") : "{}")
  common_role_cfg_list = [ for file in local.all_roles_list:
      yamldecode(fileexists("cfgmgmt/common/roles/${file}.yaml") ? file("cfgmgmt/common/roles/${file}.yaml") : "{}" )]
    
  env_cfg              = yamldecode(fileexists("cfgmgmt/${var.environment}.yaml") ? file("cfgmgmt/${var.environment}.yaml") : "{}")
  env_role_cfg_list    = [ for file in local.all_roles_list:
      yamldecode(fileexists("cfgmgmt/${var.environment}/roles/${file}.yaml") ? file("cfgmgmt/${var.environment}/roles/${file}.yaml") : "{}") ]
        
  common_role_cfg_map  = merge(local.common_role_cfg_list...)
  env_role_cfg_map     = merge(local.env_role_cfg_list...)

  cfg                  = merge(local.common_cfg, local.env_cfg, local.common_role_cfg_map, local.env_role_cfg_map, local.host_cfg)
}

Let’s have a look what actually happens in the above code.

All YAML files are stored in a Git/ Hiera repository, accessible in the sub-directory cfgmgmt.

The code declares “local” variables by issuing the resource “locals“, starting from line 1.

Line 2 will check if a file called cfgmgmt/${var.environment}/hosts/${var.node}.yaml exists and if true, loads the YAML content as an map into the local variable host_cfg. If the file doesn’t exists, a default YAML code will be loaded. In theory, each node/ host must have a file defined as it should have at least data configuration such as:

• unique node host name
• IP address
• server role
• (optionally) VLAN/ subnet configuration
• …

Line 4 splits the server role string, stored in local.host_cfg.server_role, into a list, to build the server role hierarchy further below.

Line 5 creates a list of top level server roles which need to be imported too. Example: if the server role was set to debian::databases::postgres::timescale::prometheus , the list all_roles_list will contain the following elements:

• debian
• debian::databases
• debian::databases::postgres
• debian::databases::postgres::timescale
• debian::databases::postgres::timescale::prometheus

Line 7 loads the YAML content of common.yaml, if it exists.

Line 8 loops over the all_roles_list elements, created on line 5, and will load the YAML content of the server roles (if the file exists) into a list element. The result is a list called common_role_cfg_list.

Line 11 loads the general environment configuration YAML content (if it exists) into the local variable env_cfg.

Line 12 will do the same thing as line 8, but for environment specific roles. (for instance: when certain server roles have environment specific configuration parameters).

Lines 15 and 16 will merge the elements (which in theory are maps of YAML data) of the server role lists into one big map, in the order of the list. This allows that keys can be overridden. The expansion ... notation is explained at https://www.terraform.io/language/expressions/function-calls#expanding-function-arguments.

Finally on line 18, a local variable called cfg will be created, which merges the values of:

• local.common_cfg
• local.env_cfg
• local.common_role_cfg_map
• local.env_role_cfg_map
• local.host_cfg

By providing the environment name and the host/ node name to the above code (as var.environment and var.node ), all required configuration parameters can be loaded per node in Terraform, but since we’ve used a Git repository, this information can be loaded in any kind of automation tool (required is of course that each automation implements the same kind of hierarchy code).

With Promtail (https://grafana.com/docs/loki/latest/clients/promtail/), the above created log files can be sent to Loki so that they can finally be displayed in Grafana.

The installation of both Loki and Grafana are not covered in this article. The installation of Promtail is documented at https://grafana.com/docs/loki/latest/clients/promtail/installation/.

Once Promtail is installed, create the following configuration file at /etc/promtail-local-config.yaml:

server:                                                                                                                                                                                                            
  http_listen_port: 9080                                                                                                                                                                                           
  grpc_listen_port: 0                                                                                                                                                                                              
                                                                                                                                                                                                                   
positions:                                                                                                                                                                                                         
  filename: /var/tmp/promtail_positions.yaml                                                                                                                                                                       
                                                                                                                                                                                                                   
clients:                                                                                                                                                                                                           
  - url: http://loki_server:3100/loki/api/v1/push       
                                                                                                                                                               
scrape_configs:
    - job_name: iptableslogsjson
      static_configs:
      - targets:
          - localhost
        labels:
          instance: myhostname01
          job: iptableslogsjson
          __path__: /var/log/ulog/*json
      pipeline_stages:
      - json:
          expressions:
            timestamp: timestamp
            prefix: '"oob.prefix"'
            src: src_ip
            dst: dest_ip
      - labels:
          timestamp:
          prefix:
          src:
          dst:

With the above configuration, Promtail will create 4 extra labels per log line:

• timestamp: Contains the logged timestamp
• prefix: the NFLOG prefix string
• src: the source IP address
• dst: the destination IP address

Once the logs are arriving in Loki, and Loki has been configured as a datasource in Grafana, graphs can be created using LogQL.

Example:

sum(rate({job="iptableslogsjson"} [$__interval])) by (prefix)

Create a list of the existing virtual network subnets:

data azurerm_subnet "subnets" {
  count = length(data.azurerm_virtual_network.my_vnet.subnets)

  name                 = data.azurerm_virtual_network.my_vnet.subnets[count.index]
  virtual_network_name = var.vnet_name
  resource_group_name  = var.resource_group
}

locals {
  subnets = tomap({
      for snet in data.azurerm_subnet.subnets: snet.name => snet.id
  })
}

In the above example, we first loop over all subnet names, returned by data.azurerm_virtual_network.my_vnet.subnets, to create a list of Azure virtual network subnet objects.

Afterwards we create a locals map called subnets, which contains mapping like “subnet name points to subnet ID”.

Finally, when creating Azure network interfaces with an IP configuration, you can easily lookup the correct subnet ID based on the subnet name (which you might have configured per virtual machine)

resource "azurerm_network_interface" "my_nic" {
...
  ip_configuration {
    ...
    subnet_id = lookup(local.subnets, "my_subnet_name")
  }
...

The zone files provided by https://ipdeny.com/ need to be stored locally. A simple way to achieve this is by having a cronjob downloading those periodically (for instance once per day):

#!/bin/sh

# download the latest country zone files
curl -s https://www.ipdeny.com/ipblocks/data/countries/ru.zone > /etc/ru.zone
curl -s https://www.ipdeny.com/ipblocks/data/countries/cn.zone > /etc/cn.zone


We store them directly to /etc in the above example.

In the /etc/pf.conf, first add a table based on the files you have generated with the above statements:

# add a bad hosts table based on local disk text files
# one CIDR per line
table <badhosts> persist file "/etc/ru.zone" file "/etc/cn.zone"

In the above example, we have created a table called badhosts based on two local files.

Finally we need some rules which actually blocks from and to these network ranges, an example PF block rule could be:

# block bad IP addresses
block from <badhosts> to any
block from any to <badhosts>

One simple (not full bullet-proof) way of doing this, is by setting up block rules on firewall level, which can be achieved on Linux servers with iptables and zone files of https://www.ipdeny.com/. These zone files contain the network ranges assigned to a specific country.

The network ranges are fed to a tool called ipset, which sets up of hash map that can be easily used within iptables rules.

On Debian/Ubuntu systems, ipset can be installed with the apt command:

apt install ipset

Next, create an iptables chain called “blocked_countries“, to which we will add rules afterwards. Add this chain to the beginning of the INPUT and FORWARD chain, to have early blocking in your ruleset.

iptables -N blocked_countries
iptables -I INPUT -j blocked_countries -m comment --comment "Blocked countries"
iptables -I FORWARD -j blocked_countries -m comment --comment "Blocked countries"

Finally, create a shell script which will download the required zone files from https://ipdeny.com/ and which feeds them to ipset. The example script will try to block the countries China, Russia and Belarus:

#!/bin/bash

COUNTRIES=('cn' 'ru' 'by')

for COUNTRY in "${COUNTRIES[@]}"; do
    ipset create "countries_${COUNTRY}" hash:net
done

iptables -v -F blocked_countries

for i in "${COUNTRIES[@]}"; do
    echo "Ban IP of country ${i}"
    ipset flush "countries_${i}"


    for IP in $(wget -O - https://www.ipdeny.com/ipblocks/data/countries/${i}.zone)
    do
        ipset add "countries_${i}" $IP
    done
    iptables -I blocked_countries   -m set --match-set "countries_${i}" src  -j LOGDROP -m comment   --comment "Block .${i}"
done


You can check the blocked_countries chain if packets are being blocked by your new rules:

iptables -v -n -L blocked_countries 
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain blocked_countries (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   104 LOGDROP    all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set countries_by src /* Block .by */
 2182  155K LOGDROP    all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set countries_ru src /* Block .ru */
  344 21370 LOGDROP    all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set countries_cn src /* Block .cn */


Maintaining a large number of servers cannot be done without proper programming skills. Each good system administrator must therefor make sure he knows how to automate his daily works.

Although many many programming languages exist, most persons will only write code in one. I happen to like Perl.

In this next blog post, I am going to show how to create a script which can be deployed on all the Linux servers you need to maintain and need to check for certain running services.

Of course, a tool as Nagios together with NRPE and a configured event-handler could also be used, but lately I was often in the situation that the ‘nrpe daemon’ crashed, Nagios was spewing a lot of errors and the event-handler… well, since nrpe was down, the event-handler of course couldn’t connect or do anything. So why rely on a remote triggered action, when a simple script could be used.

The following script will check a default list of services and can additionally load or overwrite these services. A regular expression can be used to check for running processes, and of course, a startup command needs to be defined. And that is all the script will and should do.

The script uses three CPAN modules:

• Proc::ProcessTable
• YAML
• File::Slurp

The first one will be used to get a full listing of all running processes and the second one will provide us a means for using configuration files.

So let’s start our script:

#!/usr/bin/env perl 
use strict; use warnings;
use utf8;

use Proc::ProcessTable;
use YAML qw/LoadFile/;
use File::Slurp;

# Default set of processes to watch
my %default_services = (
    'NRPE' => {
        'cmd'     => '/etc/init.d/nagios-nrpe-server restart',
        're'      => '/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d',
	'pidfile' => '/var/tmp/nagios-nrpe-server.pid',
    },
    'Freshclam' => {
        'cmd'     => '/etc/init.d/clamav-freshclam restart',
        're'      => '/usr/bin/freshclam -d --quiet',
	'pidfile' => '/var/tmp/clamav-freshclam.pid',
    },
    'Syslog-NG' => {
        'cmd'     => '/etc/init.d/syslog-ng restart',
        're'      => '/usr/sbin/syslog-ng -p /var/run/syslog-ng.pid',
	'pidfile' => '/var/run/syslog-ng.pid',     
    },
    'VMToolsD' => {
        'cmd'     => '/etc/init.d/vmware-tools restart',
        're'      => '/usr/sbin/vmtoolsd',
	'pidfile' => '/var/tmp/vmtoolsd.pid',
    },
    'Munin-Node' => {
        'cmd'     => '/etc/init.d/munin-node restart',
        're'      => '/usr/sbin/munin-node',
	'pidfile' => '/var/tmp/munin-node.pid',
    },
);

my (%services) = (%default_services);

Until now, no rocket science. We load the required modules, we defined our default services that need to be checked.

Next part, check if there is a configuration file on disk. The script looks for a hard-coded path ”/etc/default/watchdog.yaml”:

# Check if there is a local config file and if yes, load them in the services hash
if( -f '/etc/default/watchdog.yaml' ){
    my $local_config = LoadFile '/etc/default/watchdog.yaml';

    %services = (%default_services, %{ $local_config->{services} });
}

The last Perl statement actually allows to overwrite one or more (or even all) the default defined services.

Now let’s see if these processes are actually running. The following code was hugely inspired by Colin Keith’s comment below. I have combined his examples together with my code.

Let’s first have a look at the code:

# Get current process table
my $processes = Proc::ProcessTable->new;
my %procs; 
my %matched_procs;
foreach my $p (@{ $processes }){
    $procs{ $p->{pid} } = $p->{cmndline};
    foreach my $s (keys %services){
        if($p->{cmndline} =~ m#$services{$s}->{re}#){
            $matched_procs{$s}++;
            last;
        }
    }
}

# Search the process table for not running services
foreach my $service ( keys %services ) {
    if(exists($services{$service}->{pidfile}) && -f $services{$service}->{pidfile} ) {
        my $pid = read_file( glob($services{$service}->{pidfile}) );
 
        # If we get a pid ensure that it is running, and that we can signal it
        $pid && exists($procs{$pid}) && kill(0, $pid) && next;  
        
        # Remove the stale PID file because no running process for this PID file
        unlink( $services{$service}->{pidfile} );
    }
    else {
        # check if the configured process regex matches
        if( exists($matched_procs{$service}) ){
            # process is running but has no PID file
            next;
        }
    }
	
    # Execute the service command
    system( $services{$service}->{'cmd'} );

    # Check the exit code of the service command
    if ($? == -1) {
        print "Failed to restart '$service' with '$services{$service}->{cmd}': $!\n";
    }
    elsif ($? & 127) {
        printf "Restart of '$service' died with signal %d, %s coredump\n", ($? & 127),  ($? & 128) ? 'with':'without';
    }
    else {
        printf "Process '$service' successfully restarted, exit status:  %d\n", $? >> 8;
    }
}

Lines 2 retrieves the current process list. We will save that information in two hashes with a little less information, because we actually only need the PID and the actual ‘command line’ of each process.

At line 16 we will start looping through the processes we have defined in the %services hash.
Inspired by Colins post, we will check if the process’ PID file is still there and if one is configured. If it still exists, we will then verify if the PID stored in the PID file, exists in the process list, which we have stored in %procs. This happens in lines 18-21.
At line 21, if the process is still running and the PID matches, we will check the next service to check (&& next part)
If the process is not running anymore but the PID file was still in the defined path, then it will be removed at line 24.

Otherwise, if no PID file was found or no PID file was configured, we will check the process list with the regular expression defined for that process. We have already created a hash, %matched_procs between lines 7 and 10, which we will use for this checking. If the process exists in the hash, we will skip and check the next process to be checked.

Now, if there was no PID file or the PID file was removed at line 24, the process will be started again. This happens at line 35.
I’ve executed it with the ‘system’ function since I want to have the output of this command directly in STDOUT. And of course, the last thing to do is to check if the process started up correctly or not by checking its exit code.

Now save that script to for instance ‘watchdog.pl’ and configure it in a cron job.
Example:

*/5 * * * * root /usr/local/bin/watchdog.pl

And here’s an example of the configuration file:

services:
    Exim-Mailserver:
        cmd: /etc/init.d/exim4 restart
        re: /usr/sbin/exim4 -bd -q30m
    Ossec-Agent:
        cmd: /etc/init.d/ossec restart
        re: !!perl/regexp '(?:ossec-agentd|ossec-logcollector|ossec-syscheckd)'

Link to script source code: https://github.com/insani4c/perl_tools/tree/master/watchdog

The script itself is pretty straight-forward: it creates a server object using IO::Socket::INET and attaches that socket to an AnyEvent IO eventloop. Furthermore, upon every connection, it will call the subroutine fork_call (which is in the AnyEvent::Util module) to open up a client socket.

 
#!/usr/bin/env perl 
use strict;
use warnings;
use utf8;

use IO::Socket::INET;
use AnyEvent;
use AnyEvent::Util;
$AnyEvent::Util::MAX_FORKS = 15;

my $handled = 0;
$|++;

my $server = IO::Socket::INET->new(
    'Proto'     => 'tcp',
    'LocalAddr' => 'localhost',
    'LocalPort' => 1234,
    'Listen'    => SOMAXCONN,
    'Reuse'     => 1,
) or die "can't setup server: $!\n";
print "Listening on localhost:1234\n";

my $cv = AnyEvent->condvar;
my $w; $w = AnyEvent->io(
        fh   => \*{ $server }, 
        poll => 'r', 
        cb   => sub { 
                   $handled++;
                   $cv->begin; 
                   fork_call &handle_connections, 
                             $server->accept, 
                             sub { 
                               my ($client) = @_ ;
                               print " - Client $client closed\n"
                             } 
                    }
);
$cv->recv;

#
# Subroutines
# 
sub handle_connections {
    my ($client) =  @_;

    my $host = $client->peerhost;
    print "[Accepted connection from $host]\n";

    print $client "Hi, you're client #$handled\n";
    chomp ( my $input = <$client> );
    my $output = reverse $input;
    print $client $output, "\n";
    print $client "Bye, bye.\n";

    $cv->end;
    return $host;
}