Maintaining a large number of servers cannot be done without proper programming skills. Each good system administrator must therefor make sure he knows how to automate his daily works.

Although many many programming languages exist, most persons will only write code in one. I happen to like Perl.

In this next blog post, I am going to show how to create a script which can be deployed on all the Linux servers you need to maintain and need to check for certain running services.

Of course, a tool as Nagios together with NRPE and a configured event-handler could also be used, but lately I was often in the situation that the ‘nrpe daemon’ crashed, Nagios was spewing a lot of errors and the event-handler… well, since nrpe was down, the event-handler of course couldn’t connect or do anything. So why rely on a remote triggered action, when a simple script could be used.

The following script will check a default list of services and can additionally load or overwrite these services. A regular expression can be used to check for running processes, and of course, a startup command needs to be defined. And that is all the script will and should do.

The script uses three CPAN modules:

• Proc::ProcessTable
• YAML
• File::Slurp

The first one will be used to get a full listing of all running processes and the second one will provide us a means for using configuration files.

So let’s start our script:

#!/usr/bin/env perl 
use strict; use warnings;
use utf8;

use Proc::ProcessTable;
use YAML qw/LoadFile/;
use File::Slurp;

# Default set of processes to watch
my %default_services = (
    'NRPE' => {
        'cmd'     => '/etc/init.d/nagios-nrpe-server restart',
        're'      => '/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d',
	'pidfile' => '/var/tmp/nagios-nrpe-server.pid',
    },
    'Freshclam' => {
        'cmd'     => '/etc/init.d/clamav-freshclam restart',
        're'      => '/usr/bin/freshclam -d --quiet',
	'pidfile' => '/var/tmp/clamav-freshclam.pid',
    },
    'Syslog-NG' => {
        'cmd'     => '/etc/init.d/syslog-ng restart',
        're'      => '/usr/sbin/syslog-ng -p /var/run/syslog-ng.pid',
	'pidfile' => '/var/run/syslog-ng.pid',     
    },
    'VMToolsD' => {
        'cmd'     => '/etc/init.d/vmware-tools restart',
        're'      => '/usr/sbin/vmtoolsd',
	'pidfile' => '/var/tmp/vmtoolsd.pid',
    },
    'Munin-Node' => {
        'cmd'     => '/etc/init.d/munin-node restart',
        're'      => '/usr/sbin/munin-node',
	'pidfile' => '/var/tmp/munin-node.pid',
    },
);

my (%services) = (%default_services);

Until now, no rocket science. We load the required modules, we defined our default services that need to be checked.

Next part, check if there is a configuration file on disk. The script looks for a hard-coded path ”/etc/default/watchdog.yaml”:

# Check if there is a local config file and if yes, load them in the services hash
if( -f '/etc/default/watchdog.yaml' ){
    my $local_config = LoadFile '/etc/default/watchdog.yaml';

    %services = (%default_services, %{ $local_config->{services} });
}

The last Perl statement actually allows to overwrite one or more (or even all) the default defined services.

Now let’s see if these processes are actually running. The following code was hugely inspired by Colin Keith’s comment below. I have combined his examples together with my code.

Let’s first have a look at the code:

# Get current process table
my $processes = Proc::ProcessTable->new;
my %procs; 
my %matched_procs;
foreach my $p (@{ $processes }){
    $procs{ $p->{pid} } = $p->{cmndline};
    foreach my $s (keys %services){
        if($p->{cmndline} =~ m#$services{$s}->{re}#){
            $matched_procs{$s}++;
            last;
        }
    }
}

# Search the process table for not running services
foreach my $service ( keys %services ) {
    if(exists($services{$service}->{pidfile}) && -f $services{$service}->{pidfile} ) {
        my $pid = read_file( glob($services{$service}->{pidfile}) );
 
        # If we get a pid ensure that it is running, and that we can signal it
        $pid && exists($procs{$pid}) && kill(0, $pid) && next;  
        
        # Remove the stale PID file because no running process for this PID file
        unlink( $services{$service}->{pidfile} );
    }
    else {
        # check if the configured process regex matches
        if( exists($matched_procs{$service}) ){
            # process is running but has no PID file
            next;
        }
    }
	
    # Execute the service command
    system( $services{$service}->{'cmd'} );

    # Check the exit code of the service command
    if ($? == -1) {
        print "Failed to restart '$service' with '$services{$service}->{cmd}': $!\n";
    }
    elsif ($? & 127) {
        printf "Restart of '$service' died with signal %d, %s coredump\n", ($? & 127),  ($? & 128) ? 'with':'without';
    }
    else {
        printf "Process '$service' successfully restarted, exit status:  %d\n", $? >> 8;
    }
}

Lines 2 retrieves the current process list. We will save that information in two hashes with a little less information, because we actually only need the PID and the actual ‘command line’ of each process.

At line 16 we will start looping through the processes we have defined in the %services hash.
Inspired by Colins post, we will check if the process’ PID file is still there and if one is configured. If it still exists, we will then verify if the PID stored in the PID file, exists in the process list, which we have stored in %procs. This happens in lines 18-21.
At line 21, if the process is still running and the PID matches, we will check the next service to check (&& next part)
If the process is not running anymore but the PID file was still in the defined path, then it will be removed at line 24.

Otherwise, if no PID file was found or no PID file was configured, we will check the process list with the regular expression defined for that process. We have already created a hash, %matched_procs between lines 7 and 10, which we will use for this checking. If the process exists in the hash, we will skip and check the next process to be checked.

Now, if there was no PID file or the PID file was removed at line 24, the process will be started again. This happens at line 35.
I’ve executed it with the ‘system’ function since I want to have the output of this command directly in STDOUT. And of course, the last thing to do is to check if the process started up correctly or not by checking its exit code.

Now save that script to for instance ‘watchdog.pl’ and configure it in a cron job.
Example:

*/5 * * * * root /usr/local/bin/watchdog.pl

And here’s an example of the configuration file:

services:
    Exim-Mailserver:
        cmd: /etc/init.d/exim4 restart
        re: /usr/sbin/exim4 -bd -q30m
    Ossec-Agent:
        cmd: /etc/init.d/ossec restart
        re: !!perl/regexp '(?:ossec-agentd|ossec-logcollector|ossec-syscheckd)'

Link to script source code: https://github.com/insani4c/perl_tools/tree/master/watchdog

The script itself is pretty straight-forward: it creates a server object using IO::Socket::INET and attaches that socket to an AnyEvent IO eventloop. Furthermore, upon every connection, it will call the subroutine fork_call (which is in the AnyEvent::Util module) to open up a client socket.

 
#!/usr/bin/env perl 
use strict;
use warnings;
use utf8;

use IO::Socket::INET;
use AnyEvent;
use AnyEvent::Util;
$AnyEvent::Util::MAX_FORKS = 15;

my $handled = 0;
$|++;

my $server = IO::Socket::INET->new(
    'Proto'     => 'tcp',
    'LocalAddr' => 'localhost',
    'LocalPort' => 1234,
    'Listen'    => SOMAXCONN,
    'Reuse'     => 1,
) or die "can't setup server: $!\n";
print "Listening on localhost:1234\n";

my $cv = AnyEvent->condvar;
my $w; $w = AnyEvent->io(
        fh   => \*{ $server }, 
        poll => 'r', 
        cb   => sub { 
                   $handled++;
                   $cv->begin; 
                   fork_call &handle_connections, 
                             $server->accept, 
                             sub { 
                               my ($client) = @_ ;
                               print " - Client $client closed\n"
                             } 
                    }
);
$cv->recv;

#
# Subroutines
# 
sub handle_connections {
    my ($client) =  @_;

    my $host = $client->peerhost;
    print "[Accepted connection from $host]\n";

    print $client "Hi, you're client #$handled\n";
    chomp ( my $input = <$client> );
    my $output = reverse $input;
    print $client $output, "\n";
    print $client "Bye, bye.\n";

    $cv->end;
    return $host;
}

One thing you don’t have to do with those online services like Flickr etc. is generating smaller sizes of your photos, also called thumbnails. But if you maintain your own website, you will have to downsize them on your own.
Being a programmer, I’m the happiest when I’m the laziest. So, in case of websites and photos, I don’t want to create the zillion of thumbnails that I need on my page.
Zillion? Isn’t that a little bit exajurated? Well, mostly you will need more than one thumbnail for every picture you will upload:
– one thumbnail is needed for on the thumbnail page
– one smaller thumbnail is needed for a small thumbnail preview bar
– one even smaller thumbnail is needed for marking the photo on a Google Map
– one smaller sized photo is needed for previewing it on the website
– and finally the original image is also put available for download on the site

We’ll need at least 4 extra images for every photo. Okay, I could generate all those thumbs at home and then upload them all together, but that is wasting bandwidth and time and we know how precious they both are.

Being a programmer in my professional life, I have the deviation to automate everything I know or do, with a Perl script. And this situation is again perfect for such a script.

First of all, the following code will make heavy use of the Image::Magick Perl module, which can be downloaded from either CPAN or your distro’s repositry.

The several steps of manipulating the original image, were actually inspired by the following two links:
– http://www.imagemagick.org/Usage/thumbnails/#glass_bubble
– http://stackoverflow.com/questions/3973392/facebook-like-resizing-of-images-using-imagemagick

The second link explains how to resize and crop, the first link explains how create neat thumbnails using rounded corners, lighting effects and more. Now, both links give examples using the command line Image Magick tools. I’ve seen a lot of code which will call these command line tools instead of using the Image::Magick module. That is of course, NOT the way.

On with creating our thumbnails! First, the image needs to be resized:

my $img = 'image.png';
my $magick = new Image::Magick;
$magick->Read($img);
my ($width, $height) = $magick->Get('width', 'height');

$magick->Resize(geometry => int($width/2).'x'.int($height/2));
$magick->Resize(geometry => '180x');
$magick->Resize(geometry => 'x180<');
   
my($nwidth, $nheight) = $magick->Get('width', 'height');
my $xpos = int(( $nwidth - 120 ) / 2) - 60;
my $ypos = int(( $nheight - 120 ) / 2) - 60;
$magick->Crop(geometry => "120x120+$xpos+$ypos", gravity => 'Center');

The above code will generate a thumbnail which are 120px by 120px big (or small), using a cropped field out of the 180px by 180px scaled image. The position of the cropped field is somewhat in the middle of the image, where usually the object of the photo can be found.

Next we’ll add some effects to our downsized image. We’ll do this by cloning the resized Image Magick object, that was created in the code above:

my $thumb_mask = $magick->Clone();

Once the object has been cloned, we can start by adding effects to the new object:

$thumb_mask->Set('alpha' => 'Off');
$thumb_mask->Colorize( fill => 'white', opacity => '100%' );

$thumb_mask->Draw( fill => 'black',
                   primitive => 'polygon',
                   points => '0,0 0,15 15,0');
$thumb_mask->Draw( fill => 'white',
                       primitive => 'circle',
                       points => '15,15 15,0');
my $new_2 = $thumb_mask->Clone();
$new_2->Flip();
$thumb_mask->Composite( compose => 'Multiply', 'image' => $new_2 );

my $new_3 = $thumb_mask->Clone();
$new_3->Flop();
$thumb_mask->Composite( compose => 'Multiply', 'image' => $new_3 );

$thumb_mask->Set('background' => 'Gray50');
$thumb_mask->Set('alpha' => 'Shape');

$thumb_mask->Raise(raise => 'True', geometry => '4x4');

By now we should have a rounded gray icon, based on the size of the thumbnail we’ve created above.

Next we will create some lighting and shading effects so that the thumbnail will look more like a button.

my $thumb_lighting = $thumb_mask->Clone();
$thumb_lighting->Set('bordercolor' => 'None');
$thumb_lighting->Set('border'      => '1x1');
$thumb_lighting->Set('alpha'       => 'Extract');
$thumb_lighting->Blur('geometry'       => '0x10');
$thumb_lighting->Shade(geometry => '80x40');
$thumb_lighting->Set('alpha'       => 'On');
$thumb_lighting->Set('background'  => 'Gray50');
$thumb_lighting->Set('alpha'       => 'Background');
$thumb_lighting->AutoLevel(channel => 'alpha');
$thumb_lighting->[-1]->Function(function => 'polynomial', parameters => [3.5, -5.05, 2.05, 0.3]);

my $new_4 = $thumb_lighting->Clone();
$new_4->Set(alpha => 'Extract');
$new_4->Blur(geometry => '0x2');

$thumb_lighting->Composite(compose => 'Multiply', image => $new_4);
$thumb_lighting->Chop(geometry => '1x1');

Finally, all that rests is putting all the images togehter and saving the file to disk:

$magick->Set('alpha' => 'On');
$magick->Composite(compose => 'HardLight', image => $thumb_lighting);
$magick->Set('alpha' => 'Copy');
$magick->Composite(compose => 'CopyOpacity', image => $thumb_mask);
$magick->Write('thumbnail.png');

Important to know is, since creating rounded corner will add some transparency to the original image, the newly created thumbnail must be save as either a GIF or PNG image, while those are the only two image formats which support transparency in images.

Of course, most people don’t want to wait on a group of other people to update a piece of software.

I wrote this script to make my life a little bit easier, but it contains no rocket science. I’ve written it in Perl but it could have been just as easily written in Bash. But Perl is much cooler than Bash, and being cool is all that matters!

The script actually uses some command line tools, that should be installed. apt-cache and egrep are installed by default on Debian(-based) systems, dh-make-perl must be installed separately.

apt-get install dh-make-perl

Once dh-make-perl is installed, check out the man page for its options and arguments.

What are the steps required in the script:

• get a list of Perl packages
• filter out packages installed by dh-make-perl
• check if we really need to upgrade
• create new packages out of the CPAN versions
• optionally display the required packages

Packages created by dh-make-perl can be filtered out based on the package description. dh-make-perl adds ‘…automagically extracted from the module by dh-make-perl‘ to the package description when creating a Debian package.

It is possible that the package building of the CPAN module fails because the required modules or packages are missing. This information will be displayed when the scripts finishes. You will have to install this missing requirements and relaunch the script afterwards.

When build the Debian with dh-make-perl, the option ‘–recursive‘ will be used. This will also create Debian packages of required modules defined by the CPAN module.

When all these packages have been created, installed them with the dpkg command:

# dpkg -i $(ls -rt *.dpkg)

I’ve used the $(ls -rt *.dpkg) syntax because the packages will be in the order like they have been built.

The script:

#!/usr/bin/perl
# $Author: insaniac $
# $Id: upgrade_cpan_pkgs.pl 24 2011-03-10 13:54:32Z insaniac $
use strict; use warnings;
use Cwd;
use CPAN;

my $cwd = getcwd;
my ($script) = ($0 =~ /([^/]+)$/);
open my $LOG, '>', "$cwd/${script}_log" or die "KAK for [$cwd/$0_log]: $!n";

my %needed_pkgs;
foreach my $pkg (`apt-cache pkgnames | egrep '\-perl$'`){
    chomp($pkg);
    my $info = `apt-cache show $pkg`;
    if($info =~ m/automagically extracted from the module by dh-make-perl/ms){
        my ($cpan_pkg) = ( $pkg =~ m/^lib(.*?)-perl$/);
        $cpan_pkg =~ s/[_-]/::/g;
        my ($installed_version) = ( $info =~ m/Version:s+(.*?)-[0-9.]+$/m );
        my ($cpan_version) = CPAN::Shell->expand('Module', "/$cpan_pkg/");
        
        next unless ref $cpan_version;
        $cpan_version = $cpan_version->cpan_version;

        if($installed_version eq $cpan_version){
            print "- Package $cpan_pkg does not require updating (current version: $installed_version| CPAN version: $cpan_version)n";
            next;
        }


        print "- Building $cpan_pkg (outdated! installed version: $installed_version|CPAN version: $cpan_version)n";
        print $LOG "- Building $cpan_pkg (outdated! installed version: $installed_version|CPAN version: $cpan_version)n";
        print $LOG "="x80, "n";
        
        my $pkg_upgrade_info = `dh-make-perl --recursive --build --cpan $cpan_pkg`;
        print $LOG $pkg_upgrade_info;

        my ($need_deb_pkgs) = ($pkg_upgrade_info =~ m/Needs the following debian packages:s+(.*)$/ms);
        my ($need_build_deb_pkgs) = ($pkg_upgrade_info =~ m/Needs the following debian packages during building:s+(.*)$/ms);
        print $LOG "need_deb_pkgs: $need_deb_pkgsn"                if $need_deb_pkgs;
        print $LOG "need_build_deb_pkgs: $need_build_deb_pkgsn"    if $need_build_deb_pkgs;

        map { defined $_ && $needed_pkgs{$_}++ } split( /s*,s*/, $need_deb_pkgs )       if defined $need_deb_pkgs;
        map { defined $_ && $needed_pkgs{$_}++ } split( /s*,s*/, $need_build_deb_pkgs ) if defined $need_build_deb_pkgs;

        print $LOG "="x80, "n";
    }
}

print "Needed packages:n";
print "- $_: $needed_pkgs{$_} timesn" foreach keys %needed_pkgs;
print "n";
close $LOG;

Some output produced by the script:

$ perl ~/cheese_code/trunk/upgrade_cpan_pkgs.pl 2>&1|tee upgrade.log
CPAN: Storable loaded ok (v2.20)
Going to read '/home/insaniac/.cpan/Metadata'
  Database was generated on Sat, 05 Mar 2011 22:27:24 GMT
- Package geo::ip::pureperl does not require updating (current version: 1.25| CPAN version: 1.25)
- Package html::fromansi does not require updating (current version: 2.03| CPAN version: 2.03)
- Package net::google::calendar does not require updating (current version: 1.0| CPAN version: 1.0)
- Building poe::component::server::httpserver (outdated! installed version: 0.9.2|CPAN version: 0.009002)
Using cached Contents from Thu Mar 10 14:38:42 2011
**********
WARNING: a package named
              'libpoe-component-server-httpserver-perl'
         is already available in APT repositories
Maintainer: Johnny Morano 
Description: serve HTTP requests
- Package poe::filter::log::iptables does not require updating (current version: 0.02| CPAN version: 0.02)
- Package uri::imaps does not require updating (current version: 1.01| CPAN version: 1.01)
- Building class::accessor::chained (outdated! installed version: 0.01.1~debian|CPAN version: 0.01)
Using cached Contents from Thu Mar 10 14:38:42 2011
**********
WARNING: a package named
              'libclass-accessor-chained-perl'
         is already available in APT repositories
Maintainer: James Bromberger 
Description: make chained accessors
- Package www::google::contacts does not require updating (current version: 0.28| CPAN version: 0.28)