dark

Recursive Inotify Daemon

There aren’t many Inotify daemons available that can work recursive and offer a descent flexibility regarding Inotify signals. For basic rsync operations, lsyncd isn’t that bad but it isn’t that flexible for system administration. The fact that it is written and configured in Lua, makes it actually really complex (considering that you don’t know to develop in Lua). Most system administrators do know or at least understand Perl.

Linux::Inotify2 is a CPAN module which allows to catch Linux Inotify signals (duh) and react upon them, in realtime. Combined with the AnyEvent CPAN module, you can use a plethora of event loop libraries available in your system, to have an amazingly fast inotify daemon.

The first thing we’ll discuss, is how to turn a regular Perl script into a standalone daemon process.

sub daemonize {
# Inspired by http://stackoverflow.com/questions/766397/how-can-i-run-a-perl-script-as-a-system-daemon-in-linux
    POSIX::setsid or die "setsid: $!";
    my $pid = fork ();
    if ($pid < 0) {
        die "fork: $!";
    } 
    elsif ($pid) {
        exit 0;
    }

    chdir "/";
    umask 0;
    foreach (0 .. (POSIX::sysconf (&POSIX::_SC_OPEN_MAX) || 1024)) { 
        POSIX::close $_ 
    }

    open (STDIN, "/dev/null");
    open (STDOUT, ">>/tmp/log.txt");
    open (STDERR, ">&STDOUT");

    # Save PID to disk
    open my $pid_file, '>', '/var/run/script.pid'
        or die "Could not open PID file: $!\n";
    print { $pid_file } "$$";
    close ($pid_file);
} 

The process will get forked and the default IO handlers will be redirected. A PID file will also be created, so that the process can killed easily afterwards.
For example:

kill $(cat /var/run/script.pid)

Next we will create a inotify signal handler and search for directories to watch for changes. In our example, we will watch every subdirectory of the ‘/data/vault’ directory (the parent directory itself is not included in this example):

# Create Inotify object
my $inotify = Linux::Inotify2->new()
    or die "Failed to created inotify object: $!\n";

# Search for directories to watch
find({ wanted => sub { -d $_ 
                       && create_watcher($inotify, $File::Find::name) }  
     }
    , '/data/vault');

The ‘find’ subroutine comes from the File::Find module, which is a standard Perl module.
For every directory found, the ‘create_watcher’ subroutine will be called, with the full path of the found directory.

sub create_watcher {
    my ($inotify, $dir) = @_;
    my $watcher = $inotify->watch($dir, IN_CREATE | IN_CLOSE_WRITE | IN_MOVE | IN_DELETE, sub {
            my $e = shift;
            my $filename  = $e->fullname;
            
            if(-d $filename && $e->IN_CREATE) {
                create_watcher($inotify, $filename);
                return
            }
            elsif(-f $filename){
                if($e->IN_CLOSE_WRITE){
                    print "IN_CLOSE_WRITE $filename\n"
                }
                elsif($e->IN_MOVED_FROM){
                    print "IN_MOVE_FROM $filename\n"
                }
                elsif($e->IN_MOVED_TO){
                    print "IN_MOVED_TO $filename\n"
                }
                elsif($e->IN_DELETE){
                    print "IN_DELETE $filename\n"
                }
            }
    });
    print "Watching $dir\n";
    $W{$dir} = $watcher;
}

The watcher code will watch for the following inotify signals:

  • IN_CREATE
  • IN_CLOSE_WRITE
  • IN_MOVE
  • IN_DELETE

If a new directory is created within ‘/data/vault’, the ‘create_watcher’ subroutine will be called again for this new directory. For files, the code will check what inotify signal was spawned and act accordingly.

The last to do is to initialize the event loop, using AnyEvent:

my $cv = AnyEvent->condvar;
# Create event loop poller
my $poller = AnyEvent->io(
        fh   => $inotify->fileno,
        poll => 'r',
        cb   => sub { $inotify->poll }
);

# Receive event signals (inotify signals)
$cv->recv;

The above example shows how the inotify object ($inotify) is used in the event listener.

These examples explain how to create a simple inotify daemon, which can recursively go through directories and then watch these directories for certain inotify signals.

The full script:

#!/usr/bin/env perl 
use strict;
use warnings;
use utf8;

use AnyEvent;
use Linux::Inotify2;
use File::Find;
use POSIX;

my $PID_FILE = "/var/run/$0.pid";

# Fork this process, to run as a daemon
daemonize();

# enable autoflush to have faster logging
$|++;

# Catch kill signals
local $SIG{TERM} = sub {
    if(-f $PID_FILE){
        unlink($PID_FILE)
    }

    print("$0 daemon killed.");
    exit 0;
};
local $SIG{INT} = $SIG{TERM};

my $cv = AnyEvent->condvar;
# watcher container hash
my %W;

# Create Inotify object
my $inotify = Linux::Inotify2->new()
    or die "Failed to created inotify object: $!\n";

# Search for directories to watch
find({ wanted => sub { -d $_ 
                       && create_watcher($inotify, $File::Find::name) }  
     }
    , '/data/vault');


# Create event loop poller
my $poller = AnyEvent->io(
        fh   => $inotify->fileno,
        poll => 'r',
        cb   => sub { $inotify->poll }
);

# Receive event signals (inotify signals)
$cv->recv;

#
# Subroutines
#
sub create_watcher {
    my ($inotify, $dir) = @_;
    my $watcher = $inotify->watch($dir, IN_CREATE | IN_CLOSE_WRITE | IN_MOVE | IN_DELETE, sub {
            my $e = shift;
            my $filename  = $e->fullname;
            
            if(-d $filename && $e->IN_CREATE) {
                create_watcher($inotify, $filename);
                return
            }
            elsif(-f $filename){
                if($e->IN_CLOSE_WRITE){
                    print "IN_CLOSE_WRITE $filename\n"
                }
                elsif($e->IN_MOVED_FROM){
                    print "IN_MOVE_FROM $filename\n"
                }
                elsif($e->IN_MOVED_TO){
                    print "IN_MOVED_TO $filename\n"
                }
                elsif($e->IN_DELETE){
                    print "IN_DELETE $filename\n"
                }
            }
    });
    print "Watching $dir\n";
    $W{$dir} = $watcher;
}

sub daemonize {
# Inspired by http://stackoverflow.com/questions/766397/how-can-i-run-a-perl-script-as-a-system-daemon-in-linux
    POSIX::setsid or die "setsid: $!";
    my $pid = fork ();
    if ($pid < 0) {
        die "fork: $!";
    } 
    elsif ($pid) {
        exit 0;
    }

    chdir "/";
    umask 0;
    foreach (0 .. (POSIX::sysconf (&POSIX::_SC_OPEN_MAX) || 1024)) { 
        POSIX::close $_ 
    }

    open (STDIN, "/dev/null");
    open (STDOUT, ">>/tmp/log.txt");
    open (STDERR, ">&STDOUT");

    # Save PID to disk
    open my $pid_file, '>', $PID_FILE
        or die "Could not open PID file: $!\n";
    print { $pid_file } "$$";
    close ($pid_file);
} 
14 comments
  1. Thanks for this nice idea!

    However, I’m having some problems. For some reason, while the script works as expected when monitoring the initial directory_tree and seems to be adding newly created subdirectories to its watch list, when new files are created inside those aforementioned directories, it does not notify correctly.

    e.g.
    /data/vault
    touch /data/vault/testfile (script notifies correctly)
    mkdir -p /data/vault/testdir (script notifies and adds testdir to its watch list, or at least that’s what it says)
    touch /data/vault/testdir/testfile (script FAILS to notify correctly)

    Any ideas?

    Cheers, Nick

    1. Hi Nick.

      That is strange, and I’m unable to reproduce your problem. On my PC, the script gives the following output:

      Watching /data/vault
      IN_CLOSE_WRITE /data/vault/log.txt
      Watching /home/jmorano/vault/test_dir
      IN_CLOSE_WRITE /data/vault/test_dir/test_file_in_test_dir
      IN_CLOSE_WRITE /data/vault/test_dir/touch_test
      

      The first two IN_CLOSE_WRITE statements were created using ‘echo “bla” > file’ and the last one was a ‘touch’.

      What was the output of the script (in /tmp/log.txt) on your side?

      Cheers,
      Johnny

      1. 1. touch /data/vault/testfile
        2. mkdir -p /data/vault/testdir
        3. touch /data/vault/testdir/testfile

        IN_CLOSE_WRITE /data/vault/testfile
        Watching /data/vault/testdir

        Third command does not produce any output

        Is there a limit to the number of subdirectories inotify can watch? Cause mine is huge. Maybe that’s the problem.

        1. OK I’ve changed /proc/sys/fs/inotify/max_user_watches
          and everything works as expected. My bad!

          Cheers

          1. Cool! It must be really huge then, cause overhere it is set by default to 65536:

            # cat /proc/sys/fs/inotify/max_user_watches
            65536

            Cheers!

  2. Hi there, First of all please accept my huge gratitude for providing this script, it is a great help and i was looking for something like this very desperately.
    But as I haven’t used perl before so i need your help regarding the error message i receive, when I try to invoke the script.
    It says:
    ” Can’t locate AnyEvent.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at notifier.perl line 6.
    BEGIN failed–compilation aborted at notifier.perl line 6.”
    What do i need to do to fix this problem.
    Secondly i wanted further help regarding
    1. How to add more events. like “File modification event”
    2. Connecting this script to a postgresql data for logging the events with corresponding timestamp to the database.

    Many thanks once again for all the help.

    Cheers,

    Malik Junaid

  3. Hi Johnny,

    this is great, it work like a charm 🙂

    A small change you may consider:

    > use File::Basename;
    > my $basename = basename($0);
    > my $PID_FILE = “/var/run/$basename.pid”;

    otherwise it won’t start unless you are in the same directory as the script itself

    (I have added a init script an run it as a daemon at system startup)

    Best Regards

    Yiorgos

  4. The -f check you do for the IN_DELETE and IN_MOVED_FROM will not work because at that point (firing of the inotify2 event) that pathname no longer exists. Your example daemon will never print out those two notices.

    Also, you don’t remove any watchers when a directory is deleted. 🙂

    1. for IN_DELETE and IN_MOVED_FROM you can use following change.

      if(-d $filename && $e->IN_CREATE) {
      create_watcher($inotify, $filename);
      return
      }
      elsif($e->IN_DELETE) {
      print __LINE__ . “: IN_DELETE $filename\n”;
      }
      elsif($e->IN_MOVED_FROM){
      print “IN_MOVE_FROM $filename\n”
      }
      elsif(-f $filename){
      if($e->IN_CLOSE_WRITE){
      print “IN_CLOSE_WRITE $filename\n”
      }
      #elsif($e->IN_MOVED_FROM){
      # print “IN_MOVE_FROM $filename\n”
      #}
      elsif($e->IN_MOVED_TO){
      print “IN_MOVED_TO $filename\n”
      }
      #elsif($e->IN_DELETE){
      # print “IN_DELETE $filename\n”
      #}
      }

  5. This does not seem to be detecting deletion of files or directories (as noted in other comments), seems like the code should be working for that. Has anyone gotten this to work for deletion? It is working great for creates…

Leave a Reply to Junaid Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Previous Post

Backup to free Hetzner FTP

Next Post

PostgreSQL 9.2 Master – Slave Monitoring

Related Posts