Recursive Inotify Daemon

There aren’t many Inotify daemons available that can work recursive and offer a descent flexibility regarding Inotify signals. For basic rsync operations, lsyncd isn’t that bad but it isn’t that flexible for system administration. The fact that it is written and configured in Lua, makes it actually really complex (considering that you don’t know to develop in Lua). Most system administrators do know or at least understand Perl.

Linux::Inotify2 is a CPAN module which allows to catch Linux Inotify signals (duh) and react upon them, in realtime. Combined with the AnyEvent CPAN module, you can use a plethora of event loop libraries available in your system, to have an amazingly fast inotify daemon.

The first thing we’ll discuss, is how to turn a regular Perl script into a standalone daemon process.

sub daemonize {
# Inspired by http://stackoverflow.com/questions/766397/how-can-i-run-a-perl-script-as-a-system-daemon-in-linux
    POSIX::setsid or die "setsid: $!";
    my $pid = fork ();
    if ($pid < 0) {
        die "fork: $!";
    } 
    elsif ($pid) {
        exit 0;
    }

    chdir "/";
    umask 0;
    foreach (0 .. (POSIX::sysconf (&POSIX::_SC_OPEN_MAX) || 1024)) { 
        POSIX::close $_ 
    }

    open (STDIN, "</dev/null");
    # open (STDOUT, ">/dev/null");
    open (STDOUT, ">>/tmp/log.txt");
    open (STDERR, ">&STDOUT");

    # Save PID to disk
    open my $pid_file, '>', '/var/run/script.pid'
        or die "Could not open PID file: $!\n";
    print { $pid_file } "$$";
    close ($pid_file);
}

sub daemonize {

# Inspired by http://stackoverflow.com/questions/766397/how-can-i-run-a-perl-script-as-a-system-daemon-in-linux

POSIX::setsid or die "setsid: $!";

my $pid = fork ();

if ($pid < 0) {

die "fork: $!";

}

elsif ($pid) {

exit 0;

}

chdir "/";

umask 0;

foreach (0 .. (POSIX::sysconf (&POSIX::_SC_OPEN_MAX) || 1024)) {

POSIX::close $_

}

open (STDIN, "</dev/null");

# open (STDOUT, ">/dev/null");

open (STDOUT, ">>/tmp/log.txt");

open (STDERR, ">&STDOUT");

# Save PID to disk

open my $pid_file, '>', '/var/run/script.pid'

or die "Could not open PID file: $!\n";

print { $pid_file } "$$";

close ($pid_file);

}

The process will get forked and the default IO handlers will be redirected. A PID file will also be created, so that the process can killed easily afterwards.
For example:

kill $(cat /var/run/script.pid)

1	kill $(cat /var/run/script.pid)

Next we will create a inotify signal handler and search for directories to watch for changes. In our example, we will watch every subdirectory of the ‘/data/vault’ directory (the parent directory itself is not included in this example):

# Create Inotify object
my $inotify = Linux::Inotify2->new()
    or die "Failed to created inotify object: $!\n";

# Search for directories to watch
find({ wanted => sub { -d $_ 
                       && create_watcher($inotify, $File::Find::name) }  
     }
    , '/data/vault');

# Create Inotify object

my $inotify = Linux::Inotify2->new()

or die "Failed to created inotify object: $!\n";

# Search for directories to watch

find({ wanted => sub { -d $_

&& create_watcher($inotify, $File::Find::name) }

}

, '/data/vault');

The ‘find’ subroutine comes from the File::Find module, which is a standard Perl module.
For every directory found, the ‘create_watcher’ subroutine will be called, with the full path of the found directory.

sub create_watcher {
    my ($inotify, $dir) = @_;
    my $watcher = $inotify->watch($dir, IN_CREATE | IN_CLOSE_WRITE | IN_MOVE | IN_DELETE, sub {
            my $e = shift;
            my $filename  = $e->fullname;
            
            if(-d $filename && $e->IN_CREATE) {
                create_watcher($inotify, $filename);
                return
            }
            elsif(-f $filename){
                if($e->IN_CLOSE_WRITE){
                    print "IN_CLOSE_WRITE $filename\n"
                }
                elsif($e->IN_MOVED_FROM){
                    print "IN_MOVE_FROM $filename\n"
                }
                elsif($e->IN_MOVED_TO){
                    print "IN_MOVED_TO $filename\n"
                }
                elsif($e->IN_DELETE){
                    print "IN_DELETE $filename\n"
                }
            }
    });
    print "Watching $dir\n";
    $W{$dir} = $watcher;
}

sub create_watcher {

my ($inotify, $dir) = @_;

my $watcher = $inotify->watch($dir, IN_CREATE | IN_CLOSE_WRITE | IN_MOVE | IN_DELETE, sub {

my $e = shift;

my $filename = $e->fullname;

if(-d $filename && $e->IN_CREATE) {

create_watcher($inotify, $filename);

return

}

elsif(-f $filename){

if($e->IN_CLOSE_WRITE){

print "IN_CLOSE_WRITE $filename\n"

}

elsif($e->IN_MOVED_FROM){

print "IN_MOVE_FROM $filename\n"

}

elsif($e->IN_MOVED_TO){

print "IN_MOVED_TO $filename\n"

}

elsif($e->IN_DELETE){

print "IN_DELETE $filename\n"

}

});

print "Watching $dir\n";

$W{$dir} = $watcher;

}

The watcher code will watch for the following inotify signals:

IN_CREATE
IN_CLOSE_WRITE
IN_MOVE
IN_DELETE

If a new directory is created within ‘/data/vault’, the ‘create_watcher’ subroutine will be called again for this new directory. For files, the code will check what inotify signal was spawned and act accordingly.

The last to do is to initialize the event loop, using AnyEvent:

my $cv = AnyEvent->condvar;
# Create event loop poller
my $poller = AnyEvent->io(
        fh   => $inotify->fileno,
        poll => 'r',
        cb   => sub { $inotify->poll }
);

# Receive event signals (inotify signals)
$cv->recv;

my $cv = AnyEvent->condvar;

# Create event loop poller

my $poller = AnyEvent->io(

fh => $inotify->fileno,

poll => 'r',

cb => sub { $inotify->poll }

);

# Receive event signals (inotify signals)

$cv->recv;

The above example shows how the inotify object ($inotify) is used in the event listener.

These examples explain how to create a simple inotify daemon, which can recursively go through directories and then watch these directories for certain inotify signals.

The full script:

#!/usr/bin/env perl 
use strict;
use warnings;
use utf8;

use AnyEvent;
use Linux::Inotify2;
use File::Find;
use POSIX;

my $PID_FILE = "/var/run/$0.pid";

# Fork this process, to run as a daemon
daemonize();

# enable autoflush to have faster logging
$|++;

# Catch kill signals
local $SIG{TERM} = sub {
    if(-f $PID_FILE){
        unlink($PID_FILE)
    }

    print("$0 daemon killed.");
    exit 0;
};
local $SIG{INT} = $SIG{TERM};

my $cv = AnyEvent->condvar;
# watcher container hash
my %W;

# Create Inotify object
my $inotify = Linux::Inotify2->new()
    or die "Failed to created inotify object: $!\n";

# Search for directories to watch
find({ wanted => sub { -d $_ 
                       && create_watcher($inotify, $File::Find::name) }  
     }
    , '/data/vault');


# Create event loop poller
my $poller = AnyEvent->io(
        fh   => $inotify->fileno,
        poll => 'r',
        cb   => sub { $inotify->poll }
);

# Receive event signals (inotify signals)
$cv->recv;

#
# Subroutines
#
sub create_watcher {
    my ($inotify, $dir) = @_;
    my $watcher = $inotify->watch($dir, IN_CREATE | IN_CLOSE_WRITE | IN_MOVE | IN_DELETE, sub {
            my $e = shift;
            my $filename  = $e->fullname;
            
            if(-d $filename && $e->IN_CREATE) {
                create_watcher($inotify, $filename);
                return
            }
            elsif(-f $filename){
                if($e->IN_CLOSE_WRITE){
                    print "IN_CLOSE_WRITE $filename\n"
                }
                elsif($e->IN_MOVED_FROM){
                    print "IN_MOVE_FROM $filename\n"
                }
                elsif($e->IN_MOVED_TO){
                    print "IN_MOVED_TO $filename\n"
                }
                elsif($e->IN_DELETE){
                    print "IN_DELETE $filename\n"
                }
            }
    });
    print "Watching $dir\n";
    $W{$dir} = $watcher;
}

sub daemonize {
# Inspired by http://stackoverflow.com/questions/766397/how-can-i-run-a-perl-script-as-a-system-daemon-in-linux
    POSIX::setsid or die "setsid: $!";
    my $pid = fork ();
    if ($pid < 0) {
        die "fork: $!";
    } 
    elsif ($pid) {
        exit 0;
    }

    chdir "/";
    umask 0;
    foreach (0 .. (POSIX::sysconf (&POSIX::_SC_OPEN_MAX) || 1024)) { 
        POSIX::close $_ 
    }

    open (STDIN, "</dev/null");
    # open (STDOUT, ">/dev/null");
    open (STDOUT, ">>/tmp/log.txt");
    open (STDERR, ">&STDOUT");

    # Save PID to disk
    open my $pid_file, '>', $PID_FILE
        or die "Could not open PID file: $!\n";
    print { $pid_file } "$$";
    close ($pid_file);
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

#!/usr/bin/env perl

use strict;

use warnings;

use utf8;

use AnyEvent;

use Linux::Inotify2;

use File::Find;

use POSIX;

my $PID_FILE = "/var/run/$0.pid";

# Fork this process, to run as a daemon

daemonize();

# enable autoflush to have faster logging

$|++;

# Catch kill signals

local $SIG{TERM} = sub {

if(-f $PID_FILE){

unlink($PID_FILE)

}

print("$0 daemon killed.");

exit 0;

};

local $SIG{INT} = $SIG{TERM};

my $cv = AnyEvent->condvar;

# watcher container hash

my %W;

# Create Inotify object

my $inotify = Linux::Inotify2->new()

or die "Failed to created inotify object: $!\n";

# Search for directories to watch

find({ wanted => sub { -d $_

&& create_watcher($inotify, $File::Find::name) }

}

, '/data/vault');

# Create event loop poller

my $poller = AnyEvent->io(

fh => $inotify->fileno,

poll => 'r',

cb => sub { $inotify->poll }

);

# Receive event signals (inotify signals)

$cv->recv;

# Subroutines

sub create_watcher {

my ($inotify, $dir) = @_;

my $watcher = $inotify->watch($dir, IN_CREATE | IN_CLOSE_WRITE | IN_MOVE | IN_DELETE, sub {

my $e = shift;

my $filename = $e->fullname;

if(-d $filename && $e->IN_CREATE) {

create_watcher($inotify, $filename);

return

}

elsif(-f $filename){

if($e->IN_CLOSE_WRITE){

print "IN_CLOSE_WRITE $filename\n"

}

elsif($e->IN_MOVED_FROM){

print "IN_MOVE_FROM $filename\n"

}

elsif($e->IN_MOVED_TO){

print "IN_MOVED_TO $filename\n"

}

elsif($e->IN_DELETE){

print "IN_DELETE $filename\n"

}

});

print "Watching $dir\n";

$W{$dir} = $watcher;

}

sub daemonize {

# Inspired by http://stackoverflow.com/questions/766397/how-can-i-run-a-perl-script-as-a-system-daemon-in-linux

POSIX::setsid or die "setsid: $!";

my $pid = fork ();

if ($pid < 0) {

die "fork: $!";

}

elsif ($pid) {

exit 0;

}

chdir "/";

umask 0;

foreach (0 .. (POSIX::sysconf (&POSIX::_SC_OPEN_MAX) || 1024)) {

POSIX::close $_

}

open (STDIN, "</dev/null");

# open (STDOUT, ">/dev/null");

open (STDOUT, ">>/tmp/log.txt");

open (STDERR, ">&STDOUT");

# Save PID to disk

open my $pid_file, '>', $PID_FILE

or die "Could not open PID file: $!\n";

print { $pid_file } "$$";

close ($pid_file);

}

Comments

Recursive Inotify Daemon — 14 Comments

Pingback: Linux : SL Tech Umbrella

Thanks for this nice idea!

However, I’m having some problems. For some reason, while the script works as expected when monitoring the initial directory_tree and seems to be adding newly created subdirectories to its watch list, when new files are created inside those aforementioned directories, it does not notify correctly.

e.g.
/data/vault
touch /data/vault/testfile (script notifies correctly)
mkdir -p /data/vault/testdir (script notifies and adds testdir to its watch list, or at least that’s what it says)
touch /data/vault/testdir/testfile (script FAILS to notify correctly)

Any ideas?

Cheers, Nick

Reply ↓

Hi Nick.

That is strange, and I’m unable to reproduce your problem. On my PC, the script gives the following output:

Watching /data/vault
IN_CLOSE_WRITE /data/vault/log.txt
Watching /home/jmorano/vault/test_dir
IN_CLOSE_WRITE /data/vault/test_dir/test_file_in_test_dir
IN_CLOSE_WRITE /data/vault/test_dir/touch_test

Watching /data/vault

IN_CLOSE_WRITE /data/vault/log.txt

Watching /home/jmorano/vault/test_dir

IN_CLOSE_WRITE /data/vault/test_dir/test_file_in_test_dir

IN_CLOSE_WRITE /data/vault/test_dir/touch_test

The first two IN_CLOSE_WRITE statements were created using ‘echo “bla” > file’ and the last one was a ‘touch’.

What was the output of the script (in /tmp/log.txt) on your side?

Cheers,
Johnny

Sarakenidis Nikos on November 7, 2012 at 13:30 said:

1. touch /data/vault/testfile
2. mkdir -p /data/vault/testdir
3. touch /data/vault/testdir/testfile

IN_CLOSE_WRITE /data/vault/testfile
Watching /data/vault/testdir

Third command does not produce any output

Is there a limit to the number of subdirectories inotify can watch? Cause mine is huge. Maybe that’s the problem.

Reply ↓
- Sarakenidis Nikos on November 7, 2012 at 14:06 said:
  
  OK I’ve changed /proc/sys/fs/inotify/max_user_watches
  and everything works as expected. My bad!
  
  Cheers
  
  Reply ↓
  - Johnny Morano on November 7, 2012 at 15:13 said:
    
    Cool! It must be really huge then, cause overhere it is set by default to 65536:
    
    # cat /proc/sys/fs/inotify/max_user_watches
    65536
    
    Cheers!

Sarakenidis Nikos on November 7, 2012 at 15:38 said:

It was 8192 in one of my ubuntu servers here.. I need >250000

Reply ↓
Junaid on October 21, 2013 at 12:29 said:

Hi there, First of all please accept my huge gratitude for providing this script, it is a great help and i was looking for something like this very desperately.
But as I haven’t used perl before so i need your help regarding the error message i receive, when I try to invoke the script.
It says:
” Can’t locate AnyEvent.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at notifier.perl line 6.
BEGIN failed–compilation aborted at notifier.perl line 6.”
What do i need to do to fix this problem.
Secondly i wanted further help regarding
1. How to add more events. like “File modification event”
2. Connecting this script to a postgresql data for logging the events with corresponding timestamp to the database.

Many thanks once again for all the help.

Cheers,

Malik Junaid

Reply ↓
Pingback: Monitor creation of hidden system files
Yiorgos Stamoulis on January 23, 2014 at 09:37 said:

Hi Johnny,

this is great, it work like a charm 🙂

A small change you may consider:

> use File::Basename;
> my $basename = basename($0);
> my $PID_FILE = “/var/run/$basename.pid”;

otherwise it won’t start unless you are in the same directory as the script itself

(I have added a init script an run it as a daemon at system startup)

Best Regards

Yiorgos

Reply ↓
Cory Albrecht on September 19, 2014 at 05:44 said:

The -f check you do for the IN_DELETE and IN_MOVED_FROM will not work because at that point (firing of the inotify2 event) that pathname no longer exists. Your example daemon will never print out those two notices.

Also, you don’t remove any watchers when a directory is deleted. 🙂

Reply ↓
- BFallert on March 18, 2015 at 14:36 said:
  
  for IN_DELETE and IN_MOVED_FROM you can use following change.
  
  if(-d $filename && $e->IN_CREATE) {
  create_watcher($inotify, $filename);
  return
  }
  elsif($e->IN_DELETE) {
  print __LINE__ . “: IN_DELETE $filename\n”;
  }
  elsif($e->IN_MOVED_FROM){
  print “IN_MOVE_FROM $filename\n”
  }
  elsif(-f $filename){
  if($e->IN_CLOSE_WRITE){
  print “IN_CLOSE_WRITE $filename\n”
  }
  #elsif($e->IN_MOVED_FROM){
  # print “IN_MOVE_FROM $filename\n”
  #}
  elsif($e->IN_MOVED_TO){
  print “IN_MOVED_TO $filename\n”
  }
  #elsif($e->IN_DELETE){
  # print “IN_DELETE $filename\n”
  #}
  }
  
  Reply ↓
Jack Nond on September 22, 2014 at 14:12 said:

Here’s is a another simple watcher for directory trees: https://metacpan.org/pod/AnyEvent::Inotify::Simple#METHODS

It can easily be extended to watch deletion of file and folders too.

Reply ↓
Marcos Athanasoulis on January 29, 2015 at 08:06 said:

This does not seem to be detecting deletion of files or directories (as noted in other comments), seems like the code should be working for that. Has anyone gotten this to work for deletion? It is working great for creates…

Reply ↓

Johnny Morano's Blog

Geek talk, extreme sports and travels

Recursive Inotify Daemon

Comments

Recursive Inotify Daemon — 14 Comments

Leave a Reply Cancel reply