Skip to content
Johnny Morano's Tech Articles

Johnny Morano's Tech Articles

Ramblings of an old-fashioned space cowboy

Menu
  • About
  • Privacy Policy
Menu

Recursive Inotify Daemon

Posted on October 2, 2012November 8, 2012 by Johnny Morano

There aren’t many Inotify daemons available that can work recursive and offer a descent flexibility regarding Inotify signals. For basic rsync operations, lsyncd isn’t that bad but it isn’t that flexible for system administration. The fact that it is written and configured in Lua, makes it actually really complex (considering that you don’t know to develop in Lua). Most system administrators do know or at least understand Perl.

Linux::Inotify2 is a CPAN module which allows to catch Linux Inotify signals (duh) and react upon them, in realtime. Combined with the AnyEvent CPAN module, you can use a plethora of event loop libraries available in your system, to have an amazingly fast inotify daemon.

The first thing we’ll discuss, is how to turn a regular Perl script into a standalone daemon process.

sub daemonize {
# Inspired by http://stackoverflow.com/questions/766397/how-can-i-run-a-perl-script-as-a-system-daemon-in-linux
    POSIX::setsid or die "setsid: $!";
    my $pid = fork ();
    if ($pid < 0) {
        die "fork: $!";
    } 
    elsif ($pid) {
        exit 0;
    }

    chdir "/";
    umask 0;
    foreach (0 .. (POSIX::sysconf (&POSIX::_SC_OPEN_MAX) || 1024)) { 
        POSIX::close $_ 
    }

    open (STDIN, "/dev/null");
    open (STDOUT, ">>/tmp/log.txt");
    open (STDERR, ">&STDOUT");

    # Save PID to disk
    open my $pid_file, '>', '/var/run/script.pid'
        or die "Could not open PID file: $!\n";
    print { $pid_file } "$$";
    close ($pid_file);
} 

The process will get forked and the default IO handlers will be redirected. A PID file will also be created, so that the process can killed easily afterwards.
For example:

kill $(cat /var/run/script.pid)

Next we will create a inotify signal handler and search for directories to watch for changes. In our example, we will watch every subdirectory of the ‘/data/vault’ directory (the parent directory itself is not included in this example):

# Create Inotify object
my $inotify = Linux::Inotify2->new()
    or die "Failed to created inotify object: $!\n";

# Search for directories to watch
find({ wanted => sub { -d $_ 
                       && create_watcher($inotify, $File::Find::name) }  
     }
    , '/data/vault');

The ‘find’ subroutine comes from the File::Find module, which is a standard Perl module.
For every directory found, the ‘create_watcher’ subroutine will be called, with the full path of the found directory.

sub create_watcher {
    my ($inotify, $dir) = @_;
    my $watcher = $inotify->watch($dir, IN_CREATE | IN_CLOSE_WRITE | IN_MOVE | IN_DELETE, sub {
            my $e = shift;
            my $filename  = $e->fullname;
            
            if(-d $filename && $e->IN_CREATE) {
                create_watcher($inotify, $filename);
                return
            }
            elsif(-f $filename){
                if($e->IN_CLOSE_WRITE){
                    print "IN_CLOSE_WRITE $filename\n"
                }
                elsif($e->IN_MOVED_FROM){
                    print "IN_MOVE_FROM $filename\n"
                }
                elsif($e->IN_MOVED_TO){
                    print "IN_MOVED_TO $filename\n"
                }
                elsif($e->IN_DELETE){
                    print "IN_DELETE $filename\n"
                }
            }
    });
    print "Watching $dir\n";
    $W{$dir} = $watcher;
}

The watcher code will watch for the following inotify signals:

  • IN_CREATE
  • IN_CLOSE_WRITE
  • IN_MOVE
  • IN_DELETE

If a new directory is created within ‘/data/vault’, the ‘create_watcher’ subroutine will be called again for this new directory. For files, the code will check what inotify signal was spawned and act accordingly.

The last to do is to initialize the event loop, using AnyEvent:

my $cv = AnyEvent->condvar;
# Create event loop poller
my $poller = AnyEvent->io(
        fh   => $inotify->fileno,
        poll => 'r',
        cb   => sub { $inotify->poll }
);

# Receive event signals (inotify signals)
$cv->recv;

The above example shows how the inotify object ($inotify) is used in the event listener.

These examples explain how to create a simple inotify daemon, which can recursively go through directories and then watch these directories for certain inotify signals.

The full script:

#!/usr/bin/env perl 
use strict;
use warnings;
use utf8;

use AnyEvent;
use Linux::Inotify2;
use File::Find;
use POSIX;

my $PID_FILE = "/var/run/$0.pid";

# Fork this process, to run as a daemon
daemonize();

# enable autoflush to have faster logging
$|++;

# Catch kill signals
local $SIG{TERM} = sub {
    if(-f $PID_FILE){
        unlink($PID_FILE)
    }

    print("$0 daemon killed.");
    exit 0;
};
local $SIG{INT} = $SIG{TERM};

my $cv = AnyEvent->condvar;
# watcher container hash
my %W;

# Create Inotify object
my $inotify = Linux::Inotify2->new()
    or die "Failed to created inotify object: $!\n";

# Search for directories to watch
find({ wanted => sub { -d $_ 
                       && create_watcher($inotify, $File::Find::name) }  
     }
    , '/data/vault');


# Create event loop poller
my $poller = AnyEvent->io(
        fh   => $inotify->fileno,
        poll => 'r',
        cb   => sub { $inotify->poll }
);

# Receive event signals (inotify signals)
$cv->recv;

#
# Subroutines
#
sub create_watcher {
    my ($inotify, $dir) = @_;
    my $watcher = $inotify->watch($dir, IN_CREATE | IN_CLOSE_WRITE | IN_MOVE | IN_DELETE, sub {
            my $e = shift;
            my $filename  = $e->fullname;
            
            if(-d $filename && $e->IN_CREATE) {
                create_watcher($inotify, $filename);
                return
            }
            elsif(-f $filename){
                if($e->IN_CLOSE_WRITE){
                    print "IN_CLOSE_WRITE $filename\n"
                }
                elsif($e->IN_MOVED_FROM){
                    print "IN_MOVE_FROM $filename\n"
                }
                elsif($e->IN_MOVED_TO){
                    print "IN_MOVED_TO $filename\n"
                }
                elsif($e->IN_DELETE){
                    print "IN_DELETE $filename\n"
                }
            }
    });
    print "Watching $dir\n";
    $W{$dir} = $watcher;
}

sub daemonize {
# Inspired by http://stackoverflow.com/questions/766397/how-can-i-run-a-perl-script-as-a-system-daemon-in-linux
    POSIX::setsid or die "setsid: $!";
    my $pid = fork ();
    if ($pid < 0) {
        die "fork: $!";
    } 
    elsif ($pid) {
        exit 0;
    }

    chdir "/";
    umask 0;
    foreach (0 .. (POSIX::sysconf (&POSIX::_SC_OPEN_MAX) || 1024)) { 
        POSIX::close $_ 
    }

    open (STDIN, "/dev/null");
    open (STDOUT, ">>/tmp/log.txt");
    open (STDERR, ">&STDOUT");

    # Save PID to disk
    open my $pid_file, '>', $PID_FILE
        or die "Could not open PID file: $!\n";
    print { $pid_file } "$$";
    close ($pid_file);
} 

14 thoughts on “Recursive Inotify Daemon”

  1. Pingback: Linux : SL Tech Umbrella
  2. blank Sarakenidis Nikos says:
    November 7, 2012 at 12:04

    Thanks for this nice idea!

    However, I’m having some problems. For some reason, while the script works as expected when monitoring the initial directory_tree and seems to be adding newly created subdirectories to its watch list, when new files are created inside those aforementioned directories, it does not notify correctly.

    e.g.
    /data/vault
    touch /data/vault/testfile (script notifies correctly)
    mkdir -p /data/vault/testdir (script notifies and adds testdir to its watch list, or at least that’s what it says)
    touch /data/vault/testdir/testfile (script FAILS to notify correctly)

    Any ideas?

    Cheers, Nick

    Reply
    1. blank Johnny Morano says:
      November 7, 2012 at 12:55

      Hi Nick.

      That is strange, and I’m unable to reproduce your problem. On my PC, the script gives the following output:

      Watching /data/vault
      IN_CLOSE_WRITE /data/vault/log.txt
      Watching /home/jmorano/vault/test_dir
      IN_CLOSE_WRITE /data/vault/test_dir/test_file_in_test_dir
      IN_CLOSE_WRITE /data/vault/test_dir/touch_test
      

      The first two IN_CLOSE_WRITE statements were created using ‘echo “bla” > file’ and the last one was a ‘touch’.

      What was the output of the script (in /tmp/log.txt) on your side?

      Cheers,
      Johnny

      Reply
      1. blank Sarakenidis Nikos says:
        November 7, 2012 at 13:30

        1. touch /data/vault/testfile
        2. mkdir -p /data/vault/testdir
        3. touch /data/vault/testdir/testfile

        IN_CLOSE_WRITE /data/vault/testfile
        Watching /data/vault/testdir

        Third command does not produce any output

        Is there a limit to the number of subdirectories inotify can watch? Cause mine is huge. Maybe that’s the problem.

        Reply
        1. blank Sarakenidis Nikos says:
          November 7, 2012 at 14:06

          OK I’ve changed /proc/sys/fs/inotify/max_user_watches
          and everything works as expected. My bad!

          Cheers

          Reply
          1. blank Johnny Morano says:
            November 7, 2012 at 15:13

            Cool! It must be really huge then, cause overhere it is set by default to 65536:

            # cat /proc/sys/fs/inotify/max_user_watches
            65536

            Cheers!

  3. blank Sarakenidis Nikos says:
    November 7, 2012 at 15:38

    It was 8192 in one of my ubuntu servers here.. I need >250000

    Reply
  4. blank Junaid says:
    October 21, 2013 at 12:29

    Hi there, First of all please accept my huge gratitude for providing this script, it is a great help and i was looking for something like this very desperately.
    But as I haven’t used perl before so i need your help regarding the error message i receive, when I try to invoke the script.
    It says:
    ” Can’t locate AnyEvent.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at notifier.perl line 6.
    BEGIN failed–compilation aborted at notifier.perl line 6.”
    What do i need to do to fix this problem.
    Secondly i wanted further help regarding
    1. How to add more events. like “File modification event”
    2. Connecting this script to a postgresql data for logging the events with corresponding timestamp to the database.

    Many thanks once again for all the help.

    Cheers,

    Malik Junaid

    Reply
  5. Pingback: Monitor creation of hidden system files
  6. blank Yiorgos Stamoulis says:
    January 23, 2014 at 09:37

    Hi Johnny,

    this is great, it work like a charm 🙂

    A small change you may consider:

    > use File::Basename;
    > my $basename = basename($0);
    > my $PID_FILE = “/var/run/$basename.pid”;

    otherwise it won’t start unless you are in the same directory as the script itself

    (I have added a init script an run it as a daemon at system startup)

    Best Regards

    Yiorgos

    Reply
  7. blank Cory Albrecht says:
    September 19, 2014 at 05:44

    The -f check you do for the IN_DELETE and IN_MOVED_FROM will not work because at that point (firing of the inotify2 event) that pathname no longer exists. Your example daemon will never print out those two notices.

    Also, you don’t remove any watchers when a directory is deleted. 🙂

    Reply
    1. blank BFallert says:
      March 18, 2015 at 14:36

      for IN_DELETE and IN_MOVED_FROM you can use following change.

      if(-d $filename && $e->IN_CREATE) {
      create_watcher($inotify, $filename);
      return
      }
      elsif($e->IN_DELETE) {
      print __LINE__ . “: IN_DELETE $filename\n”;
      }
      elsif($e->IN_MOVED_FROM){
      print “IN_MOVE_FROM $filename\n”
      }
      elsif(-f $filename){
      if($e->IN_CLOSE_WRITE){
      print “IN_CLOSE_WRITE $filename\n”
      }
      #elsif($e->IN_MOVED_FROM){
      # print “IN_MOVE_FROM $filename\n”
      #}
      elsif($e->IN_MOVED_TO){
      print “IN_MOVED_TO $filename\n”
      }
      #elsif($e->IN_DELETE){
      # print “IN_DELETE $filename\n”
      #}
      }

      Reply
  8. blank Jack Nond says:
    September 22, 2014 at 14:12

    Here’s is a another simple watcher for directory trees: https://metacpan.org/pod/AnyEvent::Inotify::Simple#METHODS

    It can easily be extended to watch deletion of file and folders too.

    Reply
  9. blank Marcos Athanasoulis says:
    January 29, 2015 at 08:06

    This does not seem to be detecting deletion of files or directories (as noted in other comments), seems like the code should be working for that. Has anyone gotten this to work for deletion? It is working great for creates…

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Recent Posts

  • Use multiple Azure subscriptions in Terraform modules
  • Read the HAProxy UNIX socket file using Perl
  • A Prometheus Exporter framework written in Perl
  • Managing LDAP passwords with Perl
  • Libvirt guest startup issue with AppArmor
  • Deploy a PostgreSQL database with an initial schema using Ansible
  • Using Ansible to finalize Hashicorp Packer images

Categories

  • Automation (8)
  • Blog (60)
  • Database (4)
  • Development (37)
  • Linux (26)
  • Mac OS X (5)
  • Media (2)
  • OpenBSD (3)
  • Perl (34)
  • Photo (2)
  • PostgreSQL (4)
  • Terraform (5)
  • Web (11)

Tags

Ajax (3) Android (1) Ansible (2) API (5) AppArmor (1) Automation (5) Azure (3) azurerm (2) Bash (4) Cloud (2) CPAN (4) CSS (1) Debian (4) Dev (35) DevOps (11) EXIF (1) Facebook (1) Geotag (1) GMail (1) Google (3) Hack (2) Hashicorp (4) Hetzner (2) HTML (4) IMAP (2) IPTables (6) JavaScript (4) Libvirt (2) Linux (25) Logging (2) MacOSX (5) Media (2) Monitoring (6) MySQL (3) OpenBSD (4) Packer (1) Perl (35) PF (2) Postgresql (6) Security (7) SysAdmin (24) Terraform (4) Ubuntu (2) UNIX (9) Web 2.0 (3)

Archive

  • April 2022 (10)
  • March 2022 (6)
  • December 2016 (1)
  • March 2016 (1)
  • November 2015 (1)
  • November 2014 (1)
  • August 2014 (1)
  • May 2014 (1)
  • February 2014 (2)
  • December 2013 (1)
  • October 2013 (2)
  • September 2013 (2)
  • August 2013 (2)
  • October 2012 (1)
  • August 2012 (4)
  • March 2012 (3)
  • July 2011 (1)
  • June 2011 (2)
  • April 2011 (3)
  • March 2011 (4)
  • February 2011 (2)
  • December 2010 (2)
  • October 2010 (4)
  • September 2010 (1)
  • August 2010 (5)

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

Footer

  • Shihai Corp
  • My Photo website
© 2022 Johnny Morano's Tech Articles | Powered by Superbs Personal Blog theme
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Do not sell my personal information.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT