A bit of test code to solve it. Of course use mysqladmin to shutdown that instance first. I only bring this up because I just reset an instance and mysqladmin was in a constant loop of connection. The pid file was there, but the service for that one was not.

sub get_pid {
my $re = shift;
my $processes = Proc::ProcessTable->new();
foreach my $p (@{ $processes->table() }){
if($p->{‘cmndline’} =~ m#$re#) {
# Winner Winner Chicken Dinner
return $p->{‘pid’};
}
}

return 0;
}

my $pid = get_pid(‘mysqld_safe.+15034.+’);
printf “PID: %d\n”, $pid;
my $pid = get_pid(‘mysqld .+15034.+’);
printf “PID: %d\n”, $pid;

I can’t say I have any suggestions that would make your script better. My changes are specific to what I’m doing and I’d thought you’d be interested in seeing how I applied it to the problems I’m facing.

Hi Chris

I’d be more than happy to see your suggestions or corrections 😉

My original approach at keeping these programs up is to run each process in the foreground under a parent. On SIGCHLD the parent makes a note and starts it back up.

I never considered the approach of just monitoring /proc. That can be difficult, but I also never considered using a regular expression as a check either.

As a daemon I do run the risk of it dying. Your solution was cron. Since only one can run on a system I typically lock $0 and check iton startup to make sure another copy is not running. Using that idea I can stay a daemon and cron could try to start it every hour. It’ll just fail to lock and exit. If it died within that 30 minute cycle it would run again.

I use databases to create a concept of “organizations”. Each daemon type will spawn one child per org. That is why I started from a manager.

I extended your code to have a default section for $services. I then build the other sections off the database. Instead of a cron job I’m am doing daemon. When changes are made to an org a daemon from a group will be restarted. In your program I implemented a UNIX socket where I could connect, dump status, disable a program, enable a program, stop, etc.

Each org does have an org_config table that I use to disable a program there. When building the config I check that. I also check for config changes each 120s cycle. Before, when I added an org I had to restart the whole group. With the mods I done with your program I just create it and it works.

I did this as rush for that one VMware incident and feature creep hit at light speed. It is a mess so I’ll clean it up and send it to you. You can contact me via my email if you wish.

Chris

foreach my $p (@{ $processes }){

Is this due to some change in the module, or in perl?

Yeah you’re right! I have updated this blog post once more 😉

Hard to tell since I only saw the code after you reworked it, but it looks like the bit that was using the ‘re’ part of your structure got lost. As a result, the current version is awfully dependent on pidfiles. What if the service is running but the pidfile is gone somehow? Looks to me like your script would just happily start it up again.

It’s always good to simplify, but you have to be careful not to go too far. There’s unnecessary complexity, and then there’s necessary complexity. ;->