The Docker installation is well documented at https://docs.docker.com/get-docker/ and is well supported amongst the most popular operating systems. The installation itself will not be covered in this article. If you want to get familiar with the details of Docker, start with the documentation at https://docs.docker.com/get-started/.

All code used in this article is available at: https://github.com/insani4c/docker-monitoring-stack.

In this article, we will see how to set up a monitoring solution based on:

• Prometheus
• Prometheus Node Exporter
• Prometheus Black Exporter
• Prometheus SNMP Exporter
• Loki
• Promtail
• Grafana

To monitor the deployed containers, we will also deploy Google’s cadvisor container, to get some interesting statistics and details in our Prometheus/ Grafana setup.

The Docker Compose file, called docker-compose.yml, contains all the information of the infrastructure such as:

• network information
• volumes
• services (the containers)
• …

Let’s start from the top of the file.

version: '3.8'

name: docmon

volumes:
  grafana-data: {}
  alertmanager-data: {}
  prometheus-data: {}
  loki-data: {}

At first at line 1, the version Docker Compose version is specified, to define which specifications are allowed. At line 3 a name for the container group or stack is set. And finally starting from line 5, data volumes (think disks) are defined, which will be used by the containers. These are persistent data volumes which will be reused unless the container has been completely removed.

Next, we will define the services in the docker-compose.yml:

services:
  cadvisor:
    image: 'gcr.io/cadvisor/cadvisor:latest'
    container_name: cadvisor
    restart: always
    mem_limit: 512m
    mem_reservation: 32m
    # ports: 
    #   - '8880:8080'
    volumes:
      - '/:/rootfs:ro'
      - '/var/run:/var/run:ro'
      - '/sys:/sys:ro'
      - '/var/lib/docker/:/var/lib/docker:ro'
      - '/dev/disk/:/dev/disk:ro'
    privileged: true
    devices: 
      - '/dev/kmsg:/dev/kmsg'

  prometheus:
    image: 'prom/prometheus:latest'
    container_name: prometheus
    restart: always
    mem_limit: 2048m
    mem_reservation: 256m
    cpus: 2
    # ports:
    #   - '9090:9090'
    volumes:
      - '$PROMETHEUS_HOME/config:/etc/prometheus'
      - 'prometheus-data:/prometheus'
    extra_hosts:
      myrouter: 192.168.1.1
      myswitch: 192.168.1.10
    depends_on:
      - cadvisor

Containers are defined as services. Each service will require at least:

• a service name (example line 2 and line 20)
• an image definition

All other options are optional or required by specific images.

The first image or container defined in the above example is cadvisor. This service provides statistics from Docker and the deployed containers to Prometheus. To be able to provide this information, the container must have read access to certain file paths or sockets on the hypervisor (read: the server where the Docker containers will be running). These are provided in the volumes section of the container. Here, directory paths on the hypervisor will be provided as mount partitions in the container, and they will be mounted with the readonly (:ro) parameter so that the container can’t make any changes to them.

Furthermore it provides access to a device (to read kernel messages), set memory and cpu limits and will run the container in privilege mode. The ports section has been put in comments, as it isn’t really require to expose ports, or make them available outside the Docker ecosystem. In our example, only Prometheus must be able to connect to it, and since Prometheus will be deployed as a container, we don’t need to be able to access the web service running on the container to read out the metrics or see the statistics.

The next container defined is called prometheus. For this container, volumes will be mounted to provide the Prometheus configuration files and to store the data to the volume called prometheus-data. It also defines an extra_hosts. These are entries that are typically defined in an /etc/hosts file, which Docker does not read from the hypervisor. And instead of deploying or mounting the hypervisor’s hosts file, extra host mappings can be defined or handed to the container, which is set up in the extra_hosts section as above.

At the end of the prometheus container definition, a depends_on section is configured, which means that the prometheus container won’t be deployed until the container names defined in that section are up and running.

Next we will define all other containers (see the second tab in the above code block, called the rest).

  hypervisor:
    image: 'prom/node-exporter:latest'
    container_name: hypervisor
    mem_limit: 128m
    mem_reservation: 32m
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'
      - '/proc:/host/proc:ro'
      - '/sys:/host/sys:ro'
    command:
      - '--path.rootfs=/host'
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
      - '--collector.systemd'
      - '--collector.cgroups'
    depends_on:
      - cadvisor

  prom_snmp:
    image: 'prom/snmp-exporter:latest'
    container_name: prom_snmp
    restart: always
    mem_limit: 128m
    mem_reservation: 32m
    # ports: 
    #   - '9116:9116'
    volumes:
      - '$PROMSNMP_HOME/config:/etc/snmp_exporter'
    extra_hosts:
      myrouter: 192.168.1.1
      myswitch: 192.168.1.10
    depends_on:
      - cadvisor
      - prometheus

  alertmanager:
    image: 'prom/alertmanager:latest'
    container_name: alertmanager
    restart: always
    mem_limit: 256m
    mem_reservation: 32m 
    # ports:
    #   - 9093:9093
    volumes:
      - '$ALERTMANAGER_HOME/config/alertmanager.yml:/etc/alertmanager/config.yml'
      - 'alertmanager-data:/alertmanager'
    command:
      - '--config.file=/etc/alertmanager/config.yml'
      - '--storage.path=/alertmanager'
    depends_on:
      - cadvisor
      - prometheus

  loki:
    image: 'grafana/loki:latest'
    container_name: loki
    restart: always
    mem_limit: 32768m
    mem_reservation: 8192m
    cpus: 6 
    ports:
      - '3100:3100'
    volumes:
      - '$LOKI_HOME/config:/etc/loki'
      - 'loki-data:/loki'
    depends_on:
      - cadvisor
      - prometheus
      - alertmanager

  blackbox_exporter:
    image: 'prom/blackbox-exporter:latest'
    container_name: blackbox_exporter
    restart: always
    mem_limit: 128m
    mem_reservation: 32m
    dns:
      - 8.8.8.8
      - 8.8.4.4
    # ports:
    #   - 9115:9115
    volumes:
      - '$BLACKBOXEXPORTER_HOME/config:/etc/blackboxexporter/'
    command:
      - '--config.file=/etc/blackboxexporter/config.yml'
    depends_on:
      - cadvisor
      - prometheus

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    restart: always
    mem_limit: 256m
    mem_reservation: 64m
    volumes:
      - $PROMTAIL_HOME/config:/etc/promtail/
      # to read container labels and logs
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
      - '/var/lib/docker/containers:/var/lib/docker/containers:ro'
      - '/var/log/ulog:/var/log/ulog/:ro'
    depends_on:
      - cadvisor
      - loki

  grafana:
    image: 'grafana/grafana:latest'
    container_name: grafana
    restart: always
    mem_limit: 2048m
    mem_reservation: 256m
    ports:
      - '3000:3000'
    volumes:
      - '$GRAFANA_HOME/config:/etc/grafana'
      - 'grafana-data:/var/lib/grafana'
      - '$GRAFANA_HOME/dashboards:/var/lib/grafana/dashboards'
    depends_on:
      - cadvisor
      - prometheus
      - loki
      - alertmanager

The rest of the code will deploy:

• a container called hypervisor, which is actually the Prometheus node-exporter for the hypervisor.
• a container called prom_snmp, which will retrieve SNMP statistics
• a container called blackbox_exporter, which mainly checks webservers and their SSL certificates
• a container called promtail, which collects logs and log statistics from the hypervisor
• a container called loki, which allows to store and index logs sent to the service by promtail (either container or as a service running on some external server)

Finally, the last container deployed is the grafana container. Besides its normal configuration file grafana.ini, the Docker container will also automatically provision (the provisioning sub directory in the config directory) datasources and dashboards so that no manual after-tasks are required once the containers are running.

The datasources can be preconfigured in a YAML file called default.yaml, stored in the provisioning/datasources/ sub directory.

apiVersion: 1

datasources:
 - name: Alertmanager
   type: alertmanager 
   access: proxy
   orgId: 1
   url: http://alertmanager:9093
   version: 1
   editable: false
   isDefault: false
   uid: DS_ALERTMANAGER
   jsonData:
    implementation: prometheus
 - name: Prometheus
   type: prometheus
   access: proxy
   orgId: 1
   url: http://prometheus:9090
   version: 1
   editable: false
   isDefault: true
   uid: DS_PROMETHEUS
   jsonData:
    alertmanagerUid: DS_ALERTMANAGER
    manageAlerts: true
    prometheusType: Prometheus
    prometheusVersion: 2.39.1
 - name: Loki
   type: loki 
   access: proxy
   orgId: 1
   url: http://loki:3100
   version: 1
   editable: false
   isDefault: false
   uid: DS_LOKI
   jsonData:
    alertmanagerUid: DS_ALERTMANAGER
    manageAlerts: true

Same thing goes for dashboards we want to have automatically deployed:

apiVersion: 1

providers:
 - name: 'default'
   orgId: 1
   folder: 'Custom'
   folderUid: ''
   type: file
   options:
     path: /var/lib/grafana/dashboards

Finally, if Docker is running on multiple network interfaces (for it is a hosted server, or it has internal and external IP addresses), you might want to limit access to the container to specific networks only.

Below is a netfilter example, which allows traffic only coming from 192.168.1.0/24 and from the network interface enp35so:

iptables -I DOCKER-USER -i enp35s0 ! -s 192.168.1.0/24 -m conntrack --ctdir ORIGINAL -j DROP

The chain DOCKER-USER is not flushed by Docker and thus can be created in a general firewall script of netfilter configuration, even at boot time:

-N DOCKER-USER
-I DOCKER-USER -i enp35s0 ! -s 192.168.1.0/24 -m conntrack --ctdir ORIGINAL -j DROP

The statistics part contains quite some interesting information for monitoring and alerting.

The below Perl code snippit will loop over a glob of socket files (for instance when you have multiple HAProxy configurations running as separate processes) and print out the values returned by the “show info” command.

use IO::Socket::UNIX;

foreach my $socket_file (glob("/run/haproxy/*.sock")){
    print "- Reading socket: $socket_file\n";
    my $client = IO::Socket::UNIX->new(
        Type => SOCK_STREAM(),
        Peer => $socket_file,
    );

    print "- show info\n";
    print $client "show info\n";
    my $header = <$client>;
    chomp($header);

    $header =~ s/^#\s+//;
    my @keys = split ',', $header;
    print "- header:$header\n";

    while (my $line = <$client>){
        next unless $line =~ /^.+/;

        chomp($line);
        my @values = split ',', $line;
        print " - Got $line\n";
        print "   $keys[$_]: ".($values[$_]//'')."\n" foreach 0..$#keys;
    }

    close $client;
}

A small example of how to use the framework:

my $exporter = Prometheus::Exporter->new({
    listen_port => 9090, 
    listen_addr => "127.0.0.1", 
    max_threads => 5,
});

$exporter->register_metrics({
    test_metric        => {type => "gauge",     desc => "A test metric"},
    test_metric_labels => {type => "gauge",     desc => "A test metric", labels => ["code=42", "code=99"]},
    test_counter       => {type => "counter",   desc => "A test metric"},
    test_histogram     => {type => "histogram", buckets => ['0.3', '0.6', '1.2', '+Inf']},
});

$exporter->register_collector(sub {
    my $timeout = int(rand(5));
    sleep $timeout;

    $exporter->get_metric("test_metric")->value(rand(100));
    $exporter->get_metric("test_metric_labels")->value([rand(42), rand(99)]);

    $test_counter += int(rand(20));
    $exporter->get_metric("test_counter")->value($test_counter);

    $histo_buckets{"0.3"}  += rand(20);
    $histo_buckets{"0.6"}  += $histo_buckets{"0.3"} + rand(20);
    $histo_buckets{"1.2"}  += $histo_buckets{"0.6"} + rand(20);
    $histo_buckets{"+Inf"} += $histo_buckets{"1.2"} + rand(20);
    my $histo_sum = 2.0 * $histo_buckets{"+Inf"};
    my $histo_count = $histo_buckets{"+Inf"};
    $exporter->get_metric("test_histogram")->value(\%histo_buckets, $histo_sum, $histo_count);
});

$exporter->run;

The framework will start a small HTTP daemon once run() is called and will handle all client requests by using threads. On each request, the framework will call the subroutine or coderef defined at register_collector(). Currently, that coderef must store the observed values by using the construct seen in the above example, by calling the value() method on the registered metric objects.

Also currently, the histogram implementation is not yet supporting labels.

With Promtail (https://grafana.com/docs/loki/latest/clients/promtail/), the above created log files can be sent to Loki so that they can finally be displayed in Grafana.

The installation of both Loki and Grafana are not covered in this article. The installation of Promtail is documented at https://grafana.com/docs/loki/latest/clients/promtail/installation/.

Once Promtail is installed, create the following configuration file at /etc/promtail-local-config.yaml:

server:                                                                                                                                                                                                            
  http_listen_port: 9080                                                                                                                                                                                           
  grpc_listen_port: 0                                                                                                                                                                                              
                                                                                                                                                                                                                   
positions:                                                                                                                                                                                                         
  filename: /var/tmp/promtail_positions.yaml                                                                                                                                                                       
                                                                                                                                                                                                                   
clients:                                                                                                                                                                                                           
  - url: http://loki_server:3100/loki/api/v1/push       
                                                                                                                                                               
scrape_configs:
    - job_name: iptableslogsjson
      static_configs:
      - targets:
          - localhost
        labels:
          instance: myhostname01
          job: iptableslogsjson
          __path__: /var/log/ulog/*json
      pipeline_stages:
      - json:
          expressions:
            timestamp: timestamp
            prefix: '"oob.prefix"'
            src: src_ip
            dst: dest_ip
      - labels:
          timestamp:
          prefix:
          src:
          dst:

With the above configuration, Promtail will create 4 extra labels per log line:

• timestamp: Contains the logged timestamp
• prefix: the NFLOG prefix string
• src: the source IP address
• dst: the destination IP address

Once the logs are arriving in Loki, and Loki has been configured as a datasource in Grafana, graphs can be created using LogQL.

Example:

sum(rate({job="iptableslogsjson"} [$__interval])) by (prefix)

In the example script we will be using a Storable state file and we will the statistics as a JSON object in the PostgreSQL database.

First let’s take a look at the query we will be executing:

SELECT schemaname, relname, seq_tup_read 
FROM pg_stat_all_tables 
WHERE seq_tup_read > '0' 
      AND relname NOT LIKE 'pg_%'
ORDER BY seq_tup_read desc

As you can see, PostgreSQL stores all the information we need about our tables in just one table, called pg_stat_all_tables. In this table there is a column called seq_tup_read, which will contain the information we need.

Just reading out this information is not going to be enough, because it contains information since the startup of your PostgreSQL database. Since production databases aren’t restarted (that often), we will have to compare this information with some previous information (hence the Storable state file).
Our plan is to run the script in a cronjob, each 5 minutes.

The statistics are also stored in as a JSON object in a database, just so that we could build some web interface for the statistics, in a later stage. And we want to keep a history of these statistics.

Furthermore the script will setuid to postgres (same like su – postgres on the command line), so that it could connect to the PostgreSQL UNIX socket file.

use strict;
use warnings;
use utf8;

use DBI;
use DateTime;
use POSIX qw/setuid/;
use Text::ASCIITable;
use JSON;

my $db   = 'mydatabase';
if(scalar @ARGV){
    $db = shift @ARGV;
}

my $host = '/var/run/postgresql';
my $user = 'postgres';
my $pass = 'undef';

my $state_db   = 'database_statistics';
my $state_host = '192.168.1.1';
my $state_user = 'skeletor';
my $state_pass = 'he-manisawhimp';

my $state_file = '/var/tmp/sequence_read.state';

# suid to postgres
setuid(scalar getpwnam 'postgres');

# define and open up the state file
my $state = {};
$state = retrieve $state_file if -f $state_file;

my $now      = DateTime->now;

# Connect to the database which we want to monitor
my $dbh = DBI->connect("dbi:Pg:dbname=$db;host=$host", $user, $pass) 
                or die "Could not connect to database: $!\n";

# Connect to the database that will be used to store the statistics
my $state_dbh = DBI->connect("dbi:Pg:dbname=$state_db;host=$state_host", $state_user, $state_pass) 
                or die "Could not connect to the State database '$state_db': $!\n";

my $sql = <<EOF;
SELECT schemaname, relname, seq_tup_read 
FROM pg_stat_all_tables 
WHERE seq_tup_read > '0' 
      AND relname NOT LIKE 'pg_%'
ORDER BY seq_tup_read desc
EOF

# Get the statistics
my $results = $dbh->selectall_arrayref( $sql, undef);

# Store the statistics as a JSON object in the second databse
eval {
    $state_dbh->do('INSERT INTO mydbschema.seq_tup_read (data) VALUES(?)', undef, encode_json($results));
};
if($@){
    print "Insert into state-db failed: $@\n";
}

# Prepare a nice ASCII table for output
my $t = Text::ASCIITable->new({ headingText => 'Seq Tup Read ' . $now->ymd('-')     . ' ' . $now->hms(':')});
$t->setCols('Schema Name','Relation Name ', 'Seq Tup Read', 'Increase (delta)');

my $row_count = 0;
foreach my $r (@{$results}){
    last if $row_count > 25;

    my (@values) = (@{$r});
    my ($increase, $delta) = (0, 0);
    # Calculate the increase and its delta
    if(defined $state->{last}{$r->[0].':'.$r->[1]}{seq_tup_read}){
        $increase = $r->[2] - $state->{last}{$r->[0].':'.$r->[1]}{seq_tup_read};
        $delta    = $increase / $state->{last}{$r->[0].':'.$r->[1]}{seq_tup_read} * 100;
        my $str = sprintf '%.0f (%.4f %%)', $increase, $delta;
        push @values, ($str);
    }
    else {
        push @values, '0 (0%)';
    }
    # Store this information for the next run of the script
    $state->{last}{$r->[0].':'.$r->[1]}{seq_tup_read} = $r->[2];
    $state->{last}{$r->[0].':'.$r->[1]}{delta}        = $delta;
    $state->{last}{$r->[0].':'.$r->[1]}{increase}     = $increase;

    # Only add the information to ASCII output table if there was an increase
    next unless $increase > 0;
    $t->addRow(@values);
    $row_count++;
}
# Print out the ASCII table
print $t;

nstore $state, $state_file;

Statistics about indexes can be easily retrieved from the PostgreSQL database system. All required information is stored in two tables:

• pg_stat_user_indexes
• pg_index

When joining these two tables, interesting information can be read in the following columns:

• idx_scan: has the query planner used this index for an ‘Index Scan’, the number returned is the amount of times it was used
• idx_tup_read: how many tuples have been read by using the index
• idx_tup_fetch: how many tuples have been fetch by using the index

A neat function called pg_relation_size() allows to fetch the on-disk size of a relation, in this case the index.

Based on this information, the monitoring query will be built up as follows:

SELECT 
    relid::regclass AS table, 
    indexrelid::regclass AS index, 
    pg_size_pretty(pg_relation_size(indexrelid::regclass)) AS index_size, 
    idx_tup_read, 
    idx_tup_fetch, 
    idx_scan
FROM 
    pg_stat_user_indexes 
    JOIN pg_index USING (indexrelid) 
WHERE 
    idx_scan = 0 
    AND indisunique IS FALSE

Now, all we need to do is write a script which stores this information in some kind of file and periodically report about the statistics.

First of all we will need a configuration file, which contains the database credentials.
I’ve chosen YAML because it is so versatile.

It will contain two important sets of information:

• The database credentials
• path to the state file

Example:

dsn: "dbi:Pg:host=/var/run/postgresql;database=testdb"
user: postgres
pass:
state_file: /var/tmp/monitor_unused_indexes.state

As you can see, we will be connect to the PostgreSQL database by using its UNIX socket.

The script will use Text::ASCIITable to output the statistics in a nice table. Storable is used to save our statistics to disk.

In the below script, we will check if an index was unused in a timespan of 30 days. If yes, the script will report this index to STDOUT.
Therefore, we will store a score and timestamp for each unused index in the state file.

#!/usr/bin/env perl
use strict;
use warnings;
use utf8;
use DBI;
use Storable qw/nstore retrieve/;
use YAML qw/LoadFile/;
use POSIX qw/setuid/;
use Getopt::Long;
use DateTime;
use Text::ASCIITable;

my $cfg_file = './monitor_unused_indexes.yaml';
my $verbose = 0;
GetOptions("cfg=s" => \$cfg_file,
           "verbose|v" => \$verbose, 
        );

my $sql = <<EOS;
SELECT 
    relid::regclass AS table, 
    indexrelid::regclass AS index, 
    pg_size_pretty(pg_relation_size(indexrelid::regclass)) AS index_size, 
    idx_tup_read, 
    idx_tup_fetch, 
    idx_scan
FROM 
    pg_stat_user_indexes 
    JOIN pg_index USING (indexrelid) 
WHERE 
    idx_scan = 0 
    AND indisunique IS FALSE
EOS

my ($cfg) = LoadFile($cfg_file);

# suid to postgres, other whatever user is configured in the config.yaml file
setuid(scalar getpwnam $cfg->{user});

# Connect to the database
my $dbh = DBI->connect($cfg->{dsn}, $cfg->{user}, $cfg->{pass}) 
            or die "Could not connect to database: $! (DBI ERROR: ".$DBI::errstr.")\n";

my $state;
if(-f $cfg->{state_file}){
    $state = retrieve $cfg->{state_file};
}

# Fetch the statistics
my $results = $dbh->selectall_arrayref( $sql, undef );

my $now_dt   = DateTime->now;

# Initialize the ASCII table
my $t = Text::ASCIITable->new({ headingText => 'INDEX STATISTICS'});
$t->setCols(qw/Table Index Index_Size idx_tup_read idx_tup_fetch idx_scan/);

# Analyze the results
foreach my $r (@$results){
    if($verbose){
        $t->addRow(@{$r});
    }
    # Only update the state file if --verbose was not specified.
    # This way the script can be check manually with --verbose many times and executed for instance
    # from a cronjob once a day without --verbose
    else {
        if(defined $state->{unused_indexes}{$r->[1]}){
            my $first_dt = DateTime->from_epoch( epoch => $state->{unused_indexes}{$r->[1]}{first_hit} );
            if($first_dt->add(days => $state->{unused_indexes}{$r->[1]}{score})->day == $now_dt->day ) {
                $state->{unused_indexes}{$r->[1]}{score}++;
            }
            else {
                $state->{unused_indexes}{$r->[1]}{score}     = 1;
                $state->{unused_indexes}{$r->[1]}{first_hit} = $now_dt->epoch;;
            }
        }
        else {
            $state->{unused_indexes}{$r->[1]}{score}     = 1;
            $state->{unused_indexes}{$r->[1]}{first_hit} = $now_dt->epoch;;
        }
    }
}

# Print out the statistics table, if --verbose was specified
print $t if $verbose; 

# Store the statistics to disk in a state file
nstore $state, $cfg->{state_file};

foreach my $idx (keys %{ $state->{unused_indexes} }){
    my $first_dt = DateTime->from_epoch( epoch => $state->{unused_indexes}{$idx}{first_hit} );
    if( $first_dt->add(days => 30) <= $now_dt ){
        my $line = "Index: $idx ready for deletion";
        $line .= " (score:" . $state->{unused_indexes}{$idx}{score};
        $line .= " (|first_hit:" . DateTime->from_epoch(epoch => $state->{unused_indexes}{$idx}{first_hit})->ymd . ")";

        print $line."\n" if $verbose;
    }
}

The script must be executed as ‘postgres’ user.

#!/bin/bash

# $Id: check_slave_replication.sh 3421 2013-08-09 07:52:44Z jmorano $

STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3
 
## Master (p_) and Slave (s_) DB Server Information	
export s_host=$1
export s_port=$2
export p_db=$3
export p_host=$4
export p_port=$5
 
export psql=/opt/postgresql/bin/psql
export bc=/usr/bin/bc
 
## Limits
export  critical_limit=83886080 # 5 * 16MB, size of 5 WAL files
export   warning_limit=16777216 # 16 MB, size of 1 WAL file
 
master_lag=$($psql -U postgres -h$p_host -p$p_port -A -t -c "SELECT pg_xlog_location_diff(pg_current_xlog_location(), '0/0') AS offset" $p_db)
slave_lag=$($psql -U postgres  -h$s_host -p$s_port -A -t -c "SELECT pg_xlog_location_diff(pg_last_xlog_receive_location(), '0/0') AS receive" $p_db)
replay_lag=$($psql -U postgres -h$s_host -p$s_port -A -t -c "SELECT pg_xlog_location_diff(pg_last_xlog_replay_location(), '0/0') AS replay" $p_db)
replay_timediff=$($psql -U postgres -h$s_host -p$s_port -A -t -c "SELECT NOW() - pg_last_xact_replay_timestamp() AS replication_delay" $p_db)
 
if [[ $master_lag -eq '' || $slave_lag -eq '' || $replay_lag -eq '' ]]; then
    echo "CRITICAL: Stream has no value to compare (is replication configured or connectivity problem?)"
    exit $STATE_CRITICAL
else
    if [[ $master_lag -eq $slave_lag && $master_lag -eq $replay_lag && $slave_lag -eq $replay_lag ]] ; then
        echo "OK: Stream: MASTER:$master_lag Slave:$slave_lag Replay:$replay_lag"
        exit $STATE_OK
    else
        if [[ $master_lag -eq $slave_lag ]] ; then
            if [[ $master_lag -ne $replay_lag ]] ; then
                if [ $(bc <<< $master_lag-$replay_lag) -lt $warning_limit ]; then
                    echo "OK: Stream: MASTER:$master_lag Replay:$replay_lag :: REPLAY BEHIND"
                    exit $STATE_OK
                else
                    echo "WARNING: Stream: MASTER:$master_lag Replay:$replay_lag :: REPLAY $(bc <<< $master_lag-$replay_lag)bytes BEHIND (${replay_timediff}seconds)"
                    exit $STATE_WARNING
                fi
            fi
        else
            if [ $(bc <<< $master_lag-$slave_lag) -gt $critical_limit ]; then
                echo "CRITICAL: Stream: MASTER:$master_lag Slave:$slave_lag :: STREAM BEYOND CRITICAL LIMIT ($(bc <<< $master_lag-$slave_lag)bytes)"
                exit $STATE_CRITICAL
            else
                if [ $(bc <<< $master_lag-$slave_lag) -lt $warning_limit ]; then
                    echo "OK: Stream: MASTER:$master_lag Slave:$slave_lag Replay:$replay_lag :: STREAM BEHIND"
                    exit $STATE_OK
                else
                    echo "WARNING: Stream: MASTER:$master_lag Slave:$slave_lag :: STREAM BEYOND WARNING LIMIT ($(bc <<< $master_lag-$replay_lag)bytes)"
                    exit $STATE_WARNING
                fi
            fi
        fi
        echo "UNKNOWN: Stream: MASTER: $master_lag Slave: $slave_lag Replay: $replay_lag"
        exit $STATE_UNKNOWN
    fi
fi

Possible outputs:

$ bash check_slave_replication.sh 192.168.0.1 5432 live 192.168.0.2 5432
WARNING: Stream: MASTER:1907958306184 Replay:1907878056888 :: REPLAY 80249296bytes BEHIND (00:03:14.056747seconds)
$ bash check_slave_replication.sh 192.168.0.1 5432 live 192.168.0.2 5432
OK: Stream: MASTER:2055690128376 Slave:2055690143144 Replay:2055690193744 :: STREAM BEHIND
$ bash check_slave_replication.sh 192.168.0.1 5432 live 192.168.0.2 5432
OK: Stream: MASTER:2055690497120 Replay:2055690497328 :: REPLAY BEHIND
$ bash check_slave_replication.sh 192.168.0.1 5432 live 192.168.0.2 5432
OK: Stream: MASTER:2055691704672 Slave:2055691704672 Replay:2055691704672