The Apache configuration in the below example will actually require 2 web servers:

• one proxy host, which will accept the SSL connection, verify, check for ACLs and then forward the connection unencrypted internally
• one internal web server which will actually contain the WebAPI scripts

This article explains how to use Mojolicious for the WebAPI side and LWP::UserAgent to send and receive the WebAPI calls. We will furthermore use JSON to send and receive information.

First we need to have or create a set of OpenSSL certificates.
The below example uses self signed certificates, since they don’t cost any money and suit perfect for the purpose of this example.
There a million howto’s on the internet which explains these steps very thoroughly, so I won’t reinvent the wheel. I’m just going to post the steps I took to create:

• a CA certificate
• a client certificate
• a server certificate

cd /path/to/SSL
cp /etc/ssl/openssl.cnf example.cnf
vim example.cnf  # Edit the file to your needs
openssl genrsa -aes256 -out private/example_com_ca.key 4096
openssl req -config example.cnf -new -x509 -extensions v3_ca -key private/example_com_ca.key -out certs/example_com_ca.crt -days 3650
openssl req -config example.cnf -new -nodes -keyout private/client01.key -out client01.csr -days 365
openssl ca -config example.cnf -policy policy_anything -out certs/client01.crt -infiles client01.csr
openssl req -config example.cnf -new -nodes -keyout private/server.key -out server.csr -days 365
openssl ca -config example.cnf -policy policy_anything -out certs/server.crt -infiles server.csr

Next we will need to configure our web server (this example uses the Apache web server) in order to use our self signed certificates, and to proxy forward our WebAPI calls.

SSLEngine on
SSLCertificateFile       /path/to/SSL/certs/server.crt
SSLCertificateKeyFile    /path/to/SSL/private/server.key
SSLCertificateChainFile  /path/to/SSL/certs/example_com_ca.crt
SSLCACertificateFile     /path/to/SSL/certs/example_com_ca.crt
SSLVerifyClient require

ProxyPass /send/         http://internal-host/send.pl/
ProxyPassReverse /send/  http://internal-host/send.pl/

<Proxy *>
            Options FollowSymLinks MultiViews
            AllowOverride All
            Order deny,allow
            allow from localhost
            allow from 8.8.8.8 # The client IP address
            deny from all
</Proxy>

The above the configuration for the external proxy server. The internal web server should have a pretty straight-forward configuration:

• a cgi-handler for the Perl extension ‘.pl’

I could have also send those proxy requests to an internal Mojolicious application, listening on a specific port. I’ll leave that for another article.

The test client script is going to make a SSL connection to the external web server, send some JSON and wait for the server to send some JSON data back. The interesting part in the below script is how to set up the SSL connection.

#!/usr/bin/perl
use strict; use warnings;

use HTTP::Request;
use LWP::UserAgent;
use IO::Socket::SSL;
use JSON;

my $data = {
    username  => 'skipper',
    password  => 'secret',
    variable  => 'value',
};

my $uri = 'https://example.com/send/event';
my $json = encode_json( $data );
my $req = HTTP::Request->new( 'POST', $uri );
$req->header( 'Content-Type' => 'application/json' );
$req->content( $json );
 
my $lwp = LWP::UserAgent->new(
    ssl_opts => {
        SSL_use_cert    => 1,
        SSL_version     => 'TLSv12',
        SSL_verify_mode => SSL_VERIFY_PEER,
        SSL_ca_file     => "/path/to/SSL/certs/example_com_ca.crt",
        SSL_cert_file   => "/path/to/SSL/certs/client01.crt",
        SSL_key_file    => "/path/to/SSL/private/client01.key",
    },
) or die "SSL Connection failed: $!";
my $res = $lwp->request( $req );
if ($res->is_success) {
    print "RESPONSE:", $res->content . "\n";
} 
else {
    print "ERROR: ", $res->status_line . "\n";
}

The server example uses the Mojolicious frame work. Mojolicious is the porn for every Perl WebAPI developer. If you don’t know it, you should be ashamed and start reading about it right away.

#!/usr/bin/env perl
 
use Mojolicious::Lite;

# A helper to identify visitors
helper whois => sub {
    my $c               = shift;
    my $headers         = $c->req->headers;
    my $agent           = $c->req->headers->user_agent || 'Anonymous';
    my $local_ip        = $c->tx->remote_address;
    my $remote_ip       = $headers->header('x-forwarded-for');

    return { 
        agent      => $agent, 
        local_ip   => $local_ip,
        remote_ip  => $remote_ip,
   };
};

any '/' => sub {
  my $c = shift;
  $c->render( text => "There is nothing to see here, move along" );
};
 
post '/event' => sub {
    my $c = shift;
    my $json = $c->req->json;
    my $data = {
        username        => $json->{username},
        password        => $json->{password},
        whois           => $c->whois,
    };
    $c->render( json => $data );
};



### IMPORTANT
app->secret('some_cool_secret');
app->start;

Example output of the test client script:

$ perl test_send.pl 
RESPONSE:
{"whois":{"remote_ip":"176.9.64.17","agent":"libwww-perl\/6.04","local_ip":"176.9.64.17"},"password":"secret","username":"skipper"}

• sorted
• searched
• paginated
• scroll infinitely
• themed
• …

And most important: it’s for free! I’ve always wanted to create an infinite scrolling table and now it’s just too easy:

$(document).ready( function() {
    oTable = $('#ip_data').dataTable( {
        "bProcessing":     true,
        "bServerSide":     true,
        "bPaginate":       true,  
        "bScrollInfinite": true,
        "bScrollCollapse": true,
        "sScrollY":        "200px",
        "sAjaxSource":     "get_ip_data.pl",
    } );
} );

And that’s it! Well, ok, you need to include the Javascript file and CSS files of the Datatables Project of course and you need to create the table in HTML.

For instance:

And then you will need a Perl script providing you the data for the table.
The below example allows to

• search the tables
• scroll infinitely
• sort on the columns

It also supplies the Datatables table with a total amount or rows in the database table.

This following script will be saved as ”get_ip_data.pl”

#!/usr/bin/perl
use strict; use warnings;
use DBI;
use JSON;
use CGI;

my @columns = qw/ip country_name city latitude longitude/;

my $q = CGI->new;
my $db = DBI->connect("dbi:mysql:host=localhost;db=testdb", 'testuser', 'xxxsecret');

my $params = $q->Vars;

# Get the total count of rows in the table
my $sql_count = "select count(id) from geo_data";
my $count = $db->selectrow_arrayref($sql_count)->[0];

# Start building up the database query
my @values;
my $sql = "select ip,country_name,city,latitude,longitude from geo_data";

# if a search parameter was supplied in the AJAX call, build the WHERE part in the SQL statement
if( $params->{sSearch} ){
    $sql .= ' WHERE ';
    $sql .= 'ip LIKE ? OR country_name LIKE ? or city LIKE ? or latitude LIKE ? or longitude LIKE ?';
    push @values, ('%'.$params->{sSearch}.'%','%'.$params->{sSearch}.'%','%'.$params->{sSearch}.'%','%'.$params->{sSearch}.'%','%'.$params->{sSearch}.'%');
}

# if a sorting parameter was supplied in the AJAX call, build up the ORDER BY part in the SQL statement
if( $params->{iSortingCols} ){
    $sql .= ' ORDER BY';
    foreach my $c (0 .. ( $params->{iSortingCols} -1 )){
        $sql .= ' ' . $columns[ $params->{"iSortCol_$c"} ] . ' ' . $params->{"sSortDir_$c"};
        $sql .= ','
    }
    $sql =~ s/,$//;
}

# Limit the output and also allow to paginate or scroll infinitely
$sql .= " LIMIT ? OFFSET ?";
push @values, (($params->{iDisplayLength} > 0 ? $params->{iDisplayLength} : 25), ( $params->{iDisplayStart} // 0));

# Fetch the data from the database
my $data = $db->selectall_arrayref($sql, { Slice => [] }, @values);

# Return the JSON object
print $q->header('application/json');
my $json = encode_json({ aaData => $data, iTotalRecords => $count, iTotalDisplayRecords => $count, sEcho => int($params->{sEcho}) });
print $json;

An example can be found overhere: http://www.moretrix.com/~insaniac/map/map.pl

Before we look at the Google API for GeoChart, let’s first set up a script which will get data out of a database and return it in a JSON formatted object.
In this example we will use Perl and three Perl modules:

• DBI
• JSON
• CGI

When converting database values to a JSON object (or text string), it is very important that all data is properly type-casted.
In the following example you will see that we do:

$_->[1] = int($_->[1]) foreach @$data;

This snippet will actually make our INTEGER value into a real integer. The DBI module had returned it as a normal string (that’s just how DBI works).

#!/usr/bin/perl
use strict; use warnings;
use DBI;
use JSON;
use CGI;

my $q = CGI->new;
my $db = DBI->connect("dbi:mysql:host=localhost;db=testdb", 'testuser', 'xxxsecret');

my $sql = "SELECT country_name, count(id) as total from geo_data group by country_name";
my $data = $db->selectall_arrayref($sql);
$_->[1] = int($_->[1]) foreach @$data;
unshift(@$data, ['Country', 'Attacks']);

print $q->header('application/json');
my $json = encode_json($data);
print $json;

The above Perl script will be saved as ”get_countries_data.pl”.

In the below Javascript example, we will use the Google API for the GeoChart Map and jQuery for making the AJAX call to our Perl script. Since the Perl script already provides the data in a JSON format, we do not need to convert or parse it.
Furthermore the Javascript code is pretty straightforward and based on the example found at https://developers.google.com/chart/interactive/docs/gallery/geochart, except for the AJAX part.

    google.load('visualization', '1', {packages: ['geochart']});
    google.setOnLoadCallback(drawVisualization);

    function drawVisualization() {
        var options = {
            height: '500',
            width: '1200',
            colorAxis: {minValue: 0,  colors: ['#FFC26B', 
                                               '#FFAF3B', 
                                               '#FF9700', 
                                               '#C1852F', 
                                               '#A86400']},
            datalessRegionColor: '#FAFAFA',
            backgroundColor: '#F4EFE7',
        };
    
        $.ajax({
            type: 'POST',
            url: "get_countries_data.pl",
            dataType: "json",
            async: false,
            success: function(json_data) {
                var data = google.visualization.arrayToDataTable(json_data);
                var chart = new google.visualization.GeoChart(
                                   document.getElementById('visualization') );

                chart.draw(data, options);
            }
        });
    }

An example of this setup can be found at http://www.moretrix.com/~insaniac/map/map.pl

I’m going to use to beef up my image browser which I quickly wrote to simply the contents of directory full with images. (can be found overhere http://yamamoto.moretrix.com/~insaniac/grtp/

Here’s the link to the article.

Nowadays writing programs for tasks concerning Web 2.0 alike websites, is rather simple and easy. The only thing you will have to do, is learn their API. These API are mostly XML based requests and replies.

Most programming languages then write wrapper classes that call and access those API’s, like was done in Perl.
Thanks to the following modules, which can be found on CPAN, I wrote a small script which loads my contacts on Facebook and then updates or creates my contacts in GMail.

The script will retrieve some contact information, like the birthday, email addresses, status messages, and website information from Facebook as well as the profile picture of the contact. Then it will search your GMail Contact list for the user and will update the contact if it is found. Non-existing contacts will be ignored.

The modules used in the script are:

use WWW::Facebook::API;
use WWW::Google::Contacts;
use HTTP::Request;
use LWP;

All these modules can be downloaded from CPAN.

You will need to create a Facebook application, or you can use the application I have added. Before you can use the Facebook API, your application needs to be registered and known to Facebook. If you are not sure how to create and register an application at Facebook, please use mine. (No worries, I won’t do evil stuff)

After the module have been called in the script, we need to initialize some variables:

my $TMP    = $ENV{HOME}.'/tmp';
my %MONTHS = (
        January => '01',      February => '02',     March     => '03',
        April   => '04',      May      => '05',     June      => '06',
        July    => '07',      August   => '08',     September => '09',
        October => '10',      November => '11',     December  => '12',
);

my $facebook_api      = 'dad7ea6af99ba4ee11d6007f0a27cc6a';
my $facebook_secret   = '19d45131916886ffdad8d0a6899d6794';
my $facebook_clientid = '141446449211973';
my $facebook_browser  = '/usr/bin/firefox';
my $gmail_user        = 'user@gmail.com';
my $gmail_password    = 'secret';

• $TMP = we need to save the profile pics from Facebook and GMail somewhere on disk
• %MONTHS = conversion table for updating birthdates on GMail
• $facebook_ variables = numbers you get when you register an application on Facebook
• $gmail_ variables = obviously you will need a GMail account to update your contacts

my $client = WWW::Facebook::API->new(
    desktop         => 1,
    api_version     => '1.0',
    api_key         => $facebook_api,
    secret          => $facebook_secret,
);
$client->app_id($facebook_clientid);

local $SIG{INT} = sub {
    print "Logging out of Facebookn";
    my $r = $client->auth->logout;
    exit(1);
};

my $token = $client->auth->login(browser => $facebook_browser);
$client->auth->get_session($token);

my $google = WWW::Google::Contacts->new( username => $gmail_user, password => $gmail_password );
my $http   = LWP::UserAgent->new();

In the above part a Facebook object is created and a Facebook session will be created. If this is the first run that the program runs, it will ask on a Facebook page in your browser, to allow this application. Once allowed, the Facebook page can be closed.

The script will then also create a GMail object and a LWP object. The LWP object is needed to download the profile picture from the Facebook friend page.

In the next part, the friends list will be downloaded from Facebook, binded with the fields of information we are searching for.

print "About to get friends from Facebook...n";
my $friends_info = $client->users->get_info(
        uids   => $client->friends->get, 
        fields => [ qw/name first_name last_name status pic_big birthday email website about_me/ ] 
);

foreach my $friend (@{$friends_info}){

Ok, now we are ready to start taking the information from Facebook and updating it in GMail. The next part is the full content of the foreach loop, that was started above.

    # search google contact
    print "Searching for $friend->{name}...n";
    my @contacts = $google->contacts->search({full_name => $friend->{name}});
    if(not scalar @contacts){
        print "- $friend->{name} not found in Gmail Contacts Listn";
        next;
    }

    my $contact = pop @contacts;
    next unless defined $contact;

    if( defined $friend->{birthday} and $friend->{birthday} ne ''){
        my ($month, $day, $year) = ($friend->{birthday} =~ /^(D+) (d+)(?:, (d+))/);
        if(defined $year and defined $month and defined $day){
            $day   = sprintf '%02d', $day;
            $contact->birthday("${year}-$MONTHS{$month}-$day");
        }
    }

    if( defined $friend->{about_me} and $friend->{about_me} ne ''){
        $contact->notes($friend->{about_me});
    }

    if( defined $friend->{website} and $friend->{website} ne ''){
        my $websites = $contact->website;
        $websites ||= [];
        $contact->website($friend->{website}, @$websites) 
                        unless grep /Q$friend->{website}E/, @$websites;
    }

    if( defined $friend->{email} and $friend->{email} ne ''){
        my $emails = $contact->email;
        $emails ||= [];
        $contact->email($friend->{email}, @$emails) 
                        unless grep /Q$friend->{email}E/, @$emails;
    }

    if( defined $friend->{status}->{message} and $friend->{status}->{message} ne ''){
        my $jots = $contact->jot;
        $jots ||= [];
        $contact->jot($friend->{status}->{message}, @$jots)
                        unless grep /Q$friend->{status}->{message}E/, @$jots;
    }

    eval {
        $contact->update();
        print "- Contact information updatedn";
    };
    print "- Updating ".$contact->full_name." failed: $@n" if $@;

    # First update the contact information and then update the profile photo
    if( defined $friend->{pic_big} and $friend->{pic_big} ne ''){
        my $req = HTTP::Request->new('GET', $friend->{pic_big});
        my $file = $http->request($req);

        if($file->is_success){
            my $fpic = $TMP.'/fpic'.$friend->{uid}.'.jpg';
            my $gpic = $TMP.'/gpic'.$friend->{uid}.'.jpg';

            # save the Facebook profile pic to disk
            open my $fh, '>', $fpic or die "ERROR: $!n";
            print $fh $file->decoded_content;
            close $fh;

            # create a backup of the existing profile pic and then
            # delete it from Gmail Contacts
            if ( $contact->photo->exists ) {
                eval {
                    $contact->photo->to_file($gpic);
                };
                print "- Save of GMail profile pic failed: $@n" if $@;
            }

            # if a Facebook profile pic was downloaded, now upload it to 
            # Gmail Contacts
            if ( -f $fpic ) {
                eval {
                    $contact->photo->from_file($fpic);
                    $contact->photo->update;
                    print "- Profile pic ".$contact->full_name." updatedn";
                };
                print "- Failed to update profile pic: $@n" if $@;
            }
        }
        else {
            print "Failed to download profile picture of ". $friend->{name} ."n";
        }
    }

Set Permissions on Facebook:
The Facebook application needs more permissions than the basic permissions which are granted the first time you use the application. Copy paste the following URL into browser and you will presented a Facebook to allow or disallow the extended permissions for this application:

http://www.facebook.com/dialog/oauth?client_id=141446449211973&redirect_uri=https://jmorano.moretrix.com&scope=email,read_stream,user_birthday,friends_birthday,user_website,friends_website,export_stream,friends_online_presence,friends_status,sms,user_status,friends_about_me,friends_hometown,friends_location,publish_stream,read_stream,status_update

References:

• http://developers.facebook.com/docs/reference/api/user/
• http://developers.facebook.com/docs/authentication/
• http://search.cpan.org/~merixzon/WWW-Google-Contacts-0.28/lib/WWW/Google/Contacts.pm
• http://search.cpan.org/~unobe/WWW-Facebook-API-0.4.18/lib/WWW/Facebook/API.pm