Skip to content
Johnny Morano's Tech Articles

Johnny Morano's Tech Articles

Ramblings of an old-fashioned space cowboy

Menu
  • About
  • Privacy Policy
Menu

Retrieve a file from an authenticated website (in Perl)

Posted on February 9, 2011 by insaniac

At my daily job as a Perl developer, I was asked to write a Perl script which would download a backup file from a certain web interface. The file however, was in a secured part of the web site. This means, that the script would first have to authenticate against the web site and then afterwards retrieve the requested file.

Thanks to the great LWP Perl module, this is again a rather easy and simple job.

Let’s start by explaining the script step-by-step:

  1. Call your modules
    #!/usr/bin/perl
    use strict; use warnings;
    
    use LWP;
    use HTTP::Cookies;
    use HTTP::Request;
    use File::Path qw{mkpath};
    

    The modules HTTP::Cookies and HTTP::Request are required to create the cookie jar file and to make a HTTP request (duh!). These two modules will create the objects that are required for the LWP user-agent object we are about to create.

  2. Check your temporary path:
    my $HOME = $ENV{HOME} . '/tmp';
    mkpath $HOME   unless -d $HOME;
    

    We will need this location to save our cookie jar file and the file we want to download

  3. Create a cookie jar and a LWP user-agent:
    # create a cookie jar on disk
    my $cookies = HTTP::Cookies->new(
            file     => $HOME.'/cookies.txt',
            autosave => 1,
            );
    
    # create an user-agent and assign the cookie jar to it
    my $http = LWP::UserAgent->new();
    $http->cookie_jar($cookies);
    

    No rocket science here, just create the objects!

  4. Send the authentication request:
    # try to log in
    my $login = $http->post(
            'https://www.example.com/auth/login.pl', [
                username => 'user',
                password => 'secret', ]
            );
    

    Once authenticated, the web site will create a cookie containing a session ID. This cookie will allow us to download the file we need from the secured part of the web site.

From this point on, it is required to check whether the log in succeeded and then continue with downloading the file. If the log in failed, an error message should be displayed with the reason on the failure.

# check if log in succeeded
if($login->is_success){
    print "- logged in successfullyn";
    print "- requesting file, might take a whilen";

    # make request to download the file
    my $url       = 'https://www.example.com/auth/files/backup.zip';
    my $file_req  = HTTP::Request->new('GET', $url);
    my $get_file  = $http->request( $file_req );

    # check request status
    if($get_file->is_success){
        print "--> downloaded $url, saving it to filen";

        # save the file content to disk
        open my $fh, '>', $HOME.'/backup.zip' 
                                         or die "ERROR: $!n";
        print $fh $get_file->decoded_content;
        close $fh;

        print "nsaved file:n";
        print "-------------n";
        print "filename:  ".$get_file->filename."n";
        print "size:      ".(-s $HOME.'/backup.zip')."n";
    }
    else {
        die "ERROR: download of $url failed: " . $get_file->status_line . "n";
    }
}
else {
    die "ERROR: login failed: " . $login->status_line . "n";
}

2 thoughts on “Retrieve a file from an authenticated website (in Perl)”

  1. gautham says:
    September 26, 2012 at 21:50

    hi i have a question , i have to go in a website daily go to manually to inside the website and download a file can yuo get me ideas

    Reply
  2. N-Cen-Dainamix says:
    April 11, 2015 at 16:35

    hi i have script download file in a website follow me ..

    #!/usr/bin/perl

    #####
    # Code Perl Download file
    # BY: N-Cen-Dainamix
    #####

    use LWP;
    use LWP::Simple; # socket for download

    my $url = “http://filedownload.rar”; # url
    my $name = “file.rar”; #name

    print ” Download .. \n”;
    getstore($url,$name);
    print ” Download finish .. \n”;

    # end code

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Recent Posts

  • Use multiple Azure subscriptions in Terraform modules
  • Read the HAProxy UNIX socket file using Perl
  • A Prometheus Exporter framework written in Perl
  • Managing LDAP passwords with Perl
  • Libvirt guest startup issue with AppArmor
  • Deploy a PostgreSQL database with an initial schema using Ansible
  • Using Ansible to finalize Hashicorp Packer images

Categories

  • Automation (8)
  • Blog (60)
  • Database (4)
  • Development (37)
  • Linux (26)
  • Mac OS X (5)
  • Media (2)
  • OpenBSD (3)
  • Perl (34)
  • Photo (2)
  • PostgreSQL (4)
  • Terraform (5)
  • Web (11)

Tags

Ajax (3) Android (1) Ansible (2) API (5) AppArmor (1) Automation (5) Azure (3) azurerm (2) Bash (4) Cloud (2) CPAN (4) CSS (1) Debian (4) Dev (35) DevOps (11) EXIF (1) Facebook (1) Geotag (1) GMail (1) Google (3) Hack (2) Hashicorp (4) Hetzner (2) HTML (4) IMAP (2) IPTables (6) JavaScript (4) Libvirt (2) Linux (25) Logging (2) MacOSX (5) Media (2) Monitoring (6) MySQL (3) OpenBSD (4) Packer (1) Perl (35) PF (2) Postgresql (6) Security (7) SysAdmin (24) Terraform (4) Ubuntu (2) UNIX (9) Web 2.0 (3)

Archive

  • April 2022 (10)
  • March 2022 (6)
  • December 2016 (1)
  • March 2016 (1)
  • November 2015 (1)
  • November 2014 (1)
  • August 2014 (1)
  • May 2014 (1)
  • February 2014 (2)
  • December 2013 (1)
  • October 2013 (2)
  • September 2013 (2)
  • August 2013 (2)
  • October 2012 (1)
  • August 2012 (4)
  • March 2012 (3)
  • July 2011 (1)
  • June 2011 (2)
  • April 2011 (3)
  • March 2011 (4)
  • February 2011 (2)
  • December 2010 (2)
  • October 2010 (4)
  • September 2010 (1)
  • August 2010 (5)

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

Footer

  • Shihai Corp
  • My Photo website
© 2022 Johnny Morano's Tech Articles | Powered by Superbs Personal Blog theme
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Do not sell my personal information.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT