Call: (480) 382-8464

Perl LWP UserAgent Example: Get a http or ftp data

Perl LWP UserAgent is an amazing tool for automating web queries. Here is the simple text to get HTTP or FTP data using the Perl LWP UserAgent:


# pretty standard to have this at least in all scripts
# at least for development.  Use taint also for CGI.
use warnings;
use strict;

use LWP::UserAgent;
use CGI qw(header -no_debug);

# if we don't do this it will buffer and then print results
# I want to see results immediately as it happens.
# not every server configuration supports this

# put a page that requires authentication here to see a 401 error
# put a normal URL here to see it pull text normally.
my $URL = '';

# create a new object and request the GET method on $URL 
my $res = LWP::UserAgent->new->request(new HTTP::Request GET => $URL);

# required for any type of CGI
print header;

# at this point you can either use the print statement,
# or assign the data and manipulate it
# you can can also just print the text:
# print $res->is_success ? $res->cotnent : $res->status_line;

# but lets assume we want to manipulate further before printing:
my $data = $res->is_success ? $res->content : $res->status_line;

# simple regex to grab a pattern match from the page:
if ($data =~ m/The quick brown(.*)jumped over the lazy dog/) {
  print "we found a ' $1 ' (should be ' fox ')";
} else {
  print "probably not a typing tutorial!\n";

# What did the full data contain?
print $data;

# not required, but a good habit.

Leave a Reply

Your email address will not be published. Required fields are marked *

Main Offers/Services

Hosting and website related services offered by Digital Crunch:

Managed VPS Hosting

Server Updates, Module/Software Installs, Firewall, Security, Hosting Management, Wordpress - let us manage it for you

Hosted Email

Setup and Maintain Business Class, Secure Email Hosting including spam protection and 25GB of storage space

Linux Consulting

Server Updates, Module/Software Installs, Firewall, Security, Hosting, Wordpress and Apps, anything related to Linux

Email Marketing Services

We broadcast your content to your list, capture leads for you, follow up on schedules, make more sales for you

Website Maintenance

Content updates, menu changes, graphics changes, adding content, getting links, building traffic, building sales funnels

Hosting Tutorials

Tutorials we've written for other customers that like to get their hands dirty and learn about hosting