Scraping websites is usually pretty boring and annoying, but for some reason it always comes back. Tatsuhiko Miyagawa comes to the rescue! His Web::Scraper makes scraping the web easy and fast.
Since the documentation is scarce (there are the POD and the slides of a presentation I missed), I'll post this blog entry in which I'll show how to effectively scrape Yahoo! Search.
First we'll define what we want to see. We'll going to run a query for 'Perl'. From that page, we want to fetch the following things:
title (the linked text)
url (the actual link)
description (the text beneath the link)
So let's start our first little script:
use Data::Dumper;
use URI;
use Web::Scraper;
my $yahoo = scraper {
process "a.yschttl", 'title' => 'TEXT', 'url' => '@href';
process "div.yschabstr", 'description' => "TEXT";
result 'description', 'title', 'url';
};
print Dumper $yahoo->scrape(URI->new("http://search.yahoo.com/search?p=Perl"));
Now what happens here? The important stuff can be found in the process statements. Basically, you may translate those lines to "Fetch an A-element with the CSS class named 'yschttl' and put the text in 'title', and the href value in url. Then fetch the text of the div with the class named 'yschabstr' and put that in description.
The result looks something like this:
$VAR1 = {
'url' => 'http://www.perl.com/',
'title' => 'Perl.com',
'description' => 'Central resource for Perl developers. It contains
the Perl Language, edited by Tom Christiansen, and the Perl Reference, edited
by Clay Irving.'
};
Fun and a good start, but hey, do we really get only one result for a query on 'Perl'? No way! We need a loop!
The slides tell you to append '[]' to the key, to enable looping. The process lines then look like this:
process "a.yschttl", 'title[]' => 'TEXT', 'url[]' => '@href';
process "div.yschabstr", 'description[]' => "TEXT";
And when we run it now, the result looks like this:
$VAR1 = {
'url' => [
'http://www.perl.com/',
'http://www.perl.org/',
'http://www.perl.com/download.csp',
...
],
'title' => [
'Perl.com',
'Perl Mongers',
'Getting Perl',
...
],
'description' => [
'Central resource for Perl developers. It contains
the Perl Language, edited by Tom Christiansen, and the Perl Reference, edited by
Clay Irving.',
'Nonprofit organization, established to support the
Perl community.',
'Instructions on downloading a Perl interpreter for
your computer platform. ... On CPAN, you will find Perl source in the /src
directory. ...',
...
]
};
That looks a lot better! We now get all the search results and could loop through the different arrays to get the right title with the right url. But still we shouldn't be satisfied, for we don't want three arrays, we want one array of hashes! For that we need a little trickery; we need another process line! All the stuff we grab already is located in a big ordered list (the OL-element), so let's find that one first, and for each list element (LI) find our title,url and description. For this we don't use the CSS selectors, but we'll go for the XPath selectors (heck, we can do both, so why not?).
To grab an XPath I really suggest firebug , a FireFox addon. With the easy point and click interface, you can grab the path within seconds.
use Data::Dumper;
use URI;
use Web::Scraper;
my $yahoo = scraper {
process "/html/body/div[5]/div/div/div[2]/ol/li", 'results[]' => scraper {
process "a.yschttl", 'title' => 'TEXT', 'url' => '@href';
process "div.yschabstr", 'description' => "TEXT";
result 'description', 'title', 'url';
};
result 'results';
};
print Dumper $yahoo->scrape( URI->new("http://search.yahoo.com/search?p=Perl") );
You see that we switched our title, url and description fields back to the old notation (without []), for we don't want to loop those fields. We've moved the looping a step higher, being to the li-elements. Then we open another scraper which will dump the hashes into the results array (note the '[]' in 'results[]').
The result is exactly what we wanted:
$VAR1 = [
{
'url' => 'http://www.perl.com/',
'title' => 'Perl.com',
'description' => 'Central resource for Perl developers. It
contains the Perl Language, edited by Tom Christiansen, and the Perl Reference,
edited by Clay Irving.'
},
{
'url' => 'http://www.perl.org/',
'title' => 'Perl Mongers',
'description' => 'Nonprofit organization, established to support
the Perl community.'
},
{
'url' => 'http://www.perl.com/download.csp',
'title' => 'Getting Perl',
'description' => 'Instructions on downloading a Perl interpreter
for your computer platform. ... On CPAN, you will find Perl source in the /src
directory. ...'
},
...
];
Again Tatsuhiko impresses me with a Perl module. Well done! Very well done!
Source: http://menno.b10m.net/blog/blosxom.cgi/perl/scraping-yahoo-search-with-web-scraper.html
