Using PHP CURL Library To Scrape The Internet
Posted: Sat Jan 01, 2005 12:55 am
Using PHP CURL Library To Scrape The Internet
By Bobby Handzhiev
Have you ever though how much information is there in DMOZ? Your entire life won't be enough to collect and sort it.
Well, we had to do part of that. P.I.M. Team Bulgaria was involved in scraping the technology directories of DMOZ, google, yahoo and many more. We had a request to scrape several technology directories, to map them in a master structure, to get all the company infos in the directories and to get the URLs of all these companies. We found this task amazing!
At the beginning
The first thing you need to know when you have to scrape the net is to know how to do it :-)
There are various technologies, but the most important is to know the basis of the process:
- screen scrape
- parse the input
- sort and fulfill the output
- save the results
Screen scrape
This is a process in which you get the content of any website thru a script. One good grabber script should be able to get the content of any site regardless if it is static HTML or contains dynamic generated pages. In P.I.M. we are working mostly with PHP so this was our choice. PHP has a great supporting library called CURL ("Client URL Library") which allow us to do that.
We created a simple grabber class which has a constructor doing to the scraping job and few methods for parsing the result:
function Grabber($url)
{
$this->content="";
$ch = curl_init ();
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
$this->content = curl_exec ($ch);
curl_close ($ch);
}
The input we receive when calling the grabber is the HTML (static or generated) of the page.
Parse the input
Can you imagine that each web page has its own soul? Some pages are coded 'by hand' by their authors and all of them has different style. Some are generated with a software which, guess what, has also different style. So the deal with parsing the output is to find specifics on the page which to help us get out the scrap and exctract the usefull information. Lets get, for example, Alexa (www.alexa.com). We have to extract the technology directories only so we pointed our grabber to [url=\"http://www.alexa.com/browse/categories?catid=4\"]http://www.alexa.com/browse/categories?catid=4[/url]. The result we got had header and footer, which we did not needed, so we had to take it off. Easy: we just removed everything which is not between 2 HTML sctrings on the page: "<span class="bodyBold" Browse"> and the unique string "Languages available for this subjec" near the end.
Thus we have the core. What to do with it? Hey, that's easy! Break all of it on rows:
$rows=explode("\n",$grabber->searchtxt);
... then go thru rows:
foreach($rows as $row)
{
if(!$row) continue;
if(!strstr($row,'href')) continue;
if(strstr($row,">b< ")) continue;
//get name
$name=cut("\">","",$row);
$name=str_replace("\n","",$name);
$href=cut("href=\"","\">",$row);
$href=trim($href);
$name=htmlentities($name);
$href=strip_tags($href);
$name=trim($name);
if(!empty($name)&&!empty($href))
{
array_push($dirs,array( 'name'=>$name,'href'=>$href));
}
}
Why we do this? Well, we noticed that all the rows we need contain 'href' and DO NOT contain any bold tags. So we got rid off all rows which did not meet our requirements. Then we had to parse a little the rest of the rows. using our function 'cut()' that was very easy. Thus we got all the directories on the first level! But there are many levels, now what? Well, we had the URLs on each directory. And guess all the directories, of course, have similar page structure! So we had only to go in several cycles and to get all of. The entire Alexa was in our hands!
Read the Rest of the Article.
About the Author:
Bobby Handzhiev is a senior developer in PIM Team Bulgaria
[url=\"http://pimteam.net\"]http://pimteam.net[/url]
By Bobby Handzhiev
Have you ever though how much information is there in DMOZ? Your entire life won't be enough to collect and sort it.
Well, we had to do part of that. P.I.M. Team Bulgaria was involved in scraping the technology directories of DMOZ, google, yahoo and many more. We had a request to scrape several technology directories, to map them in a master structure, to get all the company infos in the directories and to get the URLs of all these companies. We found this task amazing!
At the beginning
The first thing you need to know when you have to scrape the net is to know how to do it :-)
There are various technologies, but the most important is to know the basis of the process:
- screen scrape
- parse the input
- sort and fulfill the output
- save the results
Screen scrape
This is a process in which you get the content of any website thru a script. One good grabber script should be able to get the content of any site regardless if it is static HTML or contains dynamic generated pages. In P.I.M. we are working mostly with PHP so this was our choice. PHP has a great supporting library called CURL ("Client URL Library") which allow us to do that.
We created a simple grabber class which has a constructor doing to the scraping job and few methods for parsing the result:
function Grabber($url)
{
$this->content="";
$ch = curl_init ();
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
$this->content = curl_exec ($ch);
curl_close ($ch);
}
The input we receive when calling the grabber is the HTML (static or generated) of the page.
Parse the input
Can you imagine that each web page has its own soul? Some pages are coded 'by hand' by their authors and all of them has different style. Some are generated with a software which, guess what, has also different style. So the deal with parsing the output is to find specifics on the page which to help us get out the scrap and exctract the usefull information. Lets get, for example, Alexa (www.alexa.com). We have to extract the technology directories only so we pointed our grabber to [url=\"http://www.alexa.com/browse/categories?catid=4\"]http://www.alexa.com/browse/categories?catid=4[/url]. The result we got had header and footer, which we did not needed, so we had to take it off. Easy: we just removed everything which is not between 2 HTML sctrings on the page: "<span class="bodyBold" Browse"> and the unique string "Languages available for this subjec" near the end.
Thus we have the core. What to do with it? Hey, that's easy! Break all of it on rows:
$rows=explode("\n",$grabber->searchtxt);
... then go thru rows:
foreach($rows as $row)
{
if(!$row) continue;
if(!strstr($row,'href')) continue;
if(strstr($row,">b< ")) continue;
//get name
$name=cut("\">","",$row);
$name=str_replace("\n","",$name);
$href=cut("href=\"","\">",$row);
$href=trim($href);
$name=htmlentities($name);
$href=strip_tags($href);
$name=trim($name);
if(!empty($name)&&!empty($href))
{
array_push($dirs,array( 'name'=>$name,'href'=>$href));
}
}
Why we do this? Well, we noticed that all the rows we need contain 'href' and DO NOT contain any bold tags. So we got rid off all rows which did not meet our requirements. Then we had to parse a little the rest of the rows. using our function 'cut()' that was very easy. Thus we got all the directories on the first level! But there are many levels, now what? Well, we had the URLs on each directory. And guess all the directories, of course, have similar page structure! So we had only to go in several cycles and to get all of. The entire Alexa was in our hands!
Read the Rest of the Article.
About the Author:
Bobby Handzhiev is a senior developer in PIM Team Bulgaria
[url=\"http://pimteam.net\"]http://pimteam.net[/url]