I'm working on a project which involves scraping from the major search engines (to be more specific - checking page rank and finding similar pages). With curl I'm calling the search engine, then with a single preg_match_all I'm getting all of the results in an array. I don't have any problems with Google and Bing, but when I wrote the script for Yahoo it worked, but it has a bug.
I'm testing with search for "autobedrijf" in the Netherlands. I have 100 results on the page, but I end up with only 74 of them in the result array. I copied the regex to this tool, inserted the Yahoo's page source and there it matches all the 100 results.
These are the first results from Yahoo:
but in the array I'm getting:
-ledensite
-Home Page
-Welkom bij autobedrijf Tolsma Exmorra
-etc.
Which means that after "Home Page" it skips 3 results. I tried to find any difference between results 5 and 6 (the last skipped and the next one), but I didn't find any reason not to match the 5th one.
This is the preg_match_all script:
$pattern = '@<div>\s*<h3>\s*<a[^<>]*\shref="([^*"]*\*\*)*([^<>]*)">
([^<>]*(<b>[^<>]*</b>)*[^<>]*(<wbr>)*[^<>]*)</a>\s*</h3>\s*</div>@siU';
preg_match_all($pattern, $result['EXE'], $matches);
The Yahoo page source for those two results looks like this:
<li><div class="res"><div><h3><a dirtyhref="http://nl.wrs.yahoo.com
/_ylt=A7x9Qb3rKqxN1XYA1PhzKAx.;_ylu=X3oDMTBydXF0bjc3BHNlYwN开发者_如何学PythonzcgR
wb3MDNQRjb2xvA2lyZAR2dGlkAw--/SIG=11n0lgcgp/EXP=1303157611/**http%3a
//www.autobedrijfgreijmans.nl/" class="yschttl spt"
href="http://nl.wrs.yahoo.com/_ylt=A7x9Qb3rKqxN1XYA1PhzKAx.;
_ylu=X3oDMTBydXF0bjc3BHNlYwNzcgRwb3MDNQRjb2xvA2lyZAR2dGlkAw--/SIG
=11n0lgcgp/EXP=1303157611/**http%3a//www.autobedrijfgreijmans.nl/">
<b>Autobedrijf</b> Greijmans Weert - Toyota <wbr>Specialist Occasions
APK</a></h3></div><div class="abstr">In het databestand van
<b>Autobedrijf</b> Greijmans vindt u zeer eenvoudig tweedehands auto's,
bedrijfswagens, klassiekers. Kijk snel op onze website</div><span
class="url">www.<b>autobedrijfgreijmans.nl</b></span> - <a
href="http://nl.wrs.yahoo.com/_ylt=A7x9Qb3rKqxN1XYA1fhzKAx./SIG=
186dh8afd/EXP=1303157611/**http%3a//74.6.239.67/search/cache%3fei=
UTF-8%26p=autobedrijf%26n=100%26va_vt=any%26vo_vt=any%26vp_vt
=any%26vst=0%26vf=all%26vm=p%26u=www.autobedrijfgreijmans.nl/
%26w=autobedrijf%26d=IxshPvbJWijU%26icp=1%26.intl=nl%26sig=yI6R7vJN31J
T92YKlVnT1g--">In de cache</a></div></li>
<li><div class="res"><div><h3><a dirtyhref="http://nl.wrs.yahoo.com
/_ylt=A7x9Qb3rKqxN1XYA1vhzKAx.;_ylu=X3oDMTBybWh0ZnN2BHNlYwNzcgRwb3MDNg
Rjb2xvA2lyZAR2dGlkAw--/SIG=11l17c21h/EXP=1303157611/**http%3a//www.
autobedrijf-tolsma.nl/" class="yschttl spt" href="http://www.autobedrijf
-tolsma.nl/">Welkom bij <b>autobedrijf</b> Tolsma Exmorra</a></h3></div>
<div class="abstr">Welkom op de vernieuwde website van <b>autobedrijf</b>
Tolsma Exmorra incl.digitale showroom <b>...</b> Welkom bij <b>autobedrijf
</b> tolsma. Op deze site nodigen wij U uit een <b>...</b></div><span
class="url">www.<b>autobedrijf-tolsma.nl</b></span> - <a href="http://nl.
wrs.yahoo.com/_ylt=A7x9Qb3rKqxN1XYA1_hzKAx./SIG=1840la9c5/EXP=1303157611/
**http%3a//74.6.239.67/search/cache%3fei=UTF-8%26p=autobedrijf%26n=100
%26va_vt=any%26vo_vt=any%26vp_vt=any%26vst=0%26vf=all%26vm=p%26u=
www.autobedrijf-tolsma.nl/%26w=autobedrijf%26d=YBZQIvbJWlDf%26icp=1%26.
intl=nl%26sig=zRU95PdBOTfII93dZ411ZA--">In de cache</a></div></li>
I'm stuck on that for more than 5 hours already and I can't figure out why the regex tools are matching both of the results, while preg_match_all
in PHP is matching only the second one.
If it matters - I'm testing on Windows with Apache and PHP 5.3.5.
Any advices are welcome, I can provide extra examples and testing code if you're interested.
An unpolished starting point:
<?php
$url = 'http://nl.search.yahoo.com/search?p=autobedrijf&toggle=1&cop=mss&ei=UTF-8&fr=yfp-t-732';
$html = file_get_contents($url);
$doc = new DOMDocument;
libxml_use_internal_errors(TRUE);
$doc->loadHTML($html);
libxml_use_internal_errors(FALSE);
$titles = $doc->getElementsByTagName('h3');
foreach($titles as $t){
echo $t->nodeValue . PHP_EOL;
}
?>
I recommend to use the Yahoo api. They offer very good services to reach results from yahoo with yahoo query language. http://developer.yahoo.com/yql/
example:
http://developer.yahoo.com/yql/console/#h=select%20*%20from%20search.web%20where%20query%3D%22autobedrijf%22
精彩评论