开发者

PHP preg_match_all matches 74 out of 100 matches

开发者 https://www.devze.com 2023-02-26 22:58 出处:网络
I\'m working on a project which involves scraping from the major search engines (to be more specific - checking page rank and finding similar pages). With curl I\'m calling the search engine, then wit

I'm working on a project which involves scraping from the major search engines (to be more specific - checking page rank and finding similar pages). With curl I'm calling the search engine, then with a single preg_match_all I'm getting all of the results in an array. I don't have any problems with Google and Bing, but when I wrote the script for Yahoo it worked, but it has a bug.

I'm testing with search for "autobedrijf" in the Netherlands. I have 100 results on the page, but I end up with only 74 of them in the result array. I copied the regex to this tool, inserted the Yahoo's page source and there it matches all the 100 results.

These are the first results from Yahoo:

PHP preg_match_all matches 74 out of 100 matches

but in the array I'm getting:

-ledensite

-Home Page

-Welkom bij autobedrijf Tolsma Exmorra

-etc.

Which means that after "Home Page" it skips 3 results. I tried to find any difference between results 5 and 6 (the last skipped and the next one), but I didn't find any reason not to match the 5th one.

This is the preg_match_all script:

$pattern = '@<div>\s*<h3>\s*<a[^<>]*\shref="([^*"]*\*\*)*([^<>]*)">
([^<>]*(<b>[^<>]*</b>)*[^<>]*(<wbr>)*[^<>]*)</a>\s*</h3>\s*</div>@siU';
preg_match_all($pattern, $result['EXE'], $matches);

The Yahoo page source for those two results looks like this:

<li><div class="res"><div><h3><a dirtyhref="http://nl.wrs.yahoo.com
/_ylt=A7x9Qb3rKqxN1XYA1PhzKAx.;_ylu=X3oDMTBydXF0bjc3BHNlYwN开发者_如何学PythonzcgR
wb3MDNQRjb2xvA2lyZAR2dGlkAw--/SIG=11n0lgcgp/EXP=1303157611/**http%3a
//www.autobedrijfgreijmans.nl/" class="yschttl spt" 
href="http://nl.wrs.yahoo.com/_ylt=A7x9Qb3rKqxN1XYA1PhzKAx.;
_ylu=X3oDMTBydXF0bjc3BHNlYwNzcgRwb3MDNQRjb2xvA2lyZAR2dGlkAw--/SIG
=11n0lgcgp/EXP=1303157611/**http%3a//www.autobedrijfgreijmans.nl/">
<b>Autobedrijf</b> Greijmans Weert - Toyota <wbr>Specialist Occasions 
APK</a></h3></div><div class="abstr">In het databestand van 
<b>Autobedrijf</b> Greijmans vindt u zeer eenvoudig tweedehands auto's, 
bedrijfswagens, klassiekers. Kijk snel op onze website</div><span 
class="url">www.<b>autobedrijfgreijmans.nl</b></span> - <a 
href="http://nl.wrs.yahoo.com/_ylt=A7x9Qb3rKqxN1XYA1fhzKAx./SIG=
186dh8afd/EXP=1303157611/**http%3a//74.6.239.67/search/cache%3fei=
UTF-8%26p=autobedrijf%26n=100%26va_vt=any%26vo_vt=any%26vp_vt
=any%26vst=0%26vf=all%26vm=p%26u=www.autobedrijfgreijmans.nl/
%26w=autobedrijf%26d=IxshPvbJWijU%26icp=1%26.intl=nl%26sig=yI6R7vJN31J
T92YKlVnT1g--">In de cache</a></div></li>

<li><div class="res"><div><h3><a dirtyhref="http://nl.wrs.yahoo.com
/_ylt=A7x9Qb3rKqxN1XYA1vhzKAx.;_ylu=X3oDMTBybWh0ZnN2BHNlYwNzcgRwb3MDNg
Rjb2xvA2lyZAR2dGlkAw--/SIG=11l17c21h/EXP=1303157611/**http%3a//www.
autobedrijf-tolsma.nl/" class="yschttl spt" href="http://www.autobedrijf
-tolsma.nl/">Welkom bij <b>autobedrijf</b> Tolsma Exmorra</a></h3></div>
<div class="abstr">Welkom op de vernieuwde website van <b>autobedrijf</b> 
Tolsma Exmorra incl.digitale showroom <b>...</b> Welkom bij <b>autobedrijf
</b> tolsma. Op deze site nodigen wij U uit een <b>...</b></div><span 
class="url">www.<b>autobedrijf-tolsma.nl</b></span> - <a href="http://nl.
wrs.yahoo.com/_ylt=A7x9Qb3rKqxN1XYA1_hzKAx./SIG=1840la9c5/EXP=1303157611/
**http%3a//74.6.239.67/search/cache%3fei=UTF-8%26p=autobedrijf%26n=100
%26va_vt=any%26vo_vt=any%26vp_vt=any%26vst=0%26vf=all%26vm=p%26u=
www.autobedrijf-tolsma.nl/%26w=autobedrijf%26d=YBZQIvbJWlDf%26icp=1%26.
intl=nl%26sig=zRU95PdBOTfII93dZ411ZA--">In de cache</a></div></li>

I'm stuck on that for more than 5 hours already and I can't figure out why the regex tools are matching both of the results, while preg_match_all in PHP is matching only the second one.

If it matters - I'm testing on Windows with Apache and PHP 5.3.5.

Any advices are welcome, I can provide extra examples and testing code if you're interested.


An unpolished starting point:

<?php

$url = 'http://nl.search.yahoo.com/search?p=autobedrijf&toggle=1&cop=mss&ei=UTF-8&fr=yfp-t-732';

$html = file_get_contents($url);

$doc = new DOMDocument;
libxml_use_internal_errors(TRUE);
$doc->loadHTML($html);
libxml_use_internal_errors(FALSE);

$titles = $doc->getElementsByTagName('h3');
foreach($titles as $t){
    echo $t->nodeValue . PHP_EOL;
}

?>


I recommend to use the Yahoo api. They offer very good services to reach results from yahoo with yahoo query language. http://developer.yahoo.com/yql/

example:

http://developer.yahoo.com/yql/console/#h=select%20*%20from%20search.web%20where%20query%3D%22autobedrijf%22

0

精彩评论

暂无评论...
验证码 换一张
取 消