I'm trying to scrape the data at this link: http://www.treasurydirect.gov/NP/BPDLogin?application=np
which contains a meta refresh.
I'm using curl_exec with CURLOPT_FOLLOWLOCATION set to true.
In phpinfo() I confirmed that safe mode and basedir are not set. However CURLOPT_FOLLOWLOCATION is still not working. Here's my code:
<?php
error_reporting(E_ALL);
$url = 'http://www.treasurydirect.gov/NP/BPDLogin?application=np';
// READ THE WEB PAGE
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,20);
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,true);
curl_setopt($ch,CURLOPT_COOKIEFILE,"cookies.txt");
curl_setopt($ch,CURLOPT_COOKIEJAR,"cookies.txt");
curl_error($ch);
$htm = curl_exec($ch);
curl_close($ch);
// MAKE OUR OUTPUT EASY TO READ
echo "<pre>";
// GET THE TITLE
$txt = strip_tags($htm, '<title>');
$rgx
= '#' // REGEX DELIMITER - START
. '\<title\>' // TITLE TAG WITH ANGLE BRACKETS ESCAPED - START
. '(.*?)' // GROUP OF ANYTHING
. '\</title\&开发者_运维问答gt;' // TITLE TAG WITH ANGLE BRACKETS ESCAPED - END
. '#' // REGEX DELIMITER - END
. 'is' // CASE-INSENSITIVE, SINGLE LINE
;
preg_match($rgx, $txt, $arr);
// DISPLAY THE TITLE
echo
'<strong>'
. $arr[1]
. '</strong>'
. PHP_EOL
. PHP_EOL
;
// SHOW THE PAGE SOURCE
$src = htmlentities($htm);
echo $src;
?>
The problem is not the meta refresh tag (which by the way never will be followed by CURLOPT_FOLLOWLOCATION option) but the HTTP user agent header. The web site checks the HTTP user agent header field against a list of accepted user agents. You could solve this by adding the following line when setting options for $ch:
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
Meta refreshes are instructions for a browser. Curl doesn't process these. CURLOPT_FOLLOWLOCATION is meant for following redirects.
精彩评论