开发者

CURLOPT_FOLLOWLOCATION not working

开发者 https://www.devze.com 2023-03-05 17:07 出处:网络
I\'m trying to scrape the data at this link: http://www.treasurydirect.gov/NP/BPDLogin?application=np which contains a meta refresh.

I'm trying to scrape the data at this link: http://www.treasurydirect.gov/NP/BPDLogin?application=np

which contains a meta refresh.

I'm using curl_exec with CURLOPT_FOLLOWLOCATION set to true.

In phpinfo() I confirmed that safe mode and basedir are not set. However CURLOPT_FOLLOWLOCATION is still not working. Here's my code:

<?php 
error_reporting(E_ALL);

$url = 'http://www.treasurydirect.gov/NP/BPDLogin?application=np';

// READ THE WEB PAGE
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,20);
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,true);
curl_setopt($ch,CURLOPT_COOKIEFILE,"cookies.txt");
curl_setopt($ch,CURLOPT_COOKIEJAR,"cookies.txt");
curl_error($ch);
$htm = curl_exec($ch);
curl_close($ch);

// MAKE OUR OUTPUT EASY TO READ
echo "<pre>";

// GET THE TITLE
$txt = strip_tags($htm, '<title>');
$rgx
= '#'            // REGEX DELIMITER - START
. '\<title\>'    // TITLE TAG WITH ANGLE BRACKETS ESCAPED - START
. '(.*?)'        // GROUP OF ANYTHING
. '\</title\&开发者_运维问答gt;'   // TITLE TAG WITH ANGLE BRACKETS ESCAPED - END
. '#'            // REGEX DELIMITER - END
. 'is'           // CASE-INSENSITIVE, SINGLE LINE
;
preg_match($rgx, $txt, $arr);

// DISPLAY THE TITLE
echo
'<strong>'
. $arr[1]
. '</strong>'
. PHP_EOL
. PHP_EOL
;

// SHOW THE PAGE SOURCE
$src = htmlentities($htm);
echo $src;

?>


The problem is not the meta refresh tag (which by the way never will be followed by CURLOPT_FOLLOWLOCATION option) but the HTTP user agent header. The web site checks the HTTP user agent header field against a list of accepted user agents. You could solve this by adding the following line when setting options for $ch:

curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");


Meta refreshes are instructions for a browser. Curl doesn't process these. CURLOPT_FOLLOWLOCATION is meant for following redirects.

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号