开发者

How to crawl a wordpress blog?

开发者 https://www.devze.com 2023-02-13 13:20 出处:网络
I write a c program to crawl blogs. It works well until it meets this blog: www.ipujia.com. I send the HTTP request:

I write a c program to crawl blogs. It works well until it meets this blog: www.ipujia.com. I send the HTTP request:

GET http://www.ipujia.com/ HTTP/1.0

to the website and get the response as below:

HTTP/1.1 301 Moved Permanently
Date: Sun, 27 Feb 2011 13:15:26 GMT
Server: Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8e-fips-rhel5
mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 开发者_运维问答mod_perl/2.0.4 
Perl/v5.8.8
X-Powered-By: PHP/5.2.14
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
Last-Modified: Sun, 27 Feb 2011 13:15:27 GMT
Location: http://http/www.ipujia.com/
Content-Length: 0
Connection: close
Content-Type: text/html; charset=UTF-8

This is strange because I cannot get the index page following the Location. Does anyone have any ideas?


The Location field in the response contains a malformed URI.

Location: http://http/www.ipujia.com/ (notice the protocol error) Should be

Location: http://www.ipujia.com/

Unless you are in control of the server there is little you could do here.

To solve it could you not parse the "Location" response and attempt to extract a valid URI from the it?

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号