开发者

Opening a WIKI URL with a comma using `open-uri`

开发者 https://www.devze.com 2022-12-22 01:50 出处:网络
I am running in to OpenURI::HTTPError: 403 Forbidden error when I try to open a URL with a comma (OR other special characters like .).

I am running in to OpenURI::HTTPError: 403 Forbidden error when I try to open a URL with a comma (OR other special characters like .). I am able to open the same url in a browser.

require 'open-uri'
url = "http://en.wikipedia.org/wiki/Thor_Industries,_Inc."
f = open(url)
# throws OpenURI::HTTPError: 403 Forbidden error

How do I escape such URL?

I have tried to escape the url with C开发者_开发知识库GI::escape and I get the same error.

f = open(CGI::escape(url))


Typically, one would simply require the module cgi, then use CGI::escape(str).

require 'cgi'
require 'open-uri'
escaped_page = CGI::escape("Thor_Industries,_Inc.")
url = "http://en.wikipedia.org/wiki/#{escaped_page}"
f = open(url)

However, this doesn't seem to work for your particular instance, and still returns a 403. I'll leave this here for reference, regardless.


Edit: Wikipedia is refusing your requests because it suspects that you are a bot. It would seem that certain pages that are clearly content are granted to you, but those that don't match its "safe" pattern (e.g. those that contain dots or commas) are subject to its screening. If you actually output the content (I did this with Net::HTTP), you get the following:

Scripts should use an informative User-Agent string with contact information, or they may be IP-blocked without notice.

Providing a user-agent string, however, solves the issue:

open("http://en.wikipedia.org/wiki/Thor_Industries,_Inc.",
  "User-Agent" => "Ruby/#{RUBY_VERSION}")
0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号