开发者

Nokogiri find only inbound links

开发者 https://www.devze.com 2022-12-31 22:18 出处:网络
I have an html document located on http://somedomain.com/somedir/example.html The document contains of four link开发者_StackOverflow社区s:

I have an html document located on http://somedomain.com/somedir/example.html

The document contains of four link开发者_StackOverflow社区s:

http://otherdomain.com/other.html

http://somedomain.com/other.html

/only.html

test.html

How I can get the full urls for the links in the current domain ?

I mean I should get:

http://somedomain.com/other.html

http://somedomain.com/only.html

http://somedomain.com/somedir/test.html

The first link should be ignored because it does'nt match my domain


Something like

doc.search("a").map do |a| 
  url = a.attribute("href")
  #this part could be a lot more robust, but you get the idea...
  full_url = url.match("^http://") ? url : "http://somedomain.com/#{url}"
end.select{|url| url.match("^http://somedomain.com")}


use a regular expression to extract the links from href="URL" then concatenate with the domain if it doesn't start with "http"

Here is a Python example:

import re
import urlparse

domain = ...
html = ...
links = re.findall('href=[\'"](.*?)[\'"]', html)
links = [urlparse.urljoin(domain, link) for link in links if link]
0

精彩评论

暂无评论...
验证码 换一张
取 消