开发者

Scraping with multiple IP, in java

开发者 https://www.devze.com 2022-12-22 19:23 出处:网络
Well basically I have a scraping application. It scrapes around n items per minute. currently i have only one IP.

Well basically I have a scraping application. It scrapes around n items per minute. currently i have only one IP.

The site i'm scraping allows me 3 connections per IP.

I'm thinking about getting another IP.

so i'll be able to get 6 connections.

in theory i should be able to get n items in 40 seconds, more or less.

currently i'm using java (commons-httpcore) to get the job done.

I'm not sure if this is java question or an OS question.

my machine has IP 1 and IP 2 how 开发者_运维知识库do i connect to, say, www.microsoft.com, using IP 1 and using IP2? how can i specify, which ip i want to use to do a connection?


// requires HttpComponents Client 4.*
DefaultHttpClient httpclient = new DefaultHttpClient();
httpclient.getParams().setParameter(
  ConnRoutePNames.LOCAL_ADDRESS, 
  InetAddress.getByName("10.10.10.10")
);

See: http://hc.apache.org/httpcomponents-client/httpclient/apidocs/index.html

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号