开发者

making LWP Useragent faster

开发者 https://www.devze.com 2022-12-10 04:08 出处:网络
I need to perform a large number of HTTP post requests, and ignore the response. I am currently doing this using LWP::UserAgent. It seems to run somewhat slow though I am not sure if it is waiting for

I need to perform a large number of HTTP post requests, and ignore the response. I am currently doing this using LWP::UserAgent. It seems to run somewhat slow though I am not sure if it is waiting for a response or what, is there anyway to speed it up and po开发者_开发百科ssibly just ignore the responses?


bigian's answer is probably the best for this, but another way to speed things up is to use LWP::ConnCache to allow LWP to re-use existing connections rather than build a new connection for every request.

Enabling it is this simple if you're pounding on just one site --

my $conn_cache = LWP::ConnCache->new;
$conn_cache->total_capacity([1]) ;
$ua->conn_cache($conn_cache) ;

I've found this to double the speed of some operations on http site, and more than double it for https sites.


LWP::Parallel

http://metacpan.org/pod/LWP::Parallel

"Introduction

ParallelUserAgent is an extension to the existing libwww module. It allows you to take a list of URLs (it currently supports HTTP, FTP, and FILE URLs. HTTPS might work, too) and connect to all of them in parallel, then wait for the results to come in."

It's great, it's worked wonders for me...

0

精彩评论

暂无评论...
验证码 换一张
取 消