I have a large array about 10000 data and want to process it with parallel::forkmanager
how can i proccess the data on chunck of 1000 i have the below code :
my $MAX_PROCESSES = 10;
my $pm = new Parallel::ForkManager($MAX_PROCESSES);
for (<>) {
my $pid = $pm->start and next;
# here i want to process my data on chunks of 1000 with 10 parallel::formanagaer
$pm->finish;
}
How can I custo开发者_如何学JAVAmize my code to do it ?
As the docs show, the data is passed from the parent to the child, so you want something that takes the following form:
for (;;) {
... get a chunk ...
my $pid = $pm->start and next;
... process chunk ...
$pm->finish;
}
So
use constant CHUNK_SIZE => 1000;
CHUNK:
for (my $eof = 0; !$eof;) {
my @chunk;
while (@chunk < CHUNK_SIZE) {
my $line = <>;
if (!$line) {
if (@chunk) {
$eof = 1; # Can't rely on a handle returning EOF twice.
last; # so we have to make a note of it.
} else {
last CHUNK;
}
}
push @chunk, $line;
}
my $pid = $pm->start and next;
... process chunk ...
$pm->finish;
}
my $max_procs = 4;
my $pm = new Parallel::ForkManager($max_procs);
foreach my $index ( 0 .. $max_procs-1 ) {
my $cmd = $$cli_to_execute[$index];
# Forks and returns the pid for the child:
my $pid = $pm->start($index) and next;
my $out = &cli_command( $cmd ) unless $pid; ### This code is the child process
$pm->finish($index, \$out); # pass an exit code to finish
}
print "Waiting for children...\n";
$pm->wait_all_children;
精彩评论