Can't return appended value from within a Python process

I am running Python 2.6.5 on Ubuntu 8.10.

For a project I am working on, I need to run several processes concurrently and save each of their outputs into their own respectiv开发者_Go百科e lists. Since I can't return the output value directly from the process, I am passing the output list as an argument to my target function and appending the output to that list. The problem is that when I try to access that list after running the process, the list is still empty. A simplified version of the problem I'm having is below along with the error message.

Code:

from multiprocessing import Process
import sys, math, os,commands

outputs = []

def calculate(a,b,outputs):
    c = a*b
    outputs.append(c)
    #return c

outputs1 = []

p1 = Process(target = calculate, args = (2,3,outputs1))

p1.start()
p1.join()

print 'len(outputs1) = ' + str(len(outputs1))

print 'outputs1 = ' + str(outputs1[0])

Error:

len(outputs1) = 0
Traceback (most recent call last):
  File "append_test.py", line 23, in <module>
    print 'outputs1 = ' + str(outputs1[0])
IndexError: list index out of range

I am trying to keep each process completely independent of the others for fear of corrupting data. I've looked into trying to use the Array module from multiprocessing, but it appears that append is specific to lists. When I run the exact same code with Thread instead of Process, I get the desired output with no problem, which leads me to believe this is an issue of memory sharing.

---

When you use separate processes, each process gets its own copy of everything in memory. That's why the parent process never sees anything in its outputs: each child process is appending to its own copy of outputs.

You need to use some form of interprocess communication. Python's multiprocessing library supplies two features for this: pipes and queues.

For example, using a Queue:

>>> from multiprocessing import Process, Queue
>>> def f(q): q.put("hello from the child process")
... 
>>> q = Queue()
>>> p = Process(target=f, args=(q,))
>>> p.start()
>>> p.join()
>>> q.get()
'hello from the child process'

---

The section titled Exchanging objects between processes in the in the online documentation of themultiprocessingmodule says that it [only] "supports two types of communication channel between processes" and goes on to mention Queues and Pipes. Notably it does not mentionlistobjects likeoutputs1. This makes sense because the two processes don't share memory AFAIK.

I'm not sure, but I also suspect you might need to put the section of your code that creates the process and starts it, etc inside of anif __name__ == '__main__':clause to prevent the subprocess from creating a sub-subprocess.

In summary, I think you're going to have to rework things to use one of those two for interprocess communications -- Queues seem like the logical choice to me.