multiprocessing + psycopg2 zombie children

I am try开发者_JAVA技巧ing to insert and update a few million rows using psycopg and multiprocessing. Going by the documentation found in http://initd.org/psycopg/docs/usage.html#thread-and-process-safety, each child has its own connection to the DB.

But during the course of execution, only one child runs while the others become zombies. The script in itself is pretty simple and here is a trimmed version of the same,

import os
import psycopg2

from multiprocessing import Process


def _target(args):
    # Each forked process will have its own connection
    # http://initd.org/psycopg/docs/usage.html#thread-and-process-safety
    conn = get_db_connection()

    # Stuff seems to execute till this point in all the children
    print os.getpid(), os.getppid()

    # Do some updates here. After this only one child is active and running
    # Others become Zombies after a while.


if __name__ == '__main__':
    args = "Foo"
    for i in xrange(3):
        p = Process(target=_target, args=(args,))
        p.start()

I also checked if the tables have an escalated lock by peeking into pg_locks, but it looks like its not the case. Am I missing something obvious?

---

your processes become zombies because there jobs are finished but the processes are not joined. I reproduced your problem with this single test (I added sleep to simulate long jobs) :

import os
import time
from multiprocessing import Process

def _target(args):
    print os.getpid(), os.getppid()
    time.sleep(2)
    print os.getpid(), "will stop"

if __name__ == '__main__':
    args = "Foo"
    for i in xrange(3):
        p = Process(target=_target, args=(args,))
        p.start()
    import time
    time.sleep(10)

when executing this, after the 3 processes print that they will stop, they become in the ps view (they don't move anymore, but are not really dead because the father still hold them).

If I replace the main part with this, i have no more zombies :

if __name__ == '__main__':
    args = "Foo"
    processes = []
    for i in xrange(3):
        p = Process(target=_target, args=(args,))
        processes.append(p)
        p.start()
    for p in processes:
        p.join()
    import time
    time.sleep(10)