Python's threads stuck in Popen.communicate

I have a Python application that uses threads as follows:

1. Create tasks queue (Queue module)
2. Create 10 threads and pass to each a queue object
3. Put tasks into queue (around 8500 tasks in total)
4. Each thread:
   • takes a task and runs some Linux commands using Popen.communicate()

Mutexes, critical sections, queue management - my thread's pool library was tested already in couple smaller projects, so no reason to think sth is spoiled there...

Everything works fine when I have up to couple thousand tasks, however when I have more (in this case over 8500), some of the threads hang. gdb shows they're stuck in python's subprocess.py in _execute child (line 1131) -> means just after os.fork() is called.

gdb:

 (gdb) pystack
 /opt/python/current/lib/python2.7/subprocess.py (1131): _execute_child
 /opt/python/current/lib/python2.7/subprocess.py (681): __init__
 /home/olibsup/tools/chelo/checks/checkUtils/osutils/cmdutils.py (115): shcmd
 /home/olibsup/tools/chelo/checks/liblist/libWorkers.py (204): workerFunction
 /home/olibsup/tools/chelo/checks/checkUtils/pools/thpool.py (160): run
 /opt/python/current/lib/python2.7/threading.py (160): __bo开发者_如何学编程otstrap_inner
 /opt/python/current/lib/python2.7/threading.py (553): __bootstrap

My ulimit shows:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 139264
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 139264
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

'top' also doesn't show anything suspicious (at least not to me):

Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16438200k total, 15705272k used,   732928k free,   751640k buffers
Swap:  3148700k total,       44k used,  3148656k free, 11692300k cached

Do you have any ideas why threads hang there?

Not all threads hang, some (5 of 10) have finished properly (when no more tasks were available).

Thank you for your help,

Zbigniew

---

In subprocess.py at line 1131/1132, a file descriptor is duplicated via os.dup.

For this reason I suspect your operating system is limiting the number of subprocesses and/or the number of file descriptors available to your application. However, I do not understand why os.dup doesn't raise/throw an exception in that case.

Try to find out your operating system's limit and stay below that limit. For UNIX-based systems you can probably use Python's resource module (though I have never used it myself): http://docs.python.org/library/resource.html