I am going through a big list of files and running a command on all of them. I want to log the output and if the command takes more than 5 minutes for a file, I would like to stop the command and go to the next file.
I have two problems:
I want to record the file name to the output file and also record t开发者_高级运维he output messages. I am using
Popen
to log the messages and usingcommunicate
so it gets logged, but all the filenames I am writing withwrite()
don't get written till the whole task is done.I am not sure how I can poll the process and quit it after 5 minutes and go to the next file.
Below is simplified code:
import os, fnmatch
import subprocess
import sys
f=open('filenames','w')
'Locate all files matching supplied filename pattern in and below supplied root directory.'''
def locate(pattern, root=os.curdir):
for path, dirs, files in os.walk(os.path.abspath(root)):
for filename in fnmatch.filter(files, pattern):
yield os.path.join(path, filename)
for filename in locate("*.dll"):
f.write(filename)
#cmd defintion is changed for simplicity
cmd='cat %s' %filename
p=subprocess.Popen(cmd,stdout=f)
p.communicate()[0]
- Use
f.flush()
afterf.write(filename)
. - See Using module 'subprocess' with timeout
Here's what I use, roughly:
from datetime import datetime
from time import time
import subprocess
import sys
import threading
import os
class Execution(threading.Thread):
comm = None
proc = None
returncode = None
stdoutdata = None
stderrdate = None
def __init__(self, comm):
threading.Thread.__init__(self)
self.comm = comm
def run(self):
self.proc = subprocess.Popen(self.comm, bufsize=-1, stdin=None,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
self.proc.wait()
(self.stdoutdata, self.stderrdata) = self.proc.communicate()
self.returncode = self.proc.poll()
for curcommand in somecommandlist:
ex = Execution(curcommand)
starttime = time()
ex.start()
ex.join(some_timeout_in_seconds)
wastimeout = False
if (ex.isAlive()):
wastimeout = True
print("TIMEOUT OCCURRED @ %s" % (datetime.today(),))
ex.proc.kill()
ex.join()
endtime = time()
duration = endtime - starttime
print("DURATION %s" % (duration,))
exitcode = ex.returncode
print("EXIT CODE: %s" % (exitcode,))
print somepostprocessing(ex.stdoutdata,ex.stderrdata)
sys.stdout.flush()
Basically: start the process inside a separate thread. This is necessary because the threading
module gives you timeout waiting, while the subprocess
module does not. Wait until the timeout; check if the thread is still alive. If the thread is live, we cause it to terminate by killing the process (here, subprocess
gives you a kill()
construct, which threading
does not) and waiting for the thread to end on its own accord.
Keep in mind that communicate()
blocks and stores the full output from the child's stderr and stdout, so this may not work if the output is very large.
精彩评论