Is there a simple way to launch a background task from a Python CGI script without waiting around for it to terminate?

In Windows, that is.

I think the answer to this question is that I need to create a Windows service. This seems ludicrously heavyweight for what I am trying to do.

I'm just trying to slap together a little prototype here for my manager, I'm not going to be responsible for productizing it... in fact, it may never even BE productized; it might just be something that a few researchers play around with.

I have a CGI script that receives a file for upload, stores it to a temporary location, then launches a background process to do some serious number-crunching on the file. Then some Javascript stuff sits around calling other CGI scripts to check on the status and update the page as needed.

All of this works, except the damn web server won't close the connection as long as the subrocess is running. I've done some searching, and it appears the answer on Unix is to make it a daemon, but I'm stuck on Windows right now and I guess the answer there is to make it a Windows service?!? This seems incredibly heavyweight to just, you know, launch a damn process and then close the server connection.

That's really the only way?

Edit: Okay, found a nifty little hack over here (the choice (3) that the guy gives): How to completely background a process in Perl CGI under IIS

I was able to modify this to make it even simpler, and although this is a klugey solution, it is perfect for the quick-and-dirty little prototype I am trying to make.

So I initially had my main script doing this:

subprocess.Popen("python.exe","myscript.py","arg1","arg2")

Which doesn't work, as I've described. Instead, I now have my main script emit this little bit of Javascript which runs after the document is fully loaded:

$("#somecrap").load("launchBackgroundProcess.py", {arg1:"foo",arg2:"bar"});

And then launchBackgroundProcess.py does the subprocess.Popen.

This solution would never scale, since it still leaves the browser connection open during the entire time the background task is running. But since this little thinger开发者_StackOverflow I am whipping up might someday have two simultaneous users at most (even then I doubt it) resources are not a concern. This allows the user to see the main page and get the Javascript updates even though there is still an http connection hanging open for no good reason.

Thanks for the answers! If I'm ever asked to productize this, I'll take at the resources Profane recommends.

---

If you haven't much experience with windows programming and don't wish to peruse the MSDN docs-- I don't blame you-- you may want to try to pick up a copy of Mark Hammond's cannonical guide to all things python and windows. It somehow never goes out-of-date on many of these sorts of recurring questions. Instead of launching the process with the every-platform solution, you'd probably be better off using the win32process module. Chapter 17 of the Hammond book covers this extensively, but you could probably get all you need by downloading the pywin ide (I think it comes bundled in the windows extensions which you can download from pypi), and looking through the help docs it has on python's windows' api. Here's an example of using the api, from a project I was working on recently. It may in fact do some of what you want with a little adaptation. You'd probably want to focus on CreationFlags. In particular, win32process.DETACHED_PROCESS is "often used to execute console programs in the background." Many other flags are available and conveniently wrapped however.

    if subprocess.mswindows:
        su=subprocess.STARTUPINFO()
        su.dwFlags |= subprocess._subprocess.STARTF_USESHOWWINDOW
    process = subprocess.Popen(['program', 'flag', 'flag2'], bufsize=-1,
              stdout=subprocess.PIPE, startupinfo=su)

---

Simplest, but not most efficient way would be to just run another python executable

from subprocess import Popen 
Popen("python somescript.py")

---

You can just use a system call using the "start" windows command. This way your python script will not wait for the completion of the started program.

---

CGI scripts are run with standard output redirected, either directly to the TCP socket or to a pipe. Typically, the connection won't close until the handle, and all copies of it, are closed. By default, the subprocess will inherit a copy of the handle.

There are two ways to prevent the connection from waiting on the subprocess. One is to prevent the subprocess from inheriting the handle, the other is for the subprocess to close its copy of the handle when it starts.

If the subprocess is in Perl, I think you could close the handle very simply:

close(STDOUT);

If you want to prevent the subprocess from inheriting the handle, you could use the SetHandleInformation function (if you have access to the Win32 API) or set bInheritHandles to FALSE in the call to CreateProcess. Alternatively, close the handle before launching the subprocess.