Running Python's subprocess.Popen with a timeout from within Zope

There are moments when you go outside the world of Python to run something on the command line in a shell. Python's subprocess module makes this doable. Run enough processes on the shell and sure enough, some of them will get stuck. This can spoil your and your users day and should be caught by robust programming practices. A timeout is one such robustness solution. But subprocess.Popen doesn't have an option for a that, even though Guido van Rossum suggested to add a timeout option to it in 2005. There are a couple of suggestions for crafting such a thing in, some more complicated, some less, some extra complicated for working in Windows, some not. I did not have the Windows requirement here, but I ran into another stumbling block.

One of the simplest solutions seems to have been to use the signal module to send a timed SIGALRM to itself. Unfortunately signals in python can be only received in the main thread. That makes the signal solution solution not work in Zope. With some help from Marius Gedminas on #zope, I came up with another solution. I've used threading to spin of a "watchdog" thread that will kill my potentially stuck subprocess after a timeout. Meanwhile, if the main thread finds that the subprocess finished normally, it can cancel out the watchdog. There's even a flag that will be set so the main thread knows if there's been success or a kill...

So, here is the little method I'm using:

import time
import threading
import signal

# ....

def run_popen_with_timeout(command_string, timeout, input_data):
    """
    Run a sub-program in subprocess.Popen, pass it the input_data,
    kill it if the specified timeout has passed.
    returns a tuple of success, stdout, stderr
    """
    kill_check = threading.Event()
    def _kill_process_after_a_timeout(pid):
        os.kill(pid, signal.SIGTERM)
        kill_check.set() # tell the main routine that we had to kill
        # use SIGKILL if hard to kill...
        return
    p = Popen(command_string, bufsize=1, shell=True,
              stdin=PIPE, stdout=PIPE, stderr=PIPE)
    pid = p.pid
    watchdog = threading.Timer(timeout, _kill_process_after_a_timeout, args=(pid, ))
    watchdog.start()
    (stdout, stderr) = p.communicate(input_data)
    watchdog.cancel() # if it's still waiting to run
    success = not kill_check.isSet()
    kill_check.clear()
    return (success, stdout, stderr)

Here is simple a test, which I run with ZopeTestCase and which assumes that run_popen_with_timeout lives in a file called utilities.py:

def test_run_popen_with_timeout(self):
    '''run_popen_with_timeout - check for running/killing a subprocess on timeout'''
    from utilities import run_popen_with_timeout
    input_data = 'bla' # not used really
    command_string = os.path.join(os.path.dirname(__file__), 'takestime.py')
    timeout = 2.0
    success, stdout, stderr = run_popen_with_timeout(command_string, timeout, input_data)
    self.failIf(success)
    self.failIf('done' in stdout)
    timeout = 10.0
    success, stdout, stderr = run_popen_with_timeout(command_string, timeout, input_data)
    self.failUnless(success)
    self.failUnless('done' in stdout, 'no "done" in stdout: ' + stdout + ' stderr: ' + stderr)

The test uses this helper script to simulate a sometimes long running process:

#!/usr/local/bin/python
# a simple test script
# to test run_popen_with_timeout
# save as "takestime.py" in your tests directory
import time
print 'starting'
for i in range(3):
    time.sleep(2)
    print 'waiting'
print 'done'