Using module subprocess with timeout

Question

Here s the Python code to run an arbitrary command returning its stdout data  or raise an exception on non-zero exit codes   proc   subprocess Popen      cmd      stderr subprocess STDOUT     Merge stdout and stderr     stdout subprocess PIPE      shell True    communicate is used to wait for the process to exit   stdoutdata  stderrdata   proc communicate     The subprocess module does not support timeout--ability to kill a process running for more than X number of seconds--therefore  communicate may take forever to run   What is the simplest way to implement timeouts in a Python program meant to run on Windows and Linux

User · Answer

If you re on Unix   import signal       class Alarm Exception       pass  def alarm handler signum  frame       raise Alarm  signal signal signal SIGALRM  alarm handler  signal alarm 5 60     5 minutes try      stdoutdata  stderrdata   proc communicate       signal alarm 0     reset the alarm except Alarm      print  Oops  taking too long         whatever else

User · Answer

There s an idea to subclass the Popen class and extend it with some simple method decorators  Let s call it ExpirablePopen   from logging import error from subprocess import Popen from threading import Event from threading import Thread   class ExpirablePopen Popen        def   init   self   args    kwargs           self timeout   kwargs pop  timeout   0          self timer   None         self done   Event            Popen   init   self   args    kwargs       def   tkill self           timeout   self timeout         if not self done wait timeout               error  Terminating process    by timeout of    secs   format self pid  timeout               self kill        def expirable func           def wrapper self   args    kwargs                 zero timeout means call of parent method             if self timeout    0                  return func self   args    kwargs                 if timer is None  need to start it             if self timer is None                  self timer   thr   Thread target self   tkill                  thr daemon   True                 thr start                result   func self   args    kwargs              self done set                return result         return wrapper      wait   expirable Popen wait      communicate   expirable Popen communicate    if   name         main         from subprocess import PIPE      print ExpirablePopen  ssh -T git bitbucket org   stdout PIPE  timeout 1  communicate

User · Answer

Prepending the Linux command timeout isn t a bad workaround and it worked for me   cmd    timeout 20    cmd subprocess Popen cmd split    stdout subprocess PIPE  stderr subprocess PIPE   output  err    p communicate

User · Answer

I don t know much about the low level details  but  given that in python 2 6 the API offers the ability to wait for threads and terminate processes  what about running the process in a separate thread   import subprocess  threading  class Command object       def   init   self  cmd           self cmd   cmd         self process   None      def run self  timeout           def target                print  Thread started              self process   subprocess Popen self cmd  shell True              self process communicate               print  Thread finished           thread   threading Thread target target          thread start            thread join timeout          if thread is alive                print  Terminating process              self process terminate               thread join           print self process returncode  command   Command  echo  Process started   sleep 2  echo  Process finished    command run timeout 3  command run timeout 1    The output of this snippet in my machine is   Thread started Process started Process finished Thread finished 0 Thread started Process started Terminating process Thread finished -15   where it can be seen that  in the first execution  the process finished correctly  return code 0   while the in the second one the process was terminated  return code -15    I haven t tested in windows  but  aside from updating the example command  I think it should work since I haven t found in the documentation anything that says that thread join or process terminate is not supported

User · Answer

Although I haven t looked at it extensively  this decorator I found at ActiveState seems to be quite useful for this sort of thing  Along with subprocess Popen      close fds True   at least I m ready for shell-scripting in Python

User · Answer

python 2 7  import time import subprocess  def run command cmd  timeout 0       start time   time time       df   subprocess Popen cmd  stdout subprocess PIPE  stderr subprocess PIPE      while timeout and df poll      None          if time time  -start time  gt   timeout              df kill               return -1         output     n  join df communicate    strip       return df returncode  output

User · Answer

I ve implemented what I could gather from a few of these  This works in Windows  and since this is a community wiki  I figure I would share my code as well   class Command threading Thread       def   init   self  cmd  outFile  errFile  timeout           threading Thread   init   self          self cmd   cmd         self process   None         self outFile   outFile         self errFile   errFile         self timed out   False         self timeout   timeout      def run self           self process   subprocess Popen self cmd  stdout   self outFile                stderr   self errFile           while  self process poll   is None and self timeout  gt  0               time sleep 1              self timeout -  1          if not self timeout  gt  0              self process terminate               self timed out   True         else              self timed out   False   Then from another class or file           outFile    tempfile SpooledTemporaryFile           errFile     tempfile SpooledTemporaryFile            executor   command Command c  outFile  errFile  timeout          executor daemon   True         executor start            executor join           if executor timed out              out    timed out          else              outFile seek 0              errFile seek 0              out   outFile read               err   errFile read            outFile close           errFile close

User · Answer

surprised nobody mentioned using timeout  timeout 5 ping -c 3 somehost  This won t for work for every use case obviously  but if your dealing with a simple script  this is hard to beat   Also available as gtimeout in coreutils via homebrew for mac users

User · Answer

jcollado s answer can be simplified using the threading Timer class   import shlex from subprocess import Popen  PIPE from threading import Timer  def run cmd  timeout sec       proc   Popen shlex split cmd   stdout PIPE  stderr PIPE      timer   Timer timeout sec  proc kill      try          timer start           stdout  stderr   proc communicate       finally          timer cancel      Examples  both take 1 second run  sleep 1   5     process ends normally at 1 second run  sleep 5   1     timeout happens at 1 second

User · Answer

This solution kills the process tree in case of shell True  passes parameters to the process  or not   has a timeout and gets the stdout  stderr and process output of the call back  it uses psutil for the kill proc tree    This was based on several solutions posted in SO including jcollado s  Posting in response to comments by Anson and jradice in jcollado s answer  Tested in Windows Srvr 2012 and Ubuntu 14 04   Please note that for Ubuntu you need to change the parent children      call to parent get children        def kill proc tree pid  including parent True     parent   psutil Process pid    children   parent children recursive True    for child in children      child kill     psutil wait procs children  timeout 5    if including parent      parent kill       parent wait 5   def run with timeout cmd  current dir  cmd parms  timeout     def target        process   subprocess Popen cmd  cwd current dir  shell True  stdout subprocess PIPE  stdin subprocess PIPE  stderr subprocess PIPE         wait for the process to terminate     if  cmd parms               out  err   process communicate       else        out  err   process communicate cmd parms      errcode   process returncode    thread   Thread target target    thread start      thread join timeout    if thread is alive        me   os getpid       kill proc tree me  including parent False      thread join

User · Answer

https   pypi python org pypi python-subprocess2 provides extensions to the subprocess module which allow you to wait up to a certain period of time  otherwise terminate    So  to wait up to 10 seconds for the process  to terminate  otherwise kill   pipe    subprocess Popen         timeout    10  results   pipe waitOrTerminate timeout    This is compatible with both windows and unix    results  is a dictionary  it contains  returnCode  which is the return of the app  or None if it had to  be killed   as well as  actionTaken   which will be  SUBPROCESS2 PROCESS COMPLETED  if the process completed normally  or a mask of  SUBPROCESS2 PROCESS TERMINATED  and SUBPROCESS2 PROCESS KILLED depending on action taken  see documentation for full details

User · Answer

The solution I use is to prefix the shell command with timelimit  If the comand takes too long  timelimit will stop it and Popen will have a returncode set by timelimit  If it is   128  it means timelimit killed the process   See also python subprocess with timeout and large output   gt 64K

User · Answer

Example of captured output after timeout tested in Python 3 7 8  try      return subprocess run command  shell True  capture output True  timeout 20  cwd cwd  universal newlines True  except subprocess TimeoutExpired as e      print e output decode encoding  quot utf-8 quot   errors  quot ignore quot        assert False   The exception subprocess TimeoutExpired has the output and other members   cmd - Command that was used to spawn the child process  timeout - Timeout in seconds  output - Output of the child process if it was captured by run   or check output    Otherwise  None  stdout - Alias for output  for symmetry with stderr  stderr - Stderr output of the child process if it was captured by run    Otherwise  None   More info  https   docs python org 3 library subprocess html subprocess TimeoutExpired

User · Answer

I ve used killableprocess successfully on Windows  Linux and Mac  If you are using Cygwin Python  you ll need OSAF s version of killableprocess because otherwise native Windows processes won t get killed

User · Answer

Another option is to write to a temporary file to prevent the stdout blocking instead of needing to poll with communicate    This worked for me where the other answers did not  for example on windows       outFile    tempfile SpooledTemporaryFile        errFile     tempfile SpooledTemporaryFile        proc   subprocess Popen args  stderr errFile  stdout outFile  universal newlines False      wait remaining sec   timeout      while proc poll   is None and wait remaining sec  gt  0          time sleep 1          wait remaining sec -  1      if wait remaining sec  lt   0          killProc proc pid          raise ProcessIncompleteError proc  timeout         read temp streams from start     outFile seek 0       errFile seek 0       out   outFile read       err   errFile read       outFile close       errFile close

User · Answer

for python 2 6   use gevent   from gevent subprocess import Popen  PIPE  STDOUT   def call sys cmd  timeout         p  Popen cmd  shell True  stdout PIPE        output      p communicate timeout timeout        assert p returncode    0  p  returncode       return output   call sys    t sh   2      t sh example  sleep 5  echo done  exit 1

User · Answer

timeout is now supported by call   and communicate   in the subprocess module  as of Python3 3    import subprocess  subprocess call  command   timeout 20  shell True    This will call the command and raise the exception  subprocess TimeoutExpired   if the command doesn t finish after 20 seconds   You can then handle the exception to continue your code  something like   try      subprocess call  command   timeout 20  shell True  except subprocess TimeoutExpired        insert code here   Hope this helps

User · Answer

if you are using python 2  give it a try  import subprocess32  try      output   subprocess32 check output command  shell True  timeout 3  except subprocess32 TimeoutExpired as e      print e

User · Answer

Was just trying to write something simpler      usr bin python  from subprocess import Popen  PIPE import datetime import time   popen   Popen    bin sleep    10     pid   popen pid sttime   time time    waittime    3  print  Start time  s   sttime   while True      popen poll        time sleep 1      rcode   popen returncode     now   time time        if   rcode is None    and    now  gt   sttime   waittime              print  Killing it now          popen kill

User · Answer

I added the solution with threading from jcollado to my Python module easyprocess   Install   pip install easyprocess   Example   from easyprocess import Proc    shell is not supported  stdout Proc  ping localhost   call timeout 1 5  stdout print stdout

User · Answer

import subprocess  optparse  os  sys  re  datetime  threading  time  glob  shutil  xml dom minidom  traceback  class OutputManager      def   init   self  filename  mode  console  logonly           self con   console         self logtoconsole   True         self logtofile   False          if filename              try                  self f   open filename  mode                  self logtofile   True                 if logonly    True                      self logtoconsole   False             except IOError                  print  sys exc value                  print   Switching to console only output    n                   self logtofile   False                 self logtoconsole   True      def write self  data           if self logtoconsole    True              self con write data          if self logtofile    True              self f write data          sys stdout flush    def getTimeString            return time strftime   Y- m- d   time gmtime     def runCommand command               Execute a command in new thread and return the     stdout and stderr content of it              try          Output   subprocess Popen command  stdout subprocess PIPE  shell True  communicate   0      except Exception as e          print   runCommand failed   s     command           print  str e           sys stdout flush           return None     return Output  def GetOs        Os          if sys platform startswith  win32            Os    win      elif sys platform startswith  linux            Os    linux      elif sys platform startswith  darwin            Os    mac      return Os   def check output  popenargs    kwargs       try          if  stdout  in kwargs               raise ValueError  stdout argument not allowed  it will be overridden                Get start time          startTime   datetime datetime now           timeoutValue 3600          cmd   popenargs 0           if sys platform startswith  win32                process   subprocess Popen  cmd  stdout subprocess PIPE  shell True           elif sys platform startswith  linux                process   subprocess Popen  cmd   stdout subprocess PIPE  shell True            elif sys platform startswith  darwin                process   subprocess Popen  cmd   stdout subprocess PIPE  shell True             stdoutdata  stderrdata   process communicate  timeout   timeoutValue           retcode   process poll                                                           Catch crash error and log it                                                       OutputHandle   None         try              if retcode  gt   1                  OutputHandle   OutputManager   CrashJob     getTimeString       txt    a    sys stdout  False                  OutputHandle write  cmd                   print  stdoutdata                  print  stderrdata                  sys stdout flush           except Exception as e              print  str e        except subprocess TimeoutExpired                                                                 Catch time out error and log it                                                               Os   GetOs               if Os     win                   killCmd    taskkill  FI   IMAGENAME eq  0     T  F              elif Os     linux                   killCmd    pkill  0               elif Os     mac                     Linux  Mac OS                 killCmd    killall -KILL  0                runCommand killCmd format  java                runCommand killCmd format  YouApp                 OutputHandle   None             try                  OutputHandle   OutputManager   KillJob     getTimeString       txt    a    sys stdout  False                  OutputHandle write  cmd               except Exception as e                  print  str e       except Exception as e              for frame in traceback extract tb sys exc info   2                            fname lineno fn text   frame                         print  Error in  s on line  d     fname  lineno

User · Answer

Here is my solution  I was using Thread and Event   import subprocess from threading import Thread  Event  def kill on timeout done  timeout  proc       if not done wait timeout           proc kill    def exec command command  timeout        done   Event       proc   subprocess Popen command  stdout subprocess PIPE  stderr subprocess PIPE       watcher   Thread target kill on timeout  args  done  timeout  proc       watcher daemon   True     watcher start        data  stderr   proc communicate       done set        return data  stderr  proc returncode   In action   In  2   exec command   sleep    10    5  Out 2            -9   In  3   exec command   sleep    10    11  Out 3            0

User · Answer

I don t know why it isn t mentionned but since Python 3 5  there s a new subprocess run universal command  that is meant to replace check call  check output      and which has the timeout parameter as well      subprocess run args     stdin None  input None  stdout None  stderr None  shell False  cwd None  timeout None  check False  encoding None  errors None    Run the command described by args  Wait for command to complete  then return a CompletedProcess instance    It raises a subprocess TimeoutExpired exception when the timeout is expired

User · Answer

Here is Alex Martelli s solution as a module with proper process killing  The other approaches do not work because they do not use proc communicate    So if you have a process that produces lots of output  it will fill its output buffer and then block until you read something from it   from os import kill from signal import alarm  signal  SIGALRM  SIGKILL from subprocess import PIPE  Popen  def run args  cwd   None  shell   False  kill tree   True  timeout   -1  env   None               Run a command with a timeout after which it will be forcibly     killed              class Alarm Exception           pass     def alarm handler signum  frame           raise Alarm     p   Popen args  shell   shell  cwd   cwd  stdout   PIPE  stderr   PIPE  env   env      if timeout    -1          signal SIGALRM  alarm handler          alarm timeout      try          stdout  stderr   p communicate           if timeout    -1              alarm 0      except Alarm          pids    p pid          if kill tree              pids extend get process children p pid           for pid in pids                process might have died before getting to this line               so wrap to avoid OSError  no such process             try                   kill pid  SIGKILL              except OSError                  pass         return -9             return p returncode  stdout  stderr  def get process children pid       p   Popen  ps --no-headers -o pid --ppid  d    pid  shell   True                stdout   PIPE  stderr   PIPE      stdout  stderr   p communicate       return  int p  for p in stdout split     if   name         main         print run  find     shell   True  timeout   3      print run  find   shell   True

User · Answer

Once you understand full process running machinery in  unix  you will easily find simplier solution   Consider this simple example how to make timeoutable communicate   meth using select select    available alsmost everythere on  nix nowadays   This also can be written with epoll poll kqueue  but select select   variant could be a good example for you  And major limitations of select select    speed and 1024 max fds  are not applicapable for your task   This works under  nix  does not create threads  does not uses signals  can be lauched from any thread  not only main   and fast enought to read 250mb s of data from stdout on my machine  i5 2 3ghz    There is a problem in join ing stdout stderr at the end of communicate  If you have huge program output this could lead to big memory usage  But you can call communicate   several times with smaller timeouts   class Popen subprocess Popen       def communicate self  input None  timeout None           if timeout is None              return subprocess Popen communicate self  input           if self stdin                Flush stdio buffer  this might block if user               has been writing to  stdin in an uncontrolled               fashion              self stdin flush               if not input                  self stdin close            read set  write set                  stdout   stderr   None          if self stdin and input              write set append self stdin          if self stdout              read set append self stdout              stdout              if self stderr              read set append self stderr              stderr               input offset   0         deadline   time time     timeout          while read set or write set              try                  rlist  wlist  xlist   select select read set  write set      max 0  deadline - time time                 except select error as ex                  if ex args 0     errno EINTR                      continue                 raise              if not  rlist or wlist                     Just break if timeout                   Since we do not close stdout stderr stdin  we can call                   communicate   several times reading data by smaller pieces                  break              if self stdin in wlist                  chunk   input input offset input offset   subprocess  PIPE BUF                  try                      bytes written   os write self stdin fileno    chunk                  except OSError as ex                      if ex errno    errno EPIPE                          self stdin close                           write set remove self stdin                      else                          raise                 else                      input offset    bytes written                     if input offset  gt   len input                           self stdin close                           write set remove self stdin                 Read stdout   stderr by 1024 bytes             for fn  tgt in                    self stdout  stdout                    self stderr  stderr                                  if fn in rlist                      data   os read fn fileno    1024                      if data                                fn close                           read set remove fn                      tgt append data           if stdout is not None              stdout      join stdout          if stderr is not None              stderr      join stderr           return  stdout  stderr

User · Answer

You can do this using select  import subprocess from datetime import datetime from select import select  def call with timeout cmd  timeout       started   datetime now       sp   subprocess Popen cmd  stdout subprocess PIPE      while True          p   select  sp stdout           timeout          if p 0               p 0  0  read           ret   sp poll           if ret is not None              return ret         if  datetime now  -started  total seconds    gt  timeout              sp kill               return None

User · Answer

Unfortunately  I m bound by very strict policies on the disclosure of source code by my employer  so I can t provide actual code   But for my taste the best solution is to create a subclass overriding Popen wait   to poll instead of wait indefinitely  and Popen   init   to accept a timeout parameter   Once you do that  all the other Popen methods  which call wait  will work as expected  including communicate

User · Answer

In Python 3 3    from subprocess import STDOUT  check output  output   check output cmd  stderr STDOUT  timeout seconds    output is a byte string that contains command s merged stdout  stderr data    check output raises CalledProcessError on non-zero exit status as specified in the question s text unlike proc communicate   method   I ve removed shell True because it is often used unnecessarily  You can  always add it back if cmd indeed requires it  If you add shell True i e   if the child process spawns its own descendants  check output   can return much later than the timeout indicates  see Subprocess timeout failure   The timeout feature is available on Python 2 x via the subprocess32 backport of the 3 2  subprocess module

User · Answer

I ve modified sussudio answer  Now function returns   returncode  stdout  stderr  timeout  - stdout and stderr is decoded to utf-8 string  def kill proc proc  timeout     timeout  value     True   proc kill    def run cmd  timeout sec     proc   subprocess Popen shlex split cmd   stdout subprocess PIPE  stderr subprocess PIPE    timeout     value   False    timer   Timer timeout sec  kill proc   proc  timeout     timer start     stdout  stderr   proc communicate     timer cancel     return proc returncode  stdout decode  utf-8    stderr decode  utf-8    timeout  value

User · Answer

I had the problem that I wanted to terminate a multithreading subprocess if it took longer than a given timeout length  I wanted to set a timeout in Popen    but it did not work  Then  I realized that Popen   wait   is equal to call   and so I had the idea to set a timeout within the  wait timeout xxx  method  which finally worked  Thus  I solved it this way   import os import sys import signal import subprocess from multiprocessing import Pool  cores for parallelization   4 timeout time   15    seconds  def main        jobs       YOUR JOB LIST         with Pool cores for parallelization  as p          p map run parallel jobs  jobs   def run parallel jobs args         Define the arguments including the paths     initial terminal command    C   Python34  python exe     Python executable     function to start    C   temp  xyz py     The multithreading script     final list    initial terminal command  function to start      final list extend args         Start the subprocess and determine the process PID     subp   subprocess Popen final list     starts the process     pid   subp pid        Wait until the return code returns from the function by considering the timeout         If not  terminate the process      try          returncode   subp wait timeout timeout time     should be zero if accomplished     except subprocess TimeoutExpired            Distinguish between Linux and Windows and terminate the process if            the timeout has been expired         if sys platform     linux2               os kill pid  signal SIGTERM          elif sys platform     win32               subp terminate    if   name         main         main

[python] Using module 'subprocess' with timeout

Examples related to python

Examples related to multithreading

Examples related to timeout

Examples related to subprocess