live output from subprocess command

Question

I m using a python script as a driver for a hydrodynamics code   When it comes time to run the simulation  I use subprocess Popen to run the code  collect the output from stdout and stderr into a subprocess PIPE --- then I can print  and save to a log-file  the output information  and check for any errors   The problem is  I have no idea how the code is progressing  If I run it directly from the command line  it gives me output about what iteration its at  what time  what the next time-step is  etc  Is there a way to both store the output  for logging and error checking   and also produce a live-streaming output  The relevant section of my code  ret val   subprocess Popen  run command  stdout subprocess PIPE  stderr subprocess PIPE  shell True   output  errors   ret val communicate   log file write output  print output if  ret val returncode        print  quot RUN failed n n s n n quot     errors      success   False  if  errors    log file write  quot  n n s n n quot    errors   Originally I was piping the run command through tee so that a copy went directly to the log-file  and the stream still output directly to the terminal -- but that way I can t store any errors  to my knowlege    My temporary solution so far  ret val   subprocess Popen  run command  stdout log file  stderr subprocess PIPE  shell True   while not ret val poll        log file flush    then  in another terminal  run tail -f log txt  s t  log file    log txt

User · Answer

import os  def execute cmd  callback       for line in iter os popen cmd  readline                    callback line  -1    execute  ls -a   print

User · Answer

Similar to previous answers but the following solution worked for for me on windows using Python3 to provide a common method to print and log in realtime  getting-realtime-output-using-python    def print and log command  logFile       with open logFile   wb   as f          command   subprocess Popen command  stdout subprocess PIPE  shell True           while True              output   command stdout readline               if not output and command poll   is not None                  f close                   break             if output                  f write output                  print str output strip     utf-8    flush True          return command poll

User · Answer

Executive Summary  or  tl dr  version   it s easy when there s at most one subprocess PIPE  otherwise it s hard   It may be time to explain a bit about how subprocess Popen does its thing    Caveat  this is for Python 2 x  although 3 x is similar  and I m quite fuzzy on the Windows variant   I understand the POSIX stuff much better    The Popen function needs to deal with zero-to-three I O streams  somewhat simultaneously   These are denoted stdin  stdout  and stderr as usual   You can provide    None  indicating that you don t want to redirect the stream   It will inherit these as usual instead   Note that on POSIX systems  at least  this does not mean it will use Python s sys stdout  just Python s actual stdout  see demo at end  An int value   This is a  raw  file descriptor  in POSIX at least     Side note  PIPE and STDOUT are actually ints internally  but are  impossible  descriptors  -1 and -2   A stream   really  any object with a fileno method   Popen will find the descriptor for that stream  using stream fileno    and then proceed as for an int value  subprocess PIPE  indicating that Python should create a pipe  subprocess STDOUT  for stderr only   tell Python to use the same descriptor as for stdout   This only makes sense if you provided a  non-None  value for stdout  and even then  it is only needed if you set stdout subprocess PIPE    Otherwise you can just provide the same argument you provided for stdout  e g   Popen      stdout stream  stderr stream      The easiest cases  no pipes   If you redirect nothing  leave all three as the default None value or supply explicit None   Pipe has it quite easy   It just needs to spin off the subprocess and let it run   Or  if you redirect to a non-PIPE   an int or a stream s fileno     it s still easy  as the OS does all the work   Python just needs to spin off the subprocess  connecting its stdin  stdout  and or stderr to the provided file descriptors   The still-easy case  one pipe  If you redirect only one stream  Pipe still has things pretty easy   Let s pick one stream at a time and watch   Suppose you want to supply some stdin  but let stdout and stderr go un-redirected  or go to a file descriptor   As the parent process  your Python program simply needs to use write   to send data down the pipe   You can do this yourself  e g    proc   subprocess Popen cmd  stdin subprocess PIPE  proc stdin write  here  have some data n     etc   or you can pass the stdin data to proc communicate    which then does the stdin write shown above   There is no output coming back so communicate   has only one other real job  it also closes the pipe for you    If you don t call proc communicate   you must call proc stdin close   to close the pipe  so that the subprocess knows there is no more data coming through    Suppose you want to capture stdout but leave stdin and stderr alone   Again  it s easy  just call proc stdout read    or equivalent  until there is no more output   Since proc stdout   is a normal Python I O stream you can use all the normal constructs on it  like   for line in proc stdout    or  again  you can use proc communicate    which simply does the read   for you   If you want to capture only stderr  it works the same as with stdout   There s one more trick before things get hard   Suppose you want to capture stdout  and also capture stderr but on the same pipe as stdout   proc   subprocess Popen cmd  stdout subprocess PIPE  stderr subprocess STDOUT    In this case  subprocess  cheats    Well  it has to do this  so it s not really cheating  it starts the subprocess with both its stdout and its stderr directed into the  single  pipe-descriptor that feeds back to its parent  Python  process   On the parent side  there s again only a single pipe-descriptor for reading the output   All the  stderr  output shows up in proc stdout  and if you call proc communicate    the stderr result  second value in the tuple  will be None  not a string   The hard cases  two or more pipes  The problems all come about when you want to use at least two pipes   In fact  the subprocess code itself has this bit   def communicate self  input None                 Optimization  If we are only using one pipe  or no pipe at       all  using select   or threads is unnecessary      if  self stdin  self stdout  self stderr  count None   gt   2    But  alas  here we ve made at least two  and maybe three  different pipes  so the count None  returns either 1 or 0   We must do things the hard way   On Windows  this uses threading Thread to accumulate results for self stdout and self stderr  and has the parent thread deliver self stdin input data  and then close the pipe    On POSIX  this uses poll if available  otherwise select  to accumulate output and deliver stdin input   All this runs in the  single  parent process thread   Threads or poll select are needed here to avoid deadlock   Suppose  for instance  that we ve redirected all three streams to three separate pipes   Suppose further that there s a small limit on how much data can be stuffed into to a pipe before the writing process is suspended  waiting for the reading process to  clean out  the pipe from the other end   Let s set that small limit to a single byte  just for illustration    This is in fact how things work  except that the limit is much bigger than one byte    If the parent  Python  process tries to write several bytes   say   go n to proc stdin  the first byte goes in and then the second causes the Python process to suspend  waiting for the subprocess to read the first byte  emptying the pipe   Meanwhile  suppose the subprocess decides to print a friendly  Hello  Don t Panic   greeting   The H goes into its stdout pipe  but the e causes it to suspend  waiting for its parent to read that H  emptying the stdout pipe   Now we re stuck  the Python process is asleep  waiting to finish saying  go   and the subprocess is also asleep  waiting to finish saying  Hello  Don t Panic     The subprocess Popen code avoids this problem with threading-or-select poll   When bytes can go over the pipes  they go   When they can t  only a thread  not the whole process  has to sleep   or  in the case of select poll  the Python process waits simultaneously for  can write  or  data available   writes to the process s stdin only when there is room  and reads its stdout and or stderr only when data are ready   The proc communicate   code  actually  communicate where the hairy cases are handled  returns once all stdin data  if any  have been sent and all stdout and or stderr data have been accumulated   If you want to read both stdout and stderr on two different pipes  regardless of any stdin redirection   you will need to avoid deadlock too   The deadlock scenario here is different   it occurs when the subprocess writes something long to stderr while you re pulling data from stdout  or vice versa   but it s still there     The Demo  I promised to demonstrate that  un-redirected  Python subprocesses write to the underlying stdout  not sys stdout   So  here is some code   from cStringIO import StringIO import os import subprocess import sys  def show1        print  start show1      save   sys stdout     sys stdout   StringIO       print  sys stdout being buffered      proc   subprocess Popen   echo    hello        proc wait       in stdout   sys stdout getvalue       sys stdout   save     print  in buffer    in stdout  def show2        print  start show2      save   sys stdout     sys stdout   open os devnull   w       print  after redirect sys stdout      proc   subprocess Popen   echo    hello        proc wait       sys stdout   save  show1   show2     When run     python out py start show1 hello in buffer  sys stdout being buffered  start show2 hello   Note that the first routine will fail if you add stdout sys stdout  as a StringIO object has no fileno   The second will omit the hello if you add stdout sys stdout since sys stdout has been redirected to os devnull    If you redirect Python s file-descriptor-1  the subprocess will follow that redirection   The open os devnull   w   call produces a stream whose fileno   is greater than 2

User · Answer

If you re able to use third-party libraries  You might be able to use something like sarge  disclosure  I m its maintainer   This library allows non-blocking access to output streams from subprocesses - it s layered over the subprocess module

User · Answer

It looks like line-buffered output will work for you  in which case something like the following might suit   Caveat  it s untested    This will only give the subprocess s stdout in real time   If you want to have both stderr and stdout in real time  you ll have to do something more complex with select   proc   subprocess Popen run command  stdout subprocess PIPE  stderr subprocess PIPE  shell True  while proc poll   is None      line   proc stdout readline       print line     log file write line     n     Might still be data on stdout at this point   Grab any   remainder  for line in proc stdout read   split   n        print line     log file write line     n     Do whatever you want with proc stderr here

User · Answer

We can also use the default file iterator for reading stdout instead of using iter construct with readline      import subprocess import sys process   subprocess Popen your command  stdout subprocess PIPE  for line in process stdout      sys stdout write line

User · Answer

Based on all the above I suggest a slightly modified version  python3     while loop calling readline  The iter solution suggested seemed to block forever for me - Python 3  Windows 7  structered so handling of read data does not need to be duplicated after poll returns not-None stderr piped into stdout so both output outputs are read Added code to get exit value of cmd    Code   import subprocess proc   subprocess Popen cmd  shell True  stdout subprocess PIPE                          stderr subprocess STDOUT  universal newlines True  while True      rd   proc stdout readline       print rd  end        and whatever you want to do        if not rd     EOF         returncode   proc poll           if returncode is not None              break         time sleep 0 1     cmd closed stdout  but not exited yet    You may want to check on ReturnCode here

User · Answer

None of the Pythonic solutions worked for me  It turned out that proc stdout read   or similar may block forever   Therefore  I use tee like this   subprocess run    my long running binary 2 gt  amp 1   tee -a my log file txt  amp  amp  exit   PIPESTATUS    shell True  check True  executable   bin bash     This solution is convenient if you are already using shell True     PIPESTATUS  captures the success status of the entire command chain  only available in Bash   If I omitted the  amp  amp  exit   PIPESTATUS   then this would always return zero since tee never fails   unbuffer might be necessary for printing each line immediately into the terminal  instead of waiting way too long until the  pipe buffer  gets filled  However  unbuffer swallows the exit status of assert  SIG Abort      2 gt  amp 1 also logs stderror to the file

User · Answer

All of the above solutions I tried failed either to separate stderr and stdout output   multiple pipes  or blocked forever when the OS pipe buffer was full which happens when the command you are running outputs too fast  there is a warning for this on python poll   manual of subprocess   The only reliable way I found was through select  but this is a posix-only solution   import subprocess import sys import os import select   returns command exit status  stdout text  stderr text   rtoutput  show realtime output while running def run script cmd rtoutput 0       p   subprocess Popen cmd  shell True  stdout subprocess PIPE  stderr subprocess PIPE      poller   select poll       poller register p stdout  select POLLIN      poller register p stderr  select POLLIN       coutput        cerror        fdhup        fdhup p stdout fileno    0     fdhup p stderr fileno    0     while sum fdhup values     lt  len fdhup           try              r   poller poll 1          except select error  err              if err args 0     EINTR                  raise             r            for fd  flags in r              if flags  amp   select POLLIN   select POLLPRI                   c   os read fd  1024                  if rtoutput                      sys stdout write c                      sys stdout flush                   if fd    p stderr fileno                        cerror  c                 else                      coutput  c             else                  fdhup fd  1     return p poll    coutput strip    cerror strip

User · Answer

TLDR for Python 3  import subprocess import sys with open  test log    wb   as f       process   subprocess Popen your command  stdout subprocess PIPE      for c in iter lambda  process stdout read 1   b              sys stdout buffer write c          f buffer write c    You have two ways of doing this  either by creating an iterator from the read or readline  functions and do  import subprocess import sys with open  test log    w   as f     replace  w  with  wb  for Python 3     process   subprocess Popen your command  stdout subprocess PIPE      for c in iter lambda  process stdout read 1           replace    with b   for Python 3         sys stdout write c          f write c   or import subprocess import sys with open  test log    w   as f     replace  w  with  wb  for Python 3     process   subprocess Popen your command  stdout subprocess PIPE      for line in iter process stdout readline          replace    with b   for Python 3         sys stdout write line          f write line   Or you can create a reader and a writer file  Pass the writer to the Popen and read from the reader import io import time import subprocess import sys  filename    test log  with io open filename   wb   as writer  io open filename   rb   1  as reader      process   subprocess Popen command  stdout writer      while process poll   is None          sys stdout write reader read            time sleep 0 5        Read the remaining     sys stdout write reader read     This way you will have the data written in the test log as well as on the standard output  The only advantage of the file approach is that your code doesn t block  So you can do whatever you want in the meantime and read whenever you want from the reader in a non-blocking way  When you use PIPE  read and readline functions will block until either one character is written to the pipe or a line is written to the pipe respectively

User · Answer

In addition to all these answer  one simple approach could also be as follows  process   subprocess Popen your command  stdout subprocess PIPE   while process stdout readable        line   process stdout readline        if not line          break      print line strip     Loop through the readable stream as long as it s readable and if it gets an empty result  stop  The key here is that readline   returns a line  with  n at the end  as long as there s an output and empty if it s really at the end  Hope this helps someone

User · Answer

I think that the subprocess communicate method is a bit misleading  it actually fills the stdout and stderr that you specify in the subprocess Popen   Yet  reading from the subprocess PIPE that you can provide to the subprocess Popen s stdout and stderr parameters will eventually fill up OS pipe buffers and deadlock your app  especially if you ve multiple processes threads that must use subprocess    My proposed solution is to provide the stdout and stderr with files - and read the files  content instead of reading from the deadlocking PIPE  These files can be tempfile NamedTemporaryFile   - which can also be accessed for reading while they re being written into by subprocess communicate    Below is a sample usage           try              with ProcessRunner   python    task py    env os environ copy    seconds to wait 0 01  as process runner                  for out in process runner                      print out          catch ProcessError as e              print e error message              raise   And this is the source code which is ready to be used with as many comments as I could provide to explain what it does   If you re using python 2  please make sure to first install the latest version of the subprocess32 package from pypi    import os import sys import threading import time import tempfile import logging  if os name     posix  and sys version info 0   lt  3        Support python 2     import subprocess32 as subprocess else        Get latest and greatest from python 3     import subprocess  logger   logging getLogger   name      class ProcessError Exception          Base exception for errors related to running the process      class ProcessTimeout ProcessError          Error that will be raised when the process execution will exceed a timeout      class ProcessRunner object       def   init   self  args  env None  timeout None  bufsize -1  seconds to wait 0 25    kwargs                       Constructor facade to subprocess Popen that receives parameters which are more specifically required for the         Process Runner  This is a class that should be used as a context manager - and that provides an iterator         for reading captured output from subprocess communicate in near realtime           Example usage            try              with ProcessRunner   python   task file path   env os environ copy    seconds to wait 0 01  as process runner                  for out in process runner                      print out          catch ProcessError as e              print e error message              raise           param args  same as subprocess Popen          param env  same as subprocess Popen          param timeout  same as subprocess communicate          param bufsize  same as subprocess Popen          param seconds to wait  time to wait between each readline from the temporary file          param kwargs  same as subprocess Popen                     self  seconds to wait   seconds to wait         self  process has timed out   False         self  timeout   timeout         self  process done   False         self  std file handle   tempfile NamedTemporaryFile           self  process   subprocess Popen args  env env  bufsize bufsize                                           stdout self  std file handle  stderr self  std file handle    kwargs          self  thread   threading Thread target self  run process          self  thread daemon   True      def   enter   self           self  thread start           return self      def   exit   self  exc type  exc val  exc tb           self  thread join           self  std file handle close        def   iter   self             read all output from stdout file that subprocess communicate fills         with open self  std file handle name   r   as stdout                while process is alive  keep reading data             while not self  process done                  out   stdout readline                   out without trailing whitespaces   out rstrip                   if out without trailing whitespaces                        yield stdout data without trailing  n                     yield out without trailing whitespaces                 else                        if there is nothing to read  then please wait a tiny little bit                     time sleep self  seconds to wait                 this is a hack  terraform seems to write to buffer after process has finished             out   stdout read               if out                  yield out          if self  process has timed out              raise ProcessTimeout  Process has timed out            if self  process returncode    0              raise ProcessError  Process has failed        def  run process self           try                Start gathering information  stdout and stderr  from the opened process             self  process communicate timeout self  timeout                Graceful termination of the opened process             self  process terminate           except subprocess TimeoutExpired              self  process has timed out   True               Force termination of the opened process             self  process kill            self  process done   True       property     def return code self           return self  process returncode

User · Answer

Here is a class which I m using in one of my projects  It redirects output of a subprocess to the log  At first I tried simply overwriting the write-method but that doesn t work as the subprocess will never call it  redirection happens on filedescriptor level   So I m using my own pipe  similar to how it s done in the subprocess-module  This has the advantage of encapsulating all logging printing logic in the adapter and you can simply pass instances of the logger to Popen  subprocess Popen   path to binary   stderr   LogAdapter  foo     class LogAdapter threading Thread        def   init   self  logname  level   logging INFO           super     init             self log   logging getLogger logname          self readpipe  self writepipe   os pipe            logFunctions                 logging DEBUG  self log debug              logging INFO  self log info              logging WARN  self log warn              logging ERROR  self log warn                     try              self logFunction   logFunctions level          except KeyError              self logFunction   self log info      def fileno self            when fileno is called this indicates the subprocess is about to fork   gt  start thread         self start           return self writepipe      def finished self             If the write-filedescriptor is not closed this thread will        prevent the whole program from exiting  You can use this method        to clean up after the subprocess has terminated            os close self writepipe       def run self           inputFile   os fdopen self readpipe           while True              line   inputFile readline                if len line     0                   no new data was added                 break              self logFunction line strip      If you don t need logging but simply want to use print   you can obviously remove large portions of the code and keep the class shorter  You could also expand it by an   enter   and   exit   method and call finished in   exit   so that you could easily use it as context

User · Answer

Why not set stdout directly to sys stdout  And if you need to output to a log as well  then you can simply override the write method of f   import sys import subprocess  class SuperFile open   class          def write self  data           sys stdout write data          super SuperFile  self  write data   f   SuperFile  log txt   w           process   subprocess Popen command  stdout f  stderr f

User · Answer

Solution 1  Log stdout AND stderr concurrently in realtime A simple solution which logs both stdout AND stderr concurrently  line-by-line in realtime into a log file  import subprocess as sp from concurrent futures import ThreadPoolExecutor   def log popen pipe p  stdfile        with open  quot mylog txt quot    quot w quot   as f           while p poll   is None              f write stdfile readline                f flush              Write the rest from the buffer         f write stdfile read      with sp Popen   quot ls quot    stdout sp PIPE  stderr sp PIPE  text True  as p       with ThreadPoolExecutor 2  as pool          r1   pool submit log popen pipe  p  p stdout          r2   pool submit log popen pipe  p  p stderr          r1 result           r2 result    Solution 2  A function read popen pipes   that allows you to iterate over both pipes  stdout stderr   concurrently in realtime import subprocess as sp from queue import Queue  Empty from concurrent futures import ThreadPoolExecutor   def enqueue output file  queue       for line in iter file readline               queue put line      file close     def read popen pipes p        with ThreadPoolExecutor 2  as pool          q stdout  q stderr   Queue    Queue            pool submit enqueue output  p stdout  q stdout          pool submit enqueue output  p stderr  q stderr           while True               if p poll   is not None and q stdout empty   and q stderr empty                    break              out line   err line                   try                  out line   q stdout get nowait                   err line   q stderr get nowait               except Empty                  pass              yield  out line  err line     The function in use   with sp Popen   quot ls quot    stdout sp PIPE  stderr sp PIPE  text True  as p       for out line  err line in read popen pipes p           print out line  end             print err line  end          p poll

User · Answer

A good but  heavyweight  solution is to use Twisted - see the bottom   If you re willing to live with only stdout something along those lines should work   import subprocess import sys popenobj   subprocess Popen   ls    -Rl    stdout subprocess PIPE  while not popenobj poll       stdoutdata   popenobj stdout readline      if stdoutdata        sys stdout write stdoutdata     else        break print  Return code   popenobj returncode    If you use read   it tries to read the entire  file  which isn t useful  what we really could use here is something that reads all the data that s in the pipe right now   One might also try to approach this with threading  e g    import subprocess import sys import threading  popenobj   subprocess Popen  ls   stdout subprocess PIPE  shell True   def stdoutprocess o      while True        stdoutdata   o stdout readline         if stdoutdata           sys stdout write stdoutdata        else           break  t   threading Thread target stdoutprocess  args  popenobj    t start   popenobj wait   t join   print  Return code   popenobj returncode   Now we could potentially add stderr as well by having two threads   Note however the subprocess docs discourage using these files directly and recommends to use communicate    mostly concerned with deadlocks which I think isn t an issue above  and the solutions are a little klunky so it really seems like the subprocess module isn t quite up to the job  also see  http   www python org dev peps pep-3145    and we need to look at something else   A more involved solution is to use Twisted as shown here  https   twistedmatrix com documents 11 1 0 core howto process html  The way you do this with Twisted is to create your process using reactor spawnprocess   and providing a ProcessProtocol that then processes output asynchronously   The Twisted sample Python code is here  https   twistedmatrix com documents 11 1 0 core howto listings process process py

[python] live output from subprocess command

Examples related to python

Examples related to shell

Examples related to logging

Examples related to error-handling

Examples related to subprocess