Constantly print Subprocess output while process is running

Question

To launch programs from my Python-scripts  I m using the following method  def execute command       process   subprocess Popen command  shell True  stdout subprocess PIPE  stderr subprocess STDOUT      output   process communicate   0      exitCode   process returncode      if  exitCode    0           return output     else          raise ProcessException command  exitCode  output   So when i launch a process like Process execute  quot mvn clean install quot    my program waits until the process is finished  and only then i get the complete output of my program  This is annoying if i m running a process that takes a while to finish  Can I let my program write the process output line by line  by polling the process output before it finishes in a loop or something  I found this article which might be related

User · Accepted Answer

You can use iter to process lines as soon as the command outputs them  lines   iter fd readline       Here s a full example showing a typical use case  thanks to  jfs for helping out    from   future   import print function   Only Python 2 x import subprocess  def execute cmd       popen   subprocess Popen cmd  stdout subprocess PIPE  universal newlines True      for stdout line in iter popen stdout readline               yield stdout line      popen stdout close       return code   popen wait       if return code          raise subprocess CalledProcessError return code  cmd     Example for path in execute   locate    a         print path  end

User · Answer

This PoC constantly reads the output from a process and can be accessed when needed  Only the last result is kept  all other output is discarded  hence prevents the PIPE from growing out of memory   import subprocess import time import threading import Queue   class FlushPipe object       def   init   self           self command     python      print date py           self process   None         self process output   Queue LifoQueue 0          self capture output   threading Thread target self output reader       def output reader self           for line in iter self process stdout readline  b                 self process output put nowait line       def start process self           self process   subprocess Popen self command                                          stdout subprocess PIPE          self capture output start        def get output for processing self           line   self process output get           print   gt  gt  gt     line   if   name         main         flush pipe   FlushPipe       flush pipe start process        now   time time       while time time   - now  lt  10          flush pipe get output for processing           time sleep 2 5       flush pipe capture output join timeout 0 001      flush pipe process kill     print date py     usr bin env python import time  if   name         main         while True          print str time time            time sleep 0 01    output  You can clearly see that there is only output from  2 5s interval nothing in between    gt  gt  gt 1520535158 51  gt  gt  gt 1520535161 01  gt  gt  gt 1520535163 51  gt  gt  gt 1520535166 01

User · Answer

In case someone wants to read from both stdout and stderr at the same time using threads  this is what I came up with   import threading import subprocess import Queue  class AsyncLineReader threading Thread       def   init   self  fd  outputQueue           threading Thread   init   self           assert isinstance outputQueue  Queue Queue          assert callable fd readline           self fd   fd         self outputQueue   outputQueue      def run self           map self outputQueue put  iter self fd readline            def eof self           return not self is alive   and self outputQueue empty         classmethod     def getForFd cls  fd  start True           queue   Queue Queue           reader   cls fd  queue           if start              reader start            return reader  queue   process   subprocess Popen command  shell True  stdout subprocess PIPE  stderr subprocess PIPE   stdoutReader  stdoutQueue    AsyncLineReader getForFd process stdout   stderrReader  stderrQueue    AsyncLineReader getForFd process stderr     Keep checking queues until there is no more output  while not stdoutReader eof   or not stderrReader eof         Process all available lines from the stdout Queue     while not stdoutQueue empty           line   stdoutQueue get          print  Received stdout      repr line            Do stuff with stdout line        Process all available lines from the stderr Queue     while not stderrQueue empty           line   stderrQueue get          print  Received stderr      repr line            Do stuff with stderr line        Sleep for a short time to avoid excessive CPU use while waiting for data     sleep 0 05   print  Waiting for async readers to finish     stdoutReader join   stderrReader join      Close subprocess  file descriptors  process stdout close   process stderr close    print  Waiting for process to exit     returnCode   process wait    if returnCode    0     raise subprocess CalledProcessError returnCode  command    I just wanted to share this  as I ended up on this question trying to do something similar  but none of the answers solved my problem  Hopefully it helps someone   Note that in my use case  an external process kills the process that we Popen

User · Answer

Ok i managed to solve it without threads  any suggestions why using threads would be better are appreciated  by using a snippet from this question Intercepting stdout of a subprocess while it is running  def execute command       process   subprocess Popen command  shell True  stdout subprocess PIPE  stderr subprocess STDOUT         Poll process for new output until finished     while True          nextline   process stdout readline           if nextline       and process poll   is not None              break         sys stdout write nextline          sys stdout flush        output   process communicate   0      exitCode   process returncode      if  exitCode    0           return output     else          raise ProcessException command  exitCode  output

User · Answer

For anyone trying the answers to this question to get the stdout from a Python script note that Python buffers its stdout  and therefore it may take a while to see the stdout   This can be rectified by adding the following after each stdout write in the target script   sys stdout flush

User · Answer

This works at least in Python3 4  import subprocess  process   subprocess Popen cmd list  stdout subprocess PIPE  for line in process stdout      print line decode   strip

User · Answer

tokland  tried your code and corrected it for 3 4 and windows dir cmd is a simple dir command  saved as cmd-file  import subprocess c    dir cmd   def execute command       popen   subprocess Popen command  stdout subprocess PIPE bufsize 1      lines iterator   iter popen stdout readline  b        while popen poll   is None          for line in lines iterator              nline   line rstrip               print nline decode  latin    end     r n  flush  True    yield line  execute c

User · Answer

In Python    3 5 using subprocess run works for me   import subprocess  cmd    echo foo  sleep 1  echo foo  sleep 2  echo foo  subprocess run cmd  shell True     getting the output during execution also works without shell True  https   docs python org 3 library subprocess html subprocess run

User · Answer

There is actually a really simple way to do this when you just want to print the output  import subprocess import sys  def execute command       subprocess check call command  stdout sys stdout  stderr subprocess STDOUT   Here we re simply pointing the subprocess to our own stdout  and using existing succeed or exception api

User · Answer

In Python 3 6 I used this   import subprocess  cmd    command  output   subprocess call cmd  shell True  print process

User · Answer

None of the answers here addressed all of my needs     No threads for stdout  no Queues  etc  either  Non-blocking as I need to check for other things going on Use PIPE as I needed to do multiple things  e g  stream output  write to a log file and return a string copy of the output    A little background  I am using a ThreadPoolExecutor to manage a pool of threads  each launching a subprocess and running them concurrency   In Python2 7  but this should work in newer 3 x as well   I don t want to use threads just for output gathering as I want as many available as possible for other things  a pool of 20 processes would be using 40 threads just to run  1 for the process thread and 1 for stdout   and more if you want stderr I guess   I m stripping back a lot of exception and such here so this is based on code that works in production  Hopefully I didn t ruin it in the copy and paste  Also  feedback very much welcome   import time import fcntl import subprocess import time  proc   subprocess Popen cmd  stdout subprocess PIPE  stderr subprocess STDOUT     Make stdout non-blocking when using read readline proc stdout   proc stdout fl   fcntl fcntl proc stdout  fcntl F GETFL  fcntl fcntl proc stdout  fcntl F SETFL  fl   os O NONBLOCK   def handle stdout proc stream  my buffer  echo streams True  log file None          A little inline function to handle the stdout business            fcntl makes readline non-blocking so it raises an IOError when empty     try          for s in iter proc stream readline           replace    with b   for Python 3             my buffer append s               if echo streams                  sys stdout write s               if log file                  log file write s      except IOError          pass    The main loop while subprocess is running stdout parts      while proc poll   is None      handle stdout proc stdout  stdout parts            Check for other things here          For example  check a multiprocessor Value  b   to proc kill        time sleep 0 01     Not sure if this is needed  but run it again just to be sure we got it all  handle stdout proc stdout  stdout parts   stdout str      join stdout parts     Just to demo   I m sure there is overhead being added here but it is not a concern in my case  Functionally it does what I need  The only thing I haven t solved is why this works perfectly for log messages but I see some print messages show up later and all at once

User · Answer

To answer the original question  the best way IMO is just redirecting subprocess stdout directly to your program s stdout  optionally  the same can be done for stderr  as in example below   p   Popen cmd  stdout sys stdout  stderr sys stderr  p communicate

User · Answer

To print subprocess  output line-by-line as soon as its stdout buffer is flushed in Python 3   from subprocess import Popen  PIPE  CalledProcessError  with Popen cmd  stdout PIPE  bufsize 1  universal newlines True  as p      for line in p stdout          print line  end       process line here  if p returncode    0      raise CalledProcessError p returncode  p args    Notice  you do not need p poll   -- the loop ends when eof is reached  And you do not need iter p stdout readline      -- the read-ahead bug is fixed in Python 3   See also  Python  read streaming input from subprocess communicate

[python] Constantly print Subprocess output while process is running

Examples related to python

Examples related to subprocess