Getting realtime output using subprocess

Question

I am trying to write a wrapper script for a command line program  svnadmin verify  that will display a nice progress indicator for the operation   This requires me to be able to see each line of output from the wrapped program as soon as it is output   I figured that I d just execute the program using subprocess Popen  use stdout PIPE  then read each line as it came in and act on it accordingly   However  when I ran the following code  the output appeared to be buffered somewhere  causing it to appear in two chunks  lines 1 through 332  then 333 through 439  the last line of output   from subprocess import Popen  PIPE  STDOUT  p   Popen  svnadmin verify  var svn repos config   stdout   PIPE           stderr   STDOUT  shell   True  for line in p stdout      print line replace   n         After looking at the documentation on subprocess a little  I discovered the bufsize parameter to Popen  so I tried setting bufsize to 1  buffer each line  and 0  no buffer   but neither value seemed to change the way the lines were being delivered   At this point I was starting to grasp for straws  so I wrote the following output loop   while True      try          print p stdout next   replace   n           except StopIteration          break   but got the same result   Is it possible to get  realtime  program output of a program executed using subprocess   Is there some other option in Python that is forward-compatible  not exec

User · Answer

I ran into the same problem awhile back   My solution was to ditch iterating for the read method  which will return immediately even if your subprocess isn t finished executing  etc

User · Answer

if you just want to forward the log to console in realtime Below code will work for both  p   subprocess Popen cmd                           shell True                           cwd work dir                           bufsize 1                           stdin subprocess PIPE                           stderr sys stderr                           stdout sys stdout

User · Answer

The Streaming subprocess stdin and stdout with asyncio in Python blog post by Kevin McCarthy shows how to do it with asyncio   import asyncio from asyncio subprocess import PIPE from asyncio import create subprocess exec   async def  read stream stream  callback       while True          line   await stream readline           if line              callback line          else              break   async def run command       process   await create subprocess exec           command  stdout PIPE  stderr PIPE            await asyncio wait                         read stream                  process stdout                  lambda x  print                       STDOUT      format x decode  UTF8                                                   read stream                  process stderr                  lambda x  print                       STDERR      format x decode  UTF8                                                           await process wait     async def main        await run  docker build -t my-docker-image latest       if   name         main         loop   asyncio get event loop       loop run until complete main

User · Answer

Found this  plug-and-play  function here  Worked like a charm   import subprocess  def myrun cmd          from http   blog kagesenshi org 2008 02 teeing-python-subprocesspopen-output html             p   subprocess Popen cmd  shell True  stdout subprocess PIPE  stderr subprocess STDOUT      stdout          while True          line   p stdout readline           stdout append line          print line          if line       and p poll      None              break     return    join stdout

User · Answer

Using pexpect with non-blocking readlines will resolve this problem  It stems from the fact that pipes are buffered  and so your app s output is getting buffered by the pipe  therefore you can t get to that output until the buffer fills or the process dies

User · Answer

Complete solution   import contextlib import subprocess    Unix  Windows and old Macintosh end-of-line newlines      n     r n     r   def unbuffered proc  stream  stdout        stream   getattr proc  stream      with contextlib closing stream           while True              out                  last   stream read 1                Don t loop forever             if last       and proc poll   is not None                  break             while last not in newlines                    Don t loop forever                 if last       and proc poll   is not None                      break                 out append last                  last   stream read 1              out      join out              yield out  def example        cmd     ls    -l            proc   subprocess Popen          cmd          stdout subprocess PIPE          stderr subprocess STDOUT            Make all end-of-lines   n          universal newlines True            for line in unbuffered proc           print line  example

User · Answer

By setting the buffer size to 1  you essentially force the process to not buffer the output  p   subprocess Popen cmd  stdout subprocess PIPE  bufsize 1  for line in iter p stdout readline  b         print line  p stdout close   p wait

User · Answer

Real Time Output Issue resolved  I encountered a similar issue in Python  while capturing the real time output from C program   I added fflush stdout   in my C code  It worked for me  Here is the code  C program   include  lt stdio h gt  void main         int count   1      while  1                printf  quot  Count   d n quot   count             fflush stdout           sleep 1            Python program     usr bin python  import os  sys import subprocess   procExe   subprocess Popen  quot    count quot   shell True  stdout subprocess PIPE  stderr subprocess PIPE  universal newlines True   while procExe poll   is None      line   procExe stdout readline       print  quot Print  quot    line   Output  Print  Count  1 Print  Count  2 Print  Count  3

User · Answer

This is the basic skeleton that I always use for this  It makes it easy to implement timeouts and is able to deal with inevitable hanging processes   import subprocess import threading import Queue  def t read stdout process  queue          Read from stdout         for output in iter process stdout readline  b             queue put output       return  process   subprocess Popen   dir                               stdout subprocess PIPE                             stderr subprocess STDOUT                             bufsize 1                             cwd  C                                 shell True   queue   Queue Queue   t stdout   threading Thread target t read stdout  args  process  queue   t stdout daemon   True t stdout start    while process poll   is None or not queue empty        try          output   queue get timeout  5       except Queue Empty          continue      if not output          continue      print output    t stdout join

User · Answer

In Python 3 x the process might hang because the output is a byte array instead of a string  Make sure you decode it into a string    Starting from Python 3 6 you can do it using the parameter encoding in Popen Constructor  The complete example   process   subprocess Popen       my command       stdout subprocess PIPE      stderr subprocess STDOUT      shell True      encoding  utf-8       errors  replace     while True      realtime output   process stdout readline        if realtime output       and process poll   is not None          break      if realtime output          print realtime output strip    flush True    Note that this code redirects stderr to stdout and handles output errors

User · Answer

Few answers suggesting python 3 x or pthon 2 x   Below code will work for both    p   subprocess Popen cmd  shell True  stdout subprocess PIPE  stderr subprocess STDOUT       stdout          while True          line   p stdout readline           if not isinstance line   str                line   line decode  utf-8           stdout append line          print  line          if  line       and p poll      None               break

User · Answer

You can direct the subprocess output to the streams directly  Simplified example   subprocess run   ls    stderr sys stderr  stdout sys stdout

User · Answer

You can try this   import subprocess import sys  process   subprocess Popen      cmd  stdout subprocess PIPE  stderr subprocess PIPE    while True      out   process stdout read 1      if out       and process poll      None          break     if out                sys stdout write out          sys stdout flush     If you use readline instead of read  there will be some cases where the input message is not printed  Try it with a command the requires an inline input and see for yourself

User · Answer

I tried this  and for some reason while the code  for line in p stdout          buffers aggressively  the variant  while True    line   p stdout readline     if not line  break         does not  Apparently this is a known bug  http   bugs python org issue3907  The issue is now  Closed  as of Aug 29  2018

User · Answer

You may use an iterator over each byte in the output of the subprocess  This allows inline update  lines ending with   r  overwrite previous output line  from the subprocess   from subprocess import PIPE  Popen  command     my command    -my arg      Open pipe to subprocess subprocess   Popen command  stdout PIPE  stderr PIPE      read each byte of subprocess while subprocess poll   is None      for c in iter lambda  subprocess stdout read 1  if subprocess poll   is None else     b             c   c decode  ascii           sys stdout write c  sys stdout flush    if subprocess returncode    0      raise Exception  The subprocess did not terminate correctly

User · Answer

This solution has been tested with Python 2 7 15  You just need to sys stdout flush   after each line read write   while proc poll   is None      line   proc stdout readline       sys stdout write line        or print line strip     you still need to force the flush      sys stdout flush

User · Answer

Depending on the use case  you might also want to disable the buffering in the subprocess itself   If the subprocess will be a Python process  you could do this before the call   os environ  PYTHONUNBUFFERED      1    Or alternatively pass this in the env argument to Popen   Otherwise  if you are on Linux Unix  you can use the stdbuf tool  E g  like   cmd     stdbuf    -oL     cmd   See also here about stdbuf or other options    See also here for the same answer

User · Answer

I used this solution to get realtime output on a subprocess  This loop will stop as soon as the process completes leaving out a need for a break statement or possible infinite loop    sub process   subprocess Popen my command  close fds True  shell True  stdout subprocess PIPE  stderr subprocess PIPE   while sub process poll   is None      out   sub process stdout read 1      sys stdout write out      sys stdout flush

[python] Getting realtime output using subprocess

Examples related to python

Examples related to subprocess