read subprocess stdout line by line

Question

My python script uses subprocess to call a linux utility that is very noisy   I want to store all of the output to a log file and show some of it to the user   I thought the following would work  but the output doesn t show up in my application until the utility has produced a significant amount of output    fake utility py  just generates lots of output over time import time i   0 while True     print hex i  512    i    1    time sleep 0 5    filters output import subprocess proc   subprocess Popen   python   fake utility py   stdout subprocess PIPE  for line in proc stdout      the real code does filtering here    print  test    line rstrip     The behavior I really want is for the filter script to print each line as it is received from the subprocess   Sorta like what tee does but with python code   What am I missing   Is this even possible     Update   If a sys stdout flush   is added to fake utility py  the code has the desired behavior in python 3 1   I m using python 2 6   You would think that using proc stdout xreadlines   would work the same as py3k  but it doesn t     Update 2   Here is the minimal working code    fake utility py  just generates lots of output over time import sys  time for i in range 10      print i    sys stdout flush      time sleep 0 5    display out put line by line import subprocess proc   subprocess Popen   python   fake utility py   stdout subprocess PIPE   works in python 3 0   for line in proc stdout  for line in iter proc stdout readline         print line rstrip

User · Answer

You want to pass these extra parameters to subprocess Popen   bufsize 1  universal newlines True   Then you can iterate as in your example    Tested with Python 3 5

User · Answer

Bit late to the party  but was surprised not to see what I think is the simplest solution here   import io import subprocess  proc   subprocess Popen   prog    arg    stdout subprocess PIPE  for line in io TextIOWrapper proc stdout  encoding  utf-8       or another encoding       do something with line    This requires Python 3

User · Answer

Indeed  if you sorted out the iterator then buffering could now be your problem  You could tell the python in the sub-process not to buffer its output   proc   subprocess Popen   python   fake utility py   stdout subprocess PIPE    becomes   proc   subprocess Popen   python   -u    fake utility py   stdout subprocess PIPE    I have needed this when calling python from within python

User · Answer

The following modification of R  mulo s answer works for me on Python 2 and 3  2 7 12 and 3 6 1    import os import subprocess  process   subprocess Popen command  stdout subprocess PIPE  while True    line   process stdout readline     if line            os write 1  line    else      break

User · Answer

I tried this with python3 and it worked  source  def output reader proc       for line in iter proc stdout readline  b             print  got line   0   format line decode  utf-8     end       def main        proc   subprocess Popen   python    fake utility py                                stdout subprocess PIPE                              stderr subprocess STDOUT       t   threading Thread target output reader  args  proc        t start        try          time sleep 0 2          import time         i   0          while True          print  hex i  512          i    1         time sleep 0 5      finally          proc terminate           try              proc wait timeout 0 2              print     subprocess exited with rc     proc returncode          except subprocess TimeoutExpired              print  subprocess did not terminate in time       t join

User · Answer

Pythont 3 5 added the methods run   and call   to the subprocess module  both returning a CompletedProcess object  With this you are fine using proc stdout splitlines    proc   subprocess run  comman  shell True  capture output True  text True  check True   for line in proc stdout splitlines       print  quot stdout  quot   line  See also How to Execute Shell Commands in Python Using the Subprocess Run Method

User · Answer

A function that allows iterating over both stdout and stderr concurrently  in realtime  line by line  In case you need to get the output stream for both stdout and stderr at the same time  you can use the following function   The function uses Queues to merge both Popen pipes into a single iterator   Here we create the function read popen pipes     from queue import Queue  Empty from concurrent futures import ThreadPoolExecutor   def enqueue output file  queue       for line in iter file readline               queue put line      file close     def read popen pipes p        with ThreadPoolExecutor 2  as pool          q stdout  q stderr   Queue    Queue            pool submit enqueue output  p stdout  q stdout          pool submit enqueue output  p stderr  q stderr           while True               if p poll   is not None and q stdout empty   and q stderr empty                    break              out line   err line                   try                  out line   q stdout get nowait               except Empty                  pass             try                  err line   q stderr get nowait               except Empty                  pass              yield  out line  err line    read popen pipes   in use   import subprocess as sp   with sp Popen my cmd  stdout sp PIPE  stderr sp PIPE  text True  as p       for out line  err line in read popen pipes p              Do stuff with each line  e g           print out line  end             print err line  end          return p poll     return status-code

User · Answer

You can also read lines w o loop  Works in python3 6   import os import subprocess  process   subprocess Popen command  stdout subprocess PIPE  list of byte strings   process stdout readlines

User · Answer

It s been a long time since I last worked with Python  but I think the problem is with the statement for line in proc stdout  which reads the entire input before iterating over it  The solution is to use readline   instead    filters output import subprocess proc   subprocess Popen   python   fake utility py   stdout subprocess PIPE  while True    line   proc stdout readline     if not line      break    the real code does filtering here   print  test    line rstrip     Of course you still have to deal with the subprocess  buffering   Note  according to the documentation the solution with an iterator should be equivalent to using readline    except for the read-ahead buffer  but  or exactly because of this  the proposed change did produce different results for me  Python 2 5 on Windows XP

[python] read subprocess stdout line by line

Examples related to python

Examples related to subprocess