A non-blocking read on a subprocess PIPE in Python

Question

I m using the subprocess module to start a subprocess and connect to its output stream  standard output   I want to be able to execute non-blocking reads on its standard output  Is there a way to make  readline non-blocking or to check if there is data on the stream before I invoke  readline  I d like this to be portable or at least work under Windows and Linux  Here is how I do it for now  it s blocking on the  readline if no data is available   p   subprocess Popen  myprogram exe   stdout   subprocess PIPE  output str   p stdout readline

User · Answer

My problem is a bit different as I wanted to collect both stdout and stderr from a running process, but ultimately the same since I wanted to render the output in a widget as its generated.

I did not want to resort to many of the proposed workarounds using Queues or additional Threads as they should not be necessary to perform such a common task as running another script and collecting its output.

After reading the proposed solutions and python docs I resolved my issue with the implementation below. Yes it only works for POSIX as I'm using the select function call.

I agree that the docs are confusing and the implementation is awkward for such a common scripting task. I believe that older versions of python have different defaults for Popen and different explanations so that created a lot of confusion. This seems to work well for both Python 2.7.12 and 3.5.2.

The key was to set bufsize=1 for line buffering and then universal_newlines=True to process as a text file instead of a binary which seems to become the default when setting bufsize=1.

class workerThread(QThread):
   def __init__(self, cmd):
      QThread.__init__(self)
      self.cmd = cmd
      self.result = None           ## return code
      self.error = None            ## flag indicates an error
      self.errorstr = ""           ## info message about the error

   def __del__(self):
      self.wait()
      DEBUG("Thread removed")

   def run(self):
      cmd_list = self.cmd.split(" ")   
      try:
         cmd = subprocess.Popen(cmd_list, bufsize=1, stdin=None
                                        , universal_newlines=True
                                        , stderr=subprocess.PIPE
                                        , stdout=subprocess.PIPE)
      except OSError:
         self.error = 1
         self.errorstr = "Failed to execute " + self.cmd
         ERROR(self.errorstr)
      finally:
         VERBOSE("task started...")
      import select
      while True:
         try:
            r,w,x = select.select([cmd.stdout, cmd.stderr],[],[])
            if cmd.stderr in r:
               line = cmd.stderr.readline()
               if line != "":
                  line = line.strip()
                  self.emit(SIGNAL("update_error(QString)"), line)
            if cmd.stdout in r:
               line = cmd.stdout.readline()
               if line == "":
                  break
               line = line.strip()
               self.emit(SIGNAL("update_output(QString)"), line)
         except IOError:
            pass
      cmd.wait()
      self.result = cmd.returncode
      if self.result < 0:
         self.error = 1
         self.errorstr = "Task terminated by signal " + str(self.result)
         ERROR(self.errorstr)
         return
      if self.result:
         self.error = 1
         self.errorstr = "exit code " + str(self.result)
         ERROR(self.errorstr)
         return
      return

ERROR, DEBUG and VERBOSE are simply macros that print output to the terminal.

This solution is IMHO 99.99% effective as it still uses the blocking readline function, so we assume the sub process is nice and outputs complete lines.

I welcome feedback to improve the solution as I am still new to Python.

User · Answer

Working from J F  Sebastian s answer  and several other sources  I ve put together a simple subprocess manager  It provides the request non-blocking reading  as well as running several processes in parallel  It doesn t use any OS-specific call  that I m aware  and thus should work anywhere   It s available from pypi  so just pip install shelljob  Refer to the project page for examples and full docs

User · Answer

Try the asyncproc module  For example   import os from asyncproc import Process myProc   Process  myprogram app    while True        check to see if process has ended     poll   myProc wait os WNOHANG      if poll    None          break       print any new output     out   myProc read       if out                print out   The module takes care of all the threading as suggested by S Lott

User · Answer

I also faced the problem described by Jesse and solved it by using  select  as Bradley  Andy and others did but in a blocking mode to avoid a busy loop  It uses a dummy Pipe as a fake stdin  The select blocks and wait for either stdin or the pipe to be ready  When a key is pressed stdin unblocks the select and the key value can be retrieved with read 1   When a different thread writes to the pipe then the pipe unblocks the select and it can be taken as an indication that the need for stdin is over  Here is some reference code   import sys import os from select import select    -------------------------------------------------------------------------       Set the pipe  fake stdin  to simulate a final key stroke   which will unblock the select statement readEnd  writeEnd   os pipe   readFile   os fdopen readEnd  writeFile   os fdopen writeEnd   w      ------------------------------------------------------------------------- def getKey           Wait for stdin or pipe  fake stdin  to be ready     dr dw de   select  sys   stdin    readFile                  If stdin is the one ready then read it and return value     if sys   stdin   in dr          return sys   stdin   read 1      For Windows use ---- gt  getch   from module msvcrt        Must finish     else          return None    ------------------------------------------------------------------------- def breakStdinRead        writeFile write          writeFile flush      -------------------------------------------------------------------------   MAIN CODE    Get key stroke key   getKey      Keyboard input if key            do your stuff with the key value    Faked keystroke else            use of stdin finished    -------------------------------------------------------------------------   OTHER THREAD CODE  breakStdinRead

User · Answer

On Unix-like systems and Python 3 5  there s os set blocking which does exactly what it says   import os import time import subprocess  cmd    python3    -c    import time    print i   time sleep 1   for i in range 5    p   subprocess Popen cmd  stdout subprocess PIPE  os set blocking p stdout fileno    False  start   time time   while True        first iteration always produces empty byte string in non-blocking mode     for i in range 2               line   p stdout readline           print i  line          time sleep 0 5      if time time    gt  start   5          break p terminate     This outputs   1 b   2 b 0 n  1 b   2 b 1 n  1 b   2 b 2 n  1 b   2 b 3 n  1 b   2 b 4 n    With os set blocking commented it s   0 b 0 n  1 b 1 n  0 b 2 n  1 b 3 n  0 b 4 n  1 b

User · Answer

Here is a module that supports non-blocking reads and background writes in python   https   pypi python org pypi python-nonblock  Provides a function   nonblock read which will read data from the stream  if available  otherwise return an empty string  or None if the stream is closed on the other side and all possible data has been read   You may also consider the python-subprocess2 module   https   pypi python org pypi python-subprocess2  which adds to the subprocess module  So on the object returned from  subprocess Popen  is added an additional method  runInBackground  This starts a thread and returns an object which will automatically be populated as stuff is written to stdout stderr  without blocking your main thread   Enjoy

User · Answer

why bothering thread amp queue  unlike readline    BufferedReader read1   wont block waiting for  r n  it returns ASAP if there is any output coming in      usr bin python from subprocess import Popen  PIPE  STDOUT import io  def   main          try          p   Popen    ping    -n    3    127 0 0 1    stdin PIPE  stdout PIPE  stderr STDOUT       except  print  Popen failed    quit       sout   io open p stdout fileno     rb   closefd False      while True          buf   sout read1 1024          if len buf     0  break         print buf   if   name         main           main

User · Answer

Try the asyncproc module  For example   import os from asyncproc import Process myProc   Process  myprogram app    while True        check to see if process has ended     poll   myProc wait os WNOHANG      if poll    None          break       print any new output     out   myProc read       if out                print out   The module takes care of all the threading as suggested by S Lott

User · Answer

Try wexpect  which is the windows alternative of pexpect   import wexpect  p   wexpect spawn  myprogram exe   p stdout readline                       regex pattern of any character output str   p after

User · Answer

Existing solutions did not work for me  details below   What finally worked was to implement readline using read 1   based on this answer   The latter does not block   from subprocess import Popen  PIPE from threading import Thread def process output myprocess    output-consuming thread     nextline   None     buf          while True           --- extract line using read 1          out   myprocess stdout read 1          if out       and myprocess poll      None  break         if out                    buf    out             if out      n                   nextline   buf                 buf              if not nextline  continue         line   nextline         nextline   None           --- do whatever you want with line here         print  Line is    line     myprocess stdout close    myprocess   Popen  myprogram exe   stdout PIPE   output-producing process p1   Thread target process output  args  dcmpid     output-consuming thread p1 daemon   True p1 start     --- do whatever here and then kill process and thread if needed if myprocess poll      None   kill process  will automatically stop thread     myprocess kill       myprocess wait   if p1 and p1 is alive     wait for thread to finish     p1 join     Why existing solutions did not work    Solutions that require readline  including the Queue based ones  always block  It is difficult  impossible   to kill the thread that executes readline  It only gets killed when the process that created it finishes  but not when the output-producing process is killed  Mixing low-level fcntl with high-level readline calls may not work properly as anonnn has pointed out  Using select poll   is neat  but doesn t work on Windows according to python docs  Using third-party libraries seems overkill for this task and adds additional dependencies

User · Answer

fcntl  select  asyncproc won t help in this case   A reliable way to read a stream without blocking regardless of operating system is to use Queue get nowait     import sys from subprocess import PIPE  Popen from threading  import Thread  try      from queue import Queue  Empty except ImportError      from Queue import Queue  Empty    python 2 x  ON POSIX    posix  in sys builtin module names  def enqueue output out  queue       for line in iter out readline  b             queue put line      out close    p   Popen   myprogram exe    stdout PIPE  bufsize 1  close fds ON POSIX  q   Queue   t   Thread target enqueue output  args  p stdout  q   t daemon   True   thread dies with the program t start          do other things here    read line without blocking try   line   q get nowait     or q get timeout  1  except Empty      print  no output yet   else    got line           do something with line

User · Answer

I have often had a similar problem  Python programs I write frequently need to have the ability to execute some primary functionality while simultaneously accepting user input from the command line  stdin   Simply putting the user input handling functionality in another thread doesn t solve the problem because readline   blocks and has no timeout  If the primary functionality is complete and there is no longer any need to wait for further user input I typically want my program to exit  but it can t because readline   is still blocking in the other thread waiting for a line  A solution I have found to this problem is to make stdin a non-blocking file using the fcntl module   import fcntl import os import sys    make stdin a non-blocking file fd   sys stdin fileno   fl   fcntl fcntl fd  fcntl F GETFL  fcntl fcntl fd  fcntl F SETFL  fl   os O NONBLOCK     user input handling thread while mainThreadIsRunning        try  input   sys stdin readline         except  continue       handleInput input    In my opinion this is a bit cleaner than using the select or signal modules to solve this problem but then again it only works on UNIX

User · Answer

I also faced the problem described by Jesse and solved it by using  select  as Bradley  Andy and others did but in a blocking mode to avoid a busy loop  It uses a dummy Pipe as a fake stdin  The select blocks and wait for either stdin or the pipe to be ready  When a key is pressed stdin unblocks the select and the key value can be retrieved with read 1   When a different thread writes to the pipe then the pipe unblocks the select and it can be taken as an indication that the need for stdin is over  Here is some reference code   import sys import os from select import select    -------------------------------------------------------------------------       Set the pipe  fake stdin  to simulate a final key stroke   which will unblock the select statement readEnd  writeEnd   os pipe   readFile   os fdopen readEnd  writeFile   os fdopen writeEnd   w      ------------------------------------------------------------------------- def getKey           Wait for stdin or pipe  fake stdin  to be ready     dr dw de   select  sys   stdin    readFile                  If stdin is the one ready then read it and return value     if sys   stdin   in dr          return sys   stdin   read 1      For Windows use ---- gt  getch   from module msvcrt        Must finish     else          return None    ------------------------------------------------------------------------- def breakStdinRead        writeFile write          writeFile flush      -------------------------------------------------------------------------   MAIN CODE    Get key stroke key   getKey      Keyboard input if key            do your stuff with the key value    Faked keystroke else            use of stdin finished    -------------------------------------------------------------------------   OTHER THREAD CODE  breakStdinRead

User · Answer

Here is my code  used to catch every output from subprocess ASAP  including partial lines  It pumps at same time and stdout and stderr in almost correct order   Tested and correctly worked on Python 2 7 linux  amp  windows       usr bin python     Runner with stdout stderr catcher   from sys import argv from subprocess import Popen  PIPE import os  io from threading import Thread import Queue def   main          if  len argv   gt  1  and  argv -1      -sub-            import time  sys         print  Application runned           time sleep 2          print  Slept 2 second          time sleep 1          print  Slept 1 additional second           time sleep 2          sys stderr write  Stderr output after 5 seconds           print  Eol on stdin          sys stderr write  Eol on stderr n           time sleep 1          print  Wow  we have end of work        else          os environ  PYTHONUNBUFFERED    1          try              p   Popen  argv     -sub-                           bufsize 0    line-buffered                        stdin PIPE  stdout PIPE  stderr PIPE           except WindowsError  W              if W winerror  193                  p   Popen  argv     -sub-                               shell True    Try to run via shell                            bufsize 0    line-buffered                            stdin PIPE  stdout PIPE  stderr PIPE               else                  raise         inp   Queue Queue           sout   io open p stdout fileno     rb   closefd False          serr   io open p stderr fileno     rb   closefd False          def Pump stream  category               queue   Queue Queue               def rdr                    while True                      buf   stream read1 8192                      if len buf  gt 0                          queue put  buf                       else                          queue put  None                           return             def clct                    active   True                 while active                      r   queue get                       try                          while True                              r1   queue get timeout 0 005                              if r1 is None                                  active   False                                 break                             else                                  r    r1                     except Queue Empty                          pass                     inp put   category  r                for tgt in  rdr  clct                   th   Thread target tgt                  th setDaemon True                  th start           Pump sout   stdout           Pump serr   stderr            while p poll   is None                App still working             try                  chan line   inp get timeout   1 0                  if chan   stdout                       print  STDOUT gt  gt    line    lt   lt                   elif chan   stderr                       print   ERROR     line                    except Queue Empty                  pass         print  Finish   if   name         main           main

User · Answer

Disclaimer  this works only for tornado  You can do this by setting the fd to be nonblocking and then use ioloop to register callbacks  I have packaged this in an egg called tornado subprocess and you can install it via PyPI   easy install tornado subprocess   now you can do something like this   import tornado subprocess import tornado ioloop      def print res  status  stdout  stderr         print status  stdout  stderr     if status    0          print  OK           print stdout     else          print  ERROR           print stderr  t   tornado subprocess Subprocess  print res  timeout 30  args    cat     etc passwd      t start   tornado ioloop IOLoop instance   start     you can also use it with a RequestHandler  class MyHandler tornado web RequestHandler       def on done self  status  stdout  stderr           self write  stdout           self finish         tornado web asynchronous     def get self           t   tornado subprocess Subprocess  self on done  timeout 30  args    cat     etc passwd              t start

User · Answer

EDIT  This implementation still blocks  Use J F Sebastian s answer instead   I tried the top answer  but the additional risk and maintenance of thread code was worrisome   Looking through the io module  and being limited to 2 6   I found BufferedReader  This is my threadless  non-blocking solution    import io from subprocess import PIPE  Popen  p   Popen   myprogram exe    stdout PIPE   SLEEP DELAY   0 001    Create an io BufferedReader on the file descriptor for stdout with io open p stdout fileno     rb   closefd False  as buffer    while p poll      None        time sleep SLEEP DELAY        while   n  in bufferedStdout peek bufferedStdout buffer size             line   buffer readline               do stuff with the line      Handle any remaining output after the process has ended   while buffer peek        line   buffer readline         do stuff with the line

User · Answer

One solution is to make another process to perform your read of the process  or make a thread of the process with a timeout   Here s the threaded version of a timeout function   http   code activestate com recipes 473878   However  do you need to read the stdout as it s coming in  Another solution may be to dump the output to a file and wait for the process to finish using p wait     f   open  myprogram output txt   w   p   subprocess Popen  myprogram exe   stdout f  p wait   f close     str   open  myprogram output txt   r   read

User · Answer

You can do this really easily in Twisted  Depending upon your existing code base  this might not be that easy to use  but if you are building a twisted application  then things like this become almost trivial  You create a ProcessProtocol class  and override the outReceived   method  Twisted  depending upon the reactor used  is usually just a big select   loop with callbacks installed to handle data from different file descriptors  often network sockets   So the outReceived   method is simply installing a callback for handling data coming from STDOUT   A simple example demonstrating this behavior is as follows   from twisted internet import protocol  reactor  class MyProcessProtocol protocol ProcessProtocol        def outReceived self  data           print data  proc   MyProcessProtocol   reactor spawnProcess proc     myprogram       myprogram    arg1    arg2    arg3    reactor run     The Twisted documentation has some good information on this   If you build your entire application around Twisted  it makes asynchronous communication with other processes  local or remote  really elegant like this  On the other hand  if your program isn t built on top of Twisted  this isn t really going to be that helpful  Hopefully this can be helpful to other readers  even if it isn t applicable for your particular application

User · Answer

I have the original questioner s problem  but did not wish to invoke threads  I mixed Jesse s solution with a direct read   from the pipe  and my own buffer-handler for line reads  however  my sub-process - ping - always wrote full lines  lt  a system page size   I avoid busy-waiting by only reading in a gobject-registered io watch  These days I usually run code within a gobject MainLoop to avoid threads  def set up ping ip  w         run the sub-process       watch the resultant pipe     p   subprocess Popen    bin ping   ip   stdout subprocess PIPE        make stdout a non-blocking file     fl   fcntl fcntl p stdout  fcntl F GETFL      fcntl fcntl p stdout  fcntl F SETFL  fl   os O NONBLOCK      stdout gid   gobject io add watch p stdout  gobject IO IN  w      return stdout gid   for shutting down  The watcher is def watch f   other       print  reading  f read       return True  And the main program sets up a ping and then calls gobject mail loop  def main        set up ping  192 168 1 8   watch        discard gid as unused here     gobject MainLoop   run    Any other work is attached to callbacks in gobject

User · Answer

Use select  amp  read 1      import subprocess      no new requirements def readAllSoFar proc  retVal         while  select select  proc stdout        0  0               retVal  proc stdout read 1    return retVal p   subprocess Popen    bin ls    stdout subprocess PIPE  while not p poll      print  readAllSoFar p     For readline  -like   lines        while not p poll      lines   readAllSoFar p  lines -1   split   n     for a in range len lines -1       print a lines   readAllSoFar p  lines -1   split   n   for a in range len lines -1     print a

User · Answer

Things are a lot better in modern Python   Here s a simple child program   hello py       usr bin env python3  while True      i   input       if i     quit           break     print f hello  i      And a program to interact with it   import asyncio   async def main        proc   await asyncio subprocess create subprocess exec             hello py   stdin asyncio subprocess PIPE  stdout asyncio subprocess PIPE           proc stdin write b bob n       print await proc stdout read 1024       proc stdin write b alice n       print await proc stdout read 1024       proc stdin write b quit n       await proc wait     asyncio run main      That prints out   b hello bob n  b hello alice n    Note that the actual pattern  which is also by almost all of the previous answers  both here and in related questions  is to set the child s stdout file descriptor to non-blocking  and then poll it in some sort of select loop  These days  of course  that loop is provided by asyncio

User · Answer

In my case I needed a logging module that catches the output from the background applications and augments it adding time-stamps  colors  etc     I ended up with a background thread that does the actual I O  Following code is only for POSIX platforms  I stripped non-essential parts    If someone is going to use this beast for long runs consider managing open descriptors  In my case it was not a big problem     - - python - - import fcntl import threading import sys  os  errno import subprocess  class Logger threading Thread       def   init   self   modules           threading Thread   init   self          try              from select import epoll  EPOLLIN             self   poll   epoll               self   evt   EPOLLIN             self   to   -1         except              from select import poll  POLLIN             print  epoll is not available              self   poll   poll               self   evt   POLLIN             self   to   100         self   fds              self daemon   True         self start        def run self           while True              events   self   poll poll self   to              for fd  ev in events                  if  ev amp self   evt     self   evt                      continue                 try                      self   fds fd  run                   except Exception  e                      print e      def add self  fd  log           assert not self   fds has key fd          self   fds fd    log         self   poll register fd  self   evt   class log      logger   Logger        def   init   self  name           self   name   name         self   piped   False      def fileno self           if self   piped              return self write         self read  self write   os pipe           fl   fcntl fcntl self read  fcntl F GETFL          fcntl fcntl self read  fcntl F SETFL  fl   os O NONBLOCK          self fdRead   os fdopen self read          self logger add self read  self          self   piped   True         return self write      def   run self  line           self chat line  nl False       def run self           while True              try  line   self fdRead readline               except IOError  exc                  if exc errno    errno EAGAIN                      return                 raise             self   run line       def chat self  line  nl True           if nl  nl     n          else  nl              sys stdout write    s   s s     self   name  line  nl    def system command  param     cwd None  env None  input None  output None       args    command    param     p   subprocess Popen args  cwd cwd  stdout output  stderr output  stdin input  env env  bufsize 0      p wait    ls   log  ls   ls chat  go   system  ls     -l         output ls   date   log  date   date chat  go   system  date   output date

User · Answer

Disclaimer  this works only for tornado  You can do this by setting the fd to be nonblocking and then use ioloop to register callbacks  I have packaged this in an egg called tornado subprocess and you can install it via PyPI   easy install tornado subprocess   now you can do something like this   import tornado subprocess import tornado ioloop      def print res  status  stdout  stderr         print status  stdout  stderr     if status    0          print  OK           print stdout     else          print  ERROR           print stderr  t   tornado subprocess Subprocess  print res  timeout 30  args    cat     etc passwd      t start   tornado ioloop IOLoop instance   start     you can also use it with a RequestHandler  class MyHandler tornado web RequestHandler       def on done self  status  stdout  stderr           self write  stdout           self finish         tornado web asynchronous     def get self           t   tornado subprocess Subprocess  self on done  timeout 30  args    cat     etc passwd              t start

User · Answer

This version of non-blocking read doesn t require special modules and will work out-of-the-box on majority of Linux distros   import os import sys import time import fcntl import subprocess  def async read fd         set non-blocking flag while preserving old flags     fl   fcntl fcntl fd  fcntl F GETFL      fcntl fcntl fd  fcntl F SETFL  fl   os O NONBLOCK        read char until EOF hit     while True          try              ch   os read fd fileno    1                EOF             if not ch  break                                                                                                                                                                           sys stdout write ch          except OSError                waiting for data be available on fd             pass  def shell args  async True         merge stderr and stdout     proc   subprocess Popen args  shell False  stdout subprocess PIPE  stderr subprocess STDOUT      if async  async read proc stdout      sout  serr   proc communicate       return  sout  serr   if   name         main         cmd    ping 8 8 8 8      sout  serr   shell cmd split

User · Answer

In my case I needed a logging module that catches the output from the background applications and augments it adding time-stamps  colors  etc     I ended up with a background thread that does the actual I O  Following code is only for POSIX platforms  I stripped non-essential parts    If someone is going to use this beast for long runs consider managing open descriptors  In my case it was not a big problem     - - python - - import fcntl import threading import sys  os  errno import subprocess  class Logger threading Thread       def   init   self   modules           threading Thread   init   self          try              from select import epoll  EPOLLIN             self   poll   epoll               self   evt   EPOLLIN             self   to   -1         except              from select import poll  POLLIN             print  epoll is not available              self   poll   poll               self   evt   POLLIN             self   to   100         self   fds              self daemon   True         self start        def run self           while True              events   self   poll poll self   to              for fd  ev in events                  if  ev amp self   evt     self   evt                      continue                 try                      self   fds fd  run                   except Exception  e                      print e      def add self  fd  log           assert not self   fds has key fd          self   fds fd    log         self   poll register fd  self   evt   class log      logger   Logger        def   init   self  name           self   name   name         self   piped   False      def fileno self           if self   piped              return self write         self read  self write   os pipe           fl   fcntl fcntl self read  fcntl F GETFL          fcntl fcntl self read  fcntl F SETFL  fl   os O NONBLOCK          self fdRead   os fdopen self read          self logger add self read  self          self   piped   True         return self write      def   run self  line           self chat line  nl False       def run self           while True              try  line   self fdRead readline               except IOError  exc                  if exc errno    errno EAGAIN                      return                 raise             self   run line       def chat self  line  nl True           if nl  nl     n          else  nl              sys stdout write    s   s s     self   name  line  nl    def system command  param     cwd None  env None  input None  output None       args    command    param     p   subprocess Popen args  cwd cwd  stdout output  stderr output  stdin input  env env  bufsize 0      p wait    ls   log  ls   ls chat  go   system  ls     -l         output ls   date   log  date   date chat  go   system  date   output date

User · Answer

The select module helps you determine where the next useful input is   However  you re almost always happier with separate threads   One does a blocking read the stdin  another does wherever it is you don t want blocked

User · Answer

Adding this answer here since it provides ability to set non-blocking pipes on Windows and Unix   All the ctypes details are thanks to  techtonik s answer   There is a slightly modified version to be used both on Unix and Windows systems    Python3 compatible  only minor change needed   Includes posix version  and defines exception to use for either    This way you can use the same function and exception for Unix and Windows code     pipe non blocking py  module      Example use       p   subprocess Popen              command              stdout subprocess PIPE                     pipe non blocking set p stdout fileno         try          data   os read p stdout fileno    1      except PortableBlockingIOError as ex          if not pipe non blocking is error blocking ex               raise ex         all            pipe non blocking set        pipe non blocking is error blocking        PortableBlockingIOError          import os   if os name     nt       def pipe non blocking set fd             Constant could define globally but avoid polluting the name-space           thanks to  https   stackoverflow com questions 34504970         import msvcrt          from ctypes import windll  byref  wintypes  WinError  POINTER         from ctypes wintypes import HANDLE  DWORD  BOOL          LPDWORD   POINTER DWORD           PIPE NOWAIT   wintypes DWORD 0x00000001           def pipe no wait pipefd               SetNamedPipeHandleState   windll kernel32 SetNamedPipeHandleState             SetNamedPipeHandleState argtypes    HANDLE  LPDWORD  LPDWORD  LPDWORD              SetNamedPipeHandleState restype   BOOL              h   msvcrt get osfhandle pipefd               res   windll kernel32 SetNamedPipeHandleState h  byref PIPE NOWAIT   None  None              if res    0                  print WinError                    return False             return True          return pipe no wait fd       def pipe non blocking is error blocking ex           if not isinstance ex  PortableBlockingIOError               return False         from ctypes import GetLastError         ERROR NO DATA   232          return  GetLastError      ERROR NO DATA       PortableBlockingIOError   OSError else      def pipe non blocking set fd           import fcntl         fl   fcntl fcntl fd  fcntl F GETFL          fcntl fcntl fd  fcntl F SETFL  fl   os O NONBLOCK          return True      def pipe non blocking is error blocking ex           if not isinstance ex  PortableBlockingIOError               return False         return True      PortableBlockingIOError   BlockingIOError   To avoid reading incomplete data  I ended up writing my own readline generator  which returns the byte string for each line    Its a generator so you can for example     def non blocking readlines f  chunk 1024               Iterate over lines  yielding b   when nothings left     or when new data is not yet available       stdout iter   iter non blocking readlines process stdout        line   next stdout iter     will be a line or b                import os      from  pipe non blocking import               pipe non blocking set              pipe non blocking is error blocking              PortableBlockingIOError                     fd   f fileno       pipe non blocking set fd       blocks           while True          try              data   os read fd  chunk              if not data                    case were reading finishes with no trailing newline                 yield b   join blocks                  blocks clear           except PortableBlockingIOError as ex              if not pipe non blocking is error blocking ex                   raise ex              yield b               continue          while True              n   data find b  n               if n    -1                  break              yield b   join blocks    data  n   1              data   data n   1               blocks clear           blocks append data

User · Answer

why bothering thread amp queue  unlike readline    BufferedReader read1   wont block waiting for  r n  it returns ASAP if there is any output coming in      usr bin python from subprocess import Popen  PIPE  STDOUT import io  def   main          try          p   Popen    ping    -n    3    127 0 0 1    stdin PIPE  stdout PIPE  stderr STDOUT       except  print  Popen failed    quit       sout   io open p stdout fileno     rb   closefd False      while True          buf   sout read1 1024          if len buf     0  break         print buf   if   name         main           main

User · Answer

Try wexpect  which is the windows alternative of pexpect   import wexpect  p   wexpect spawn  myprogram exe   p stdout readline                       regex pattern of any character output str   p after

User · Answer

The select module helps you determine where the next useful input is   However  you re almost always happier with separate threads   One does a blocking read the stdin  another does wherever it is you don t want blocked

User · Answer

Here is a simple solution based on threads which    works on both Linux and Windows  not relying on select   reads both stdout and stderr asynchronouly  doesn t rely on active polling with arbitrary waiting time  CPU friendly   doesn t use asyncio  which may conflict with other libraries   runs until the child process terminates    printer py  import time import sys  sys stdout write  Hello n   sys stdout flush   time sleep 1  sys stdout write  World  n   sys stdout flush   time sleep 1  sys stderr write  That s an error n   sys stderr flush   time sleep 2  sys stdout write  Actually  I m fine n   sys stdout flush   time sleep 1    reader py  import queue import subprocess import sys import threading   def enqueue stream stream  queue  type       for line in iter stream readline  b             queue put str type    line decode  utf-8        stream close     def enqueue process process  queue       process wait       queue put  x     p   subprocess Popen  python printer py   stdout subprocess PIPE  stderr subprocess PIPE  q   queue Queue   to   threading Thread target enqueue stream  args  p stdout  q  1   te   threading Thread target enqueue stream  args  p stderr  q  2   tp   threading Thread target enqueue process  args  p  q   te start   to start   tp start    while True      line   q get       if line 0      x           break     if line 0      2      stderr         sys stdout write   033 0 31m      ANSI red color     sys stdout write line 1        if line 0      2           sys stdout write   033 0m      reset ANSI code     sys stdout flush    tp join   to join   te join

User · Answer

EDIT  This implementation still blocks  Use J F Sebastian s answer instead   I tried the top answer  but the additional risk and maintenance of thread code was worrisome   Looking through the io module  and being limited to 2 6   I found BufferedReader  This is my threadless  non-blocking solution    import io from subprocess import PIPE  Popen  p   Popen   myprogram exe    stdout PIPE   SLEEP DELAY   0 001    Create an io BufferedReader on the file descriptor for stdout with io open p stdout fileno     rb   closefd False  as buffer    while p poll      None        time sleep SLEEP DELAY        while   n  in bufferedStdout peek bufferedStdout buffer size             line   buffer readline               do stuff with the line      Handle any remaining output after the process has ended   while buffer peek        line   buffer readline         do stuff with the line

User · Answer

This version of non-blocking read doesn t require special modules and will work out-of-the-box on majority of Linux distros   import os import sys import time import fcntl import subprocess  def async read fd         set non-blocking flag while preserving old flags     fl   fcntl fcntl fd  fcntl F GETFL      fcntl fcntl fd  fcntl F SETFL  fl   os O NONBLOCK        read char until EOF hit     while True          try              ch   os read fd fileno    1                EOF             if not ch  break                                                                                                                                                                           sys stdout write ch          except OSError                waiting for data be available on fd             pass  def shell args  async True         merge stderr and stdout     proc   subprocess Popen args  shell False  stdout subprocess PIPE  stderr subprocess STDOUT      if async  async read proc stdout      sout  serr   proc communicate       return  sout  serr   if   name         main         cmd    ping 8 8 8 8      sout  serr   shell cmd split

User · Answer

I have often had a similar problem  Python programs I write frequently need to have the ability to execute some primary functionality while simultaneously accepting user input from the command line  stdin   Simply putting the user input handling functionality in another thread doesn t solve the problem because readline   blocks and has no timeout  If the primary functionality is complete and there is no longer any need to wait for further user input I typically want my program to exit  but it can t because readline   is still blocking in the other thread waiting for a line  A solution I have found to this problem is to make stdin a non-blocking file using the fcntl module   import fcntl import os import sys    make stdin a non-blocking file fd   sys stdin fileno   fl   fcntl fcntl fd  fcntl F GETFL  fcntl fcntl fd  fcntl F SETFL  fl   os O NONBLOCK     user input handling thread while mainThreadIsRunning        try  input   sys stdin readline         except  continue       handleInput input    In my opinion this is a bit cleaner than using the select or signal modules to solve this problem but then again it only works on UNIX

User · Answer

Try the asyncproc module  For example   import os from asyncproc import Process myProc   Process  myprogram app    while True        check to see if process has ended     poll   myProc wait os WNOHANG      if poll    None          break       print any new output     out   myProc read       if out                print out   The module takes care of all the threading as suggested by S Lott

User · Answer

I add this problem to read some subprocess Popen stdout  Here is my non blocking read solution   import fcntl  def non block read output       fd   output fileno       fl   fcntl fcntl fd  fcntl F GETFL      fcntl fcntl fd  fcntl F SETFL  fl   os O NONBLOCK      try          return output read       except          return       Use example from subprocess import   sb   Popen  echo test  amp  amp  sleep 1000   shell True  stdout PIPE  sb kill      sb stdout read      lt -- This will block non block read sb stdout   test n

User · Answer

Here is a module that supports non-blocking reads and background writes in python   https   pypi python org pypi python-nonblock  Provides a function   nonblock read which will read data from the stream  if available  otherwise return an empty string  or None if the stream is closed on the other side and all possible data has been read   You may also consider the python-subprocess2 module   https   pypi python org pypi python-subprocess2  which adds to the subprocess module  So on the object returned from  subprocess Popen  is added an additional method  runInBackground  This starts a thread and returns an object which will automatically be populated as stuff is written to stdout stderr  without blocking your main thread   Enjoy

User · Answer

One solution is to make another process to perform your read of the process  or make a thread of the process with a timeout   Here s the threaded version of a timeout function   http   code activestate com recipes 473878   However  do you need to read the stdout as it s coming in  Another solution may be to dump the output to a file and wait for the process to finish using p wait     f   open  myprogram output txt   w   p   subprocess Popen  myprogram exe   stdout f  p wait   f close     str   open  myprogram output txt   r   read

User · Answer

fcntl  select  asyncproc won t help in this case   A reliable way to read a stream without blocking regardless of operating system is to use Queue get nowait     import sys from subprocess import PIPE  Popen from threading  import Thread  try      from queue import Queue  Empty except ImportError      from Queue import Queue  Empty    python 2 x  ON POSIX    posix  in sys builtin module names  def enqueue output out  queue       for line in iter out readline  b             queue put line      out close    p   Popen   myprogram exe    stdout PIPE  bufsize 1  close fds ON POSIX  q   Queue   t   Thread target enqueue output  args  p stdout  q   t daemon   True   thread dies with the program t start          do other things here    read line without blocking try   line   q get nowait     or q get timeout  1  except Empty      print  no output yet   else    got line           do something with line

User · Answer

Existing solutions did not work for me  details below   What finally worked was to implement readline using read 1   based on this answer   The latter does not block   from subprocess import Popen  PIPE from threading import Thread def process output myprocess    output-consuming thread     nextline   None     buf          while True           --- extract line using read 1          out   myprocess stdout read 1          if out       and myprocess poll      None  break         if out                    buf    out             if out      n                   nextline   buf                 buf              if not nextline  continue         line   nextline         nextline   None           --- do whatever you want with line here         print  Line is    line     myprocess stdout close    myprocess   Popen  myprogram exe   stdout PIPE   output-producing process p1   Thread target process output  args  dcmpid     output-consuming thread p1 daemon   True p1 start     --- do whatever here and then kill process and thread if needed if myprocess poll      None   kill process  will automatically stop thread     myprocess kill       myprocess wait   if p1 and p1 is alive     wait for thread to finish     p1 join     Why existing solutions did not work    Solutions that require readline  including the Queue based ones  always block  It is difficult  impossible   to kill the thread that executes readline  It only gets killed when the process that created it finishes  but not when the output-producing process is killed  Mixing low-level fcntl with high-level readline calls may not work properly as anonnn has pointed out  Using select poll   is neat  but doesn t work on Windows according to python docs  Using third-party libraries seems overkill for this task and adds additional dependencies

User · Answer

One solution is to make another process to perform your read of the process  or make a thread of the process with a timeout   Here s the threaded version of a timeout function   http   code activestate com recipes 473878   However  do you need to read the stdout as it s coming in  Another solution may be to dump the output to a file and wait for the process to finish using p wait     f   open  myprogram output txt   w   p   subprocess Popen  myprogram exe   stdout f  p wait   f close     str   open  myprogram output txt   r   read

User · Answer

My problem is a bit different as I wanted to collect both stdout and stderr from a running process, but ultimately the same since I wanted to render the output in a widget as its generated.

I did not want to resort to many of the proposed workarounds using Queues or additional Threads as they should not be necessary to perform such a common task as running another script and collecting its output.

After reading the proposed solutions and python docs I resolved my issue with the implementation below. Yes it only works for POSIX as I'm using the select function call.

I agree that the docs are confusing and the implementation is awkward for such a common scripting task. I believe that older versions of python have different defaults for Popen and different explanations so that created a lot of confusion. This seems to work well for both Python 2.7.12 and 3.5.2.

The key was to set bufsize=1 for line buffering and then universal_newlines=True to process as a text file instead of a binary which seems to become the default when setting bufsize=1.

class workerThread(QThread):
   def __init__(self, cmd):
      QThread.__init__(self)
      self.cmd = cmd
      self.result = None           ## return code
      self.error = None            ## flag indicates an error
      self.errorstr = ""           ## info message about the error

   def __del__(self):
      self.wait()
      DEBUG("Thread removed")

   def run(self):
      cmd_list = self.cmd.split(" ")   
      try:
         cmd = subprocess.Popen(cmd_list, bufsize=1, stdin=None
                                        , universal_newlines=True
                                        , stderr=subprocess.PIPE
                                        , stdout=subprocess.PIPE)
      except OSError:
         self.error = 1
         self.errorstr = "Failed to execute " + self.cmd
         ERROR(self.errorstr)
      finally:
         VERBOSE("task started...")
      import select
      while True:
         try:
            r,w,x = select.select([cmd.stdout, cmd.stderr],[],[])
            if cmd.stderr in r:
               line = cmd.stderr.readline()
               if line != "":
                  line = line.strip()
                  self.emit(SIGNAL("update_error(QString)"), line)
            if cmd.stdout in r:
               line = cmd.stdout.readline()
               if line == "":
                  break
               line = line.strip()
               self.emit(SIGNAL("update_output(QString)"), line)
         except IOError:
            pass
      cmd.wait()
      self.result = cmd.returncode
      if self.result < 0:
         self.error = 1
         self.errorstr = "Task terminated by signal " + str(self.result)
         ERROR(self.errorstr)
         return
      if self.result:
         self.error = 1
         self.errorstr = "exit code " + str(self.result)
         ERROR(self.errorstr)
         return
      return

ERROR, DEBUG and VERBOSE are simply macros that print output to the terminal.

This solution is IMHO 99.99% effective as it still uses the blocking readline function, so we assume the sub process is nice and outputs complete lines.

I welcome feedback to improve the solution as I am still new to Python.

User · Answer

The select module helps you determine where the next useful input is   However  you re almost always happier with separate threads   One does a blocking read the stdin  another does wherever it is you don t want blocked

User · Answer

This solution uses the select module to  read any available data  from an IO stream  This function blocks initially until data is available  but then reads only the data that is available and doesn t block further   Given the fact that it uses the select module  this only works on Unix   The code is fully PEP8-compliant   import select   def read available input stream  max bytes None               Blocks until any data is available  then all available data is then read and returned      This function returns an empty string when end of stream is reached       Args          input stream  The stream to read from          max bytes  int None   The maximum number of bytes to read  This function may return fewer bytes than this       Returns          str               Prepare local variables     input streams    input stream      empty list          read buffer             Initially block for input using  select      if len select select input streams  empty list  empty list  0    gt  0             Poll read-readiness using  select          def select func                return len select select input streams  empty list  empty list  0  0    gt  0            Create while function based on parameters         if max bytes is not None              def while func                    return  len read buffer   lt  max bytes  and select func           else              while func   select func          while True                Read single byte at a time             read data   input stream read 1              if len read data     0                    End of stream                 break               Append byte to string buffer             read buffer    read data               Check if more data is available             if not while func                    break        Return read buffer     return read buffer

User · Answer

Here is a simple solution based on threads which    works on both Linux and Windows  not relying on select   reads both stdout and stderr asynchronouly  doesn t rely on active polling with arbitrary waiting time  CPU friendly   doesn t use asyncio  which may conflict with other libraries   runs until the child process terminates    printer py  import time import sys  sys stdout write  Hello n   sys stdout flush   time sleep 1  sys stdout write  World  n   sys stdout flush   time sleep 1  sys stderr write  That s an error n   sys stderr flush   time sleep 2  sys stdout write  Actually  I m fine n   sys stdout flush   time sleep 1    reader py  import queue import subprocess import sys import threading   def enqueue stream stream  queue  type       for line in iter stream readline  b             queue put str type    line decode  utf-8        stream close     def enqueue process process  queue       process wait       queue put  x     p   subprocess Popen  python printer py   stdout subprocess PIPE  stderr subprocess PIPE  q   queue Queue   to   threading Thread target enqueue stream  args  p stdout  q  1   te   threading Thread target enqueue stream  args  p stderr  q  2   tp   threading Thread target enqueue process  args  p  q   te start   to start   tp start    while True      line   q get       if line 0      x           break     if line 0      2      stderr         sys stdout write   033 0 31m      ANSI red color     sys stdout write line 1        if line 0      2           sys stdout write   033 0m      reset ANSI code     sys stdout flush    tp join   to join   te join

User · Answer

This is a example to run interactive command in subprocess  and the stdout is interactive by using pseudo terminal  You can refer to  https   stackoverflow com a 43012138 3555925     usr bin env python   - - coding  utf-8 - -  import os import sys import select import termios import tty import pty from subprocess import Popen  command    bash    command    docker run -it --rm centos  bin bash  split      save original tty setting then set it to raw mode old tty   termios tcgetattr sys stdin  tty setraw sys stdin fileno       open pseudo-terminal to interact with subprocess master fd  slave fd   pty openpty      use os setsid   make it run in a new process group  or bash job control will not be enabled p   Popen command            preexec fn os setsid            stdin slave fd            stdout slave fd            stderr slave fd            universal newlines True   while p poll   is None      r  w  e   select select  sys stdin  master fd               if sys stdin in r          d   os read sys stdin fileno    10240          os write master fd  d      elif master fd in r          o   os read master fd  10240          if o              os write sys stdout fileno    o     restore tty settings back termios tcsetattr sys stdin  termios TCSADRAIN  old tty

User · Answer

On Unix-like systems and Python 3 5  there s os set blocking which does exactly what it says   import os import time import subprocess  cmd    python3    -c    import time    print i   time sleep 1   for i in range 5    p   subprocess Popen cmd  stdout subprocess PIPE  os set blocking p stdout fileno    False  start   time time   while True        first iteration always produces empty byte string in non-blocking mode     for i in range 2               line   p stdout readline           print i  line          time sleep 0 5      if time time    gt  start   5          break p terminate     This outputs   1 b   2 b 0 n  1 b   2 b 1 n  1 b   2 b 2 n  1 b   2 b 3 n  1 b   2 b 4 n    With os set blocking commented it s   0 b 0 n  1 b 1 n  0 b 2 n  1 b 3 n  0 b 4 n  1 b

User · Answer

This is a example to run interactive command in subprocess  and the stdout is interactive by using pseudo terminal  You can refer to  https   stackoverflow com a 43012138 3555925     usr bin env python   - - coding  utf-8 - -  import os import sys import select import termios import tty import pty from subprocess import Popen  command    bash    command    docker run -it --rm centos  bin bash  split      save original tty setting then set it to raw mode old tty   termios tcgetattr sys stdin  tty setraw sys stdin fileno       open pseudo-terminal to interact with subprocess master fd  slave fd   pty openpty      use os setsid   make it run in a new process group  or bash job control will not be enabled p   Popen command            preexec fn os setsid            stdin slave fd            stdout slave fd            stderr slave fd            universal newlines True   while p poll   is None      r  w  e   select select  sys stdin  master fd               if sys stdin in r          d   os read sys stdin fileno    10240          os write master fd  d      elif master fd in r          o   os read master fd  10240          if o              os write sys stdout fileno    o     restore tty settings back termios tcsetattr sys stdin  termios TCSADRAIN  old tty

User · Answer

One solution is to make another process to perform your read of the process  or make a thread of the process with a timeout   Here s the threaded version of a timeout function   http   code activestate com recipes 473878   However  do you need to read the stdout as it s coming in  Another solution may be to dump the output to a file and wait for the process to finish using p wait     f   open  myprogram output txt   w   p   subprocess Popen  myprogram exe   stdout f  p wait   f close     str   open  myprogram output txt   r   read

User · Answer

The select module helps you determine where the next useful input is   However  you re almost always happier with separate threads   One does a blocking read the stdin  another does wherever it is you don t want blocked

User · Answer

Adding this answer here since it provides ability to set non-blocking pipes on Windows and Unix   All the ctypes details are thanks to  techtonik s answer   There is a slightly modified version to be used both on Unix and Windows systems    Python3 compatible  only minor change needed   Includes posix version  and defines exception to use for either    This way you can use the same function and exception for Unix and Windows code     pipe non blocking py  module      Example use       p   subprocess Popen              command              stdout subprocess PIPE                     pipe non blocking set p stdout fileno         try          data   os read p stdout fileno    1      except PortableBlockingIOError as ex          if not pipe non blocking is error blocking ex               raise ex         all            pipe non blocking set        pipe non blocking is error blocking        PortableBlockingIOError          import os   if os name     nt       def pipe non blocking set fd             Constant could define globally but avoid polluting the name-space           thanks to  https   stackoverflow com questions 34504970         import msvcrt          from ctypes import windll  byref  wintypes  WinError  POINTER         from ctypes wintypes import HANDLE  DWORD  BOOL          LPDWORD   POINTER DWORD           PIPE NOWAIT   wintypes DWORD 0x00000001           def pipe no wait pipefd               SetNamedPipeHandleState   windll kernel32 SetNamedPipeHandleState             SetNamedPipeHandleState argtypes    HANDLE  LPDWORD  LPDWORD  LPDWORD              SetNamedPipeHandleState restype   BOOL              h   msvcrt get osfhandle pipefd               res   windll kernel32 SetNamedPipeHandleState h  byref PIPE NOWAIT   None  None              if res    0                  print WinError                    return False             return True          return pipe no wait fd       def pipe non blocking is error blocking ex           if not isinstance ex  PortableBlockingIOError               return False         from ctypes import GetLastError         ERROR NO DATA   232          return  GetLastError      ERROR NO DATA       PortableBlockingIOError   OSError else      def pipe non blocking set fd           import fcntl         fl   fcntl fcntl fd  fcntl F GETFL          fcntl fcntl fd  fcntl F SETFL  fl   os O NONBLOCK          return True      def pipe non blocking is error blocking ex           if not isinstance ex  PortableBlockingIOError               return False         return True      PortableBlockingIOError   BlockingIOError   To avoid reading incomplete data  I ended up writing my own readline generator  which returns the byte string for each line    Its a generator so you can for example     def non blocking readlines f  chunk 1024               Iterate over lines  yielding b   when nothings left     or when new data is not yet available       stdout iter   iter non blocking readlines process stdout        line   next stdout iter     will be a line or b                import os      from  pipe non blocking import               pipe non blocking set              pipe non blocking is error blocking              PortableBlockingIOError                     fd   f fileno       pipe non blocking set fd       blocks           while True          try              data   os read fd  chunk              if not data                    case were reading finishes with no trailing newline                 yield b   join blocks                  blocks clear           except PortableBlockingIOError as ex              if not pipe non blocking is error blocking ex                   raise ex              yield b               continue          while True              n   data find b  n               if n    -1                  break              yield b   join blocks    data  n   1              data   data n   1               blocks clear           blocks append data

User · Answer

I have created a library based on J  F  Sebastian s solution  You can use it   https   github com cenkalti what

User · Answer

This solution uses the select module to  read any available data  from an IO stream  This function blocks initially until data is available  but then reads only the data that is available and doesn t block further   Given the fact that it uses the select module  this only works on Unix   The code is fully PEP8-compliant   import select   def read available input stream  max bytes None               Blocks until any data is available  then all available data is then read and returned      This function returns an empty string when end of stream is reached       Args          input stream  The stream to read from          max bytes  int None   The maximum number of bytes to read  This function may return fewer bytes than this       Returns          str               Prepare local variables     input streams    input stream      empty list          read buffer             Initially block for input using  select      if len select select input streams  empty list  empty list  0    gt  0             Poll read-readiness using  select          def select func                return len select select input streams  empty list  empty list  0  0    gt  0            Create while function based on parameters         if max bytes is not None              def while func                    return  len read buffer   lt  max bytes  and select func           else              while func   select func          while True                Read single byte at a time             read data   input stream read 1              if len read data     0                    End of stream                 break               Append byte to string buffer             read buffer    read data               Check if more data is available             if not while func                    break        Return read buffer     return read buffer

User · Answer

I add this problem to read some subprocess Popen stdout  Here is my non blocking read solution   import fcntl  def non block read output       fd   output fileno       fl   fcntl fcntl fd  fcntl F GETFL      fcntl fcntl fd  fcntl F SETFL  fl   os O NONBLOCK      try          return output read       except          return       Use example from subprocess import   sb   Popen  echo test  amp  amp  sleep 1000   shell True  stdout PIPE  sb kill      sb stdout read      lt -- This will block non block read sb stdout   test n

User · Answer

Python 3 4 introduces new provisional API for asynchronous IO -- asyncio module    The approach is similar to twisted-based answer by  Bryan Ward -- define a protocol and its methods are called as soon as data is ready      usr bin env python3 import asyncio import os  class SubprocessProtocol asyncio SubprocessProtocol       def pipe data received self  fd  data           if fd    1    got stdout data  bytes              print data       def connection lost self  exc           loop stop     end loop run forever    if os name     nt       loop   asyncio ProactorEventLoop     for subprocess  pipes on Windows     asyncio set event loop loop  else      loop   asyncio get event loop   try      loop run until complete loop subprocess exec SubprocessProtocol            myprogram exe    arg1    arg2        loop run forever   finally      loop close     See  Subprocess  in the docs   There is a high-level interface asyncio create subprocess exec   that returns Process objects that allows to read a line asynchroniosly using StreamReader readline   coroutine   with async await Python 3 5  syntax       usr bin env python3 5 import asyncio import locale import sys from asyncio subprocess import PIPE from contextlib import closing  async def readline and kill  args         start child process     process   await asyncio create subprocess exec  args  stdout PIPE         read line  sequence of bytes ending with b  n   asynchronously     async for line in process stdout          print  got line    line decode locale getpreferredencoding False            break     process kill       return await process wait     wait for the child process to exit   if sys platform     win32       loop   asyncio ProactorEventLoop       asyncio set event loop loop  else      loop   asyncio get event loop    with closing loop       sys exit loop run until complete readline and kill           myprogram exe    arg1    arg2       readline and kill   performs the following tasks    start subprocess  redirect its stdout to a pipe read a line from subprocess  stdout asynchronously kill subprocess wait for it to exit   Each step could be limited by timeout seconds if necessary

User · Answer

Try the asyncproc module  For example   import os from asyncproc import Process myProc   Process  myprogram app    while True        check to see if process has ended     poll   myProc wait os WNOHANG      if poll    None          break       print any new output     out   myProc read       if out                print out   The module takes care of all the threading as suggested by S Lott

User · Answer

I have created a library based on J  F  Sebastian s solution  You can use it   https   github com cenkalti what

User · Answer

You can do this really easily in Twisted  Depending upon your existing code base  this might not be that easy to use  but if you are building a twisted application  then things like this become almost trivial  You create a ProcessProtocol class  and override the outReceived   method  Twisted  depending upon the reactor used  is usually just a big select   loop with callbacks installed to handle data from different file descriptors  often network sockets   So the outReceived   method is simply installing a callback for handling data coming from STDOUT   A simple example demonstrating this behavior is as follows   from twisted internet import protocol  reactor  class MyProcessProtocol protocol ProcessProtocol        def outReceived self  data           print data  proc   MyProcessProtocol   reactor spawnProcess proc     myprogram       myprogram    arg1    arg2    arg3    reactor run     The Twisted documentation has some good information on this   If you build your entire application around Twisted  it makes asynchronous communication with other processes  local or remote  really elegant like this  On the other hand  if your program isn t built on top of Twisted  this isn t really going to be that helpful  Hopefully this can be helpful to other readers  even if it isn t applicable for your particular application

User · Answer

Things are a lot better in modern Python   Here s a simple child program   hello py       usr bin env python3  while True      i   input       if i     quit           break     print f hello  i      And a program to interact with it   import asyncio   async def main        proc   await asyncio subprocess create subprocess exec             hello py   stdin asyncio subprocess PIPE  stdout asyncio subprocess PIPE           proc stdin write b bob n       print await proc stdout read 1024       proc stdin write b alice n       print await proc stdout read 1024       proc stdin write b quit n       await proc wait     asyncio run main      That prints out   b hello bob n  b hello alice n    Note that the actual pattern  which is also by almost all of the previous answers  both here and in related questions  is to set the child s stdout file descriptor to non-blocking  and then poll it in some sort of select loop  These days  of course  that loop is provided by asyncio

User · Answer

Working from J F  Sebastian s answer  and several other sources  I ve put together a simple subprocess manager  It provides the request non-blocking reading  as well as running several processes in parallel  It doesn t use any OS-specific call  that I m aware  and thus should work anywhere   It s available from pypi  so just pip install shelljob  Refer to the project page for examples and full docs

User · Answer

Python 3 4 introduces new provisional API for asynchronous IO -- asyncio module    The approach is similar to twisted-based answer by  Bryan Ward -- define a protocol and its methods are called as soon as data is ready      usr bin env python3 import asyncio import os  class SubprocessProtocol asyncio SubprocessProtocol       def pipe data received self  fd  data           if fd    1    got stdout data  bytes              print data       def connection lost self  exc           loop stop     end loop run forever    if os name     nt       loop   asyncio ProactorEventLoop     for subprocess  pipes on Windows     asyncio set event loop loop  else      loop   asyncio get event loop   try      loop run until complete loop subprocess exec SubprocessProtocol            myprogram exe    arg1    arg2        loop run forever   finally      loop close     See  Subprocess  in the docs   There is a high-level interface asyncio create subprocess exec   that returns Process objects that allows to read a line asynchroniosly using StreamReader readline   coroutine   with async await Python 3 5  syntax       usr bin env python3 5 import asyncio import locale import sys from asyncio subprocess import PIPE from contextlib import closing  async def readline and kill  args         start child process     process   await asyncio create subprocess exec  args  stdout PIPE         read line  sequence of bytes ending with b  n   asynchronously     async for line in process stdout          print  got line    line decode locale getpreferredencoding False            break     process kill       return await process wait     wait for the child process to exit   if sys platform     win32       loop   asyncio ProactorEventLoop       asyncio set event loop loop  else      loop   asyncio get event loop    with closing loop       sys exit loop run until complete readline and kill           myprogram exe    arg1    arg2       readline and kill   performs the following tasks    start subprocess  redirect its stdout to a pipe read a line from subprocess  stdout asynchronously kill subprocess wait for it to exit   Each step could be limited by timeout seconds if necessary

User · Answer

I have the original questioner s problem  but did not wish to invoke threads  I mixed Jesse s solution with a direct read   from the pipe  and my own buffer-handler for line reads  however  my sub-process - ping - always wrote full lines  lt  a system page size   I avoid busy-waiting by only reading in a gobject-registered io watch  These days I usually run code within a gobject MainLoop to avoid threads  def set up ping ip  w         run the sub-process       watch the resultant pipe     p   subprocess Popen    bin ping   ip   stdout subprocess PIPE        make stdout a non-blocking file     fl   fcntl fcntl p stdout  fcntl F GETFL      fcntl fcntl p stdout  fcntl F SETFL  fl   os O NONBLOCK      stdout gid   gobject io add watch p stdout  gobject IO IN  w      return stdout gid   for shutting down  The watcher is def watch f   other       print  reading  f read       return True  And the main program sets up a ping and then calls gobject mail loop  def main        set up ping  192 168 1 8   watch        discard gid as unused here     gobject MainLoop   run    Any other work is attached to callbacks in gobject

User · Answer

Here is my code  used to catch every output from subprocess ASAP  including partial lines  It pumps at same time and stdout and stderr in almost correct order   Tested and correctly worked on Python 2 7 linux  amp  windows       usr bin python     Runner with stdout stderr catcher   from sys import argv from subprocess import Popen  PIPE import os  io from threading import Thread import Queue def   main          if  len argv   gt  1  and  argv -1      -sub-            import time  sys         print  Application runned           time sleep 2          print  Slept 2 second          time sleep 1          print  Slept 1 additional second           time sleep 2          sys stderr write  Stderr output after 5 seconds           print  Eol on stdin          sys stderr write  Eol on stderr n           time sleep 1          print  Wow  we have end of work        else          os environ  PYTHONUNBUFFERED    1          try              p   Popen  argv     -sub-                           bufsize 0    line-buffered                        stdin PIPE  stdout PIPE  stderr PIPE           except WindowsError  W              if W winerror  193                  p   Popen  argv     -sub-                               shell True    Try to run via shell                            bufsize 0    line-buffered                            stdin PIPE  stdout PIPE  stderr PIPE               else                  raise         inp   Queue Queue           sout   io open p stdout fileno     rb   closefd False          serr   io open p stderr fileno     rb   closefd False          def Pump stream  category               queue   Queue Queue               def rdr                    while True                      buf   stream read1 8192                      if len buf  gt 0                          queue put  buf                       else                          queue put  None                           return             def clct                    active   True                 while active                      r   queue get                       try                          while True                              r1   queue get timeout 0 005                              if r1 is None                                  active   False                                 break                             else                                  r    r1                     except Queue Empty                          pass                     inp put   category  r                for tgt in  rdr  clct                   th   Thread target tgt                  th setDaemon True                  th start           Pump sout   stdout           Pump serr   stderr            while p poll   is None                App still working             try                  chan line   inp get timeout   1 0                  if chan   stdout                       print  STDOUT gt  gt    line    lt   lt                   elif chan   stderr                       print   ERROR     line                    except Queue Empty                  pass         print  Finish   if   name         main           main

User · Answer

Use select  amp  read 1      import subprocess      no new requirements def readAllSoFar proc  retVal         while  select select  proc stdout        0  0               retVal  proc stdout read 1    return retVal p   subprocess Popen    bin ls    stdout subprocess PIPE  while not p poll      print  readAllSoFar p     For readline  -like   lines        while not p poll      lines   readAllSoFar p  lines -1   split   n     for a in range len lines -1       print a lines   readAllSoFar p  lines -1   split   n   for a in range len lines -1     print a

[python] A non-blocking read on a subprocess.PIPE in Python

Examples related to python

Examples related to io

Examples related to subprocess

Examples related to nonblocking