How can I run an external command asynchronously from Python

Question

I need to run a shell command asynchronously from a Python script  By this I mean that I want my Python script to continue running while the external command goes off and does whatever it needs to do   I read this post       Calling an external command in Python    I then went off and did some testing  and it looks like os system   will do the job provided that I use  amp  at the end of the command so that I don t have to wait for it to return  What I am wondering is if this is the proper way to accomplish such a thing  I tried commands call   but it will not work for me because it blocks on the external command   Please let me know if using os system   for this is advisable or if I should try some other route

User · Accepted Answer

subprocess Popen does exactly what you want   from subprocess import Popen p   Popen   watch    ls      something long running       do other stuff while subprocess is running p terminate      Edit to complete the answer from comments   The Popen instance can do various other things like you can poll   it to see if it is still running  and you can communicate   with it to send it data on stdin  and wait for it to terminate

User · Answer

If you want to run many processes in parallel and then handle them when they yield results  you can use polling like in the following   from subprocess import Popen  PIPE import time  running procs         Popen    usr bin my cmd    -i  s    path   stdout PIPE  stderr PIPE      for path in   tmp file0  tmp file1  tmp file2  split     while running procs      for proc in running procs          retcode   proc poll           if retcode is not None    Process finished              running procs remove proc              break         else    No process is done  wait a bit and check again              time sleep  1              continue        Here   proc  has finished with return code  retcode      if retcode    0             Error handling         handle results proc stdout    The control flow there is a little bit convoluted because I m trying to make it small -- you can refactor to your taste   -   This has the advantage of servicing the early-finishing requests first  If you call communicate on the first running process and that turns out to run the longest  the other running processes will have been sitting there idle when you could have been handling their results

User · Answer

This is covered by Python 3 Subprocess Examples under  Wait for command to terminate asynchronously       import asyncio  proc   await asyncio create subprocess exec       ls   -lha       stdout asyncio subprocess PIPE      stderr asyncio subprocess PIPE     do something else while ls is working    if proc takes very long to complete  the CPUs are free to use cycles for    other processes stdout  stderr   await proc communicate      The process will start running as soon as the await asyncio create subprocess exec      has completed   If it hasn t finished by the time you call await proc communicate    it will wait there in order to give you your output status   If it has finished  proc communicate   will return immediately   The gist here is similar to Terrels answer but I think Terrels answer appears to overcomplicate things   See asyncio create subprocess exec for more information

User · Answer

I have the same problem trying to connect to an 3270 terminal using the s3270 scripting software in Python  Now I m solving the problem with an subclass of Process that I found here   http   code activestate com recipes 440554   And here is the sample taken from file   def recv some p  t  1  e 1  tr 5  stderr 0       if tr  lt  1          tr   1     x   time time   t     y          r          pr   p recv     if stderr          pr   p recv err     while time time    lt  x or r          r   pr           if r is None              if e                  raise Exception message              else                  break         elif r              y append r          else              time sleep max  x-time time    tr  0       return    join y   def send all p  data       while len data           sent   p send data          if sent is None              raise Exception message          data   buffer data  sent   if   name         main         if sys platform     win32           shell  commands  tail     cmd     dir  w    echo HELLO WORLD      r n       else          shell  commands  tail     sh     ls    echo HELLO WORLD      n        a   Popen shell  stdin PIPE  stdout PIPE      print recv some a       for cmd in commands          send all a  cmd   tail          print recv some a       send all a   exit    tail      print recv some a  e 0      a wait

User · Answer

I ve had good success with the asyncproc module  which deals nicely with the output from the processes  For example   import os from asynproc import Process myProc   Process  myprogram app    while True        check to see if process has ended     poll   myProc wait os WNOHANG      if poll is not None          break       print any new output     out   myProc read       if out                print out

User · Answer

There are several answers here but none of them satisfied my below requirements    I don t want to wait for command to finish or pollute my terminal with subprocess outputs  I want to run bash script with redirects  I want to support piping within my bash script  for example find       tar         The only combination that satiesfies above requirements is   subprocess Popen     my script sh  arg1   gt   redirect path to                      stdout subprocess PIPE                    stderr subprocess PIPE                   shell True

User · Answer

The accepted answer is very old   I found a better modern answer here    https   kevinmccarthy org 2016 07 25 streaming-subprocess-stdin-and-stdout-with-asyncio-in-python   and made some changes    make it work on windows make it work with multiple commands   import sys import asyncio  if sys platform     win32       asyncio set event loop policy asyncio WindowsProactorEventLoopPolicy      async def  read stream stream  cb       while True          line   await stream readline           if line              cb line          else              break   async def  stream subprocess cmd  stdout cb  stderr cb       try          process   await asyncio create subprocess exec               cmd  stdout asyncio subprocess PIPE  stderr asyncio subprocess PIPE                    await asyncio wait                                 read stream process stdout  stdout cb                    read stream process stderr  stderr cb                                   rc   await process wait           return process pid  rc     except OSError as e            the program will hang if we let any exception propagate         return e   def execute  aws           run the given coroutines in an asyncio loop     returns a list containing the values returned from each coroutine              loop   asyncio get event loop       rc   loop run until complete asyncio gather  aws       loop close       return rc   def printer label       def pr  args    kw           print label   args    kw       return pr   def name it start 0  template  s             a simple generator for task names             while True          yield template format start          start    1   def runners cmds               cmds is a list of commands to excecute as subprocesses     each item is a list appropriate for use by subprocess call             next name   name it     next       for cmd in cmds          name   next name           out   printer f  name  stdout           err   printer f  name  stderr           yield  stream subprocess cmd  out  err    if   name         main         cmds                            sh                -c                  echo   SHELL -stdout  amp  amp  sleep 1  amp  amp  echo stderr 1 gt  amp 2  amp  amp  sleep 1  amp  amp  echo done                                       bash                -c                echo  hello  Dave    amp  amp  sleep 1  amp  amp  echo dave err 1 gt  amp 2  amp  amp  sleep 1  amp  amp  echo done                       sys executable   -c    print  hello from python   import sys sys exit 2                print execute  runners cmds      It is unlikely that the example commands will work perfectly on your system  and it doesn t handle weird errors  but this code does demonstrate one way to run multiple subprocesses using asyncio and stream the output

User · Answer

What I am wondering is if this  os system    is the proper way to accomplish such a thing   No   os system   is not the proper way   That s why everyone says to use subprocess     For more information  read http   docs python org library os html os system     The subprocess module provides more   powerful facilities for spawning new   processes and retrieving their   results  using that module is   preferable to using this function  Use   the subprocess module  Check   especially the Replacing Older   Functions with the subprocess Module   section

User · Answer

Considering  I don t have to wait for it to return   one of the easiest solutions will be this   subprocess Popen         path to executable  arg1  arg2      argN       creationflags   subprocess CREATE NEW CONSOLE    pid   But    From what I read this is not  the proper way to accomplish such a thing  because of security risks created by subprocess CREATE NEW CONSOLE flag   The key things that happen here is use of subprocess CREATE NEW CONSOLE to create new console and  pid  returns process ID so that you could check program later on if you want to  so that not to wait for program to finish its job

User · Answer

Using pexpect with non-blocking readlines is another way to do this   Pexpect solves the deadlock problems  allows you to easily run the processes in the background  and gives easy ways to have callbacks when your process spits out predefined strings  and generally makes interacting with the process much easier

[python] How can I run an external command asynchronously from Python?

Examples related to python

Examples related to asynchronous

Examples related to subprocess

Examples related to scheduler