Running shell command and capturing the output

Question

I want to write a function that will execute a shell command and return its output as a string  no matter  is it an error or success message  I just want to get the same result that I would have gotten with the command line   What would be a code example that would do such a thing   For example   def run command cmd                 print run command  mysqladmin create test -uroot -pmysqladmin12     Should output something like    mysqladmin  CREATE DATABASE failed  error   Can t create database  test   database exists

User · Answer

Splitting the initial command for the subprocess might be tricky and cumbersome   Use shlex split   to help yourself out   Sample command  git log -n 5 --since  5 years ago  --until  2 year ago   The code  from subprocess import check output from shlex import split  res   check output split  git log -n 5 --since  5 years ago  --until  2 year ago     print res   gt  gt  gt  b commit 7696ab087a163e084d6870bb4e5e4d4198bdc61a nAuthor  Artur Barseghyan       Without shlex split   the code would look as follows  res   check output        git         log         -n         5         --since         5 years ago         --until         2 year ago     print res   gt  gt  gt  b commit 7696ab087a163e084d6870bb4e5e4d4198bdc61a nAuthor  Artur Barseghyan

User · Answer

In all officially maintained versions of Python  the simplest approach is to use the subprocess check output function   gt  gt  gt  subprocess check output   ls    -l    b total 0 n-rw-r--r--  1 memyself  staff  0 Mar 14 11 04 files n   check output runs a single program that takes only arguments as input 1 It returns the result exactly as printed to stdout  If you need to write input to stdin  skip ahead to the run or Popen sections  If you want to execute complex shell commands  see the note on shell True at the end of this answer  The check output function works in all officially maintained versions of Python  But for more recent versions  a more flexible approach is available  Modern versions of Python  3 5 or higher   run If you re using Python 3 5   and do not need backwards compatibility  the new run function is recommended by the official documentation for most tasks  It provides a very general  high-level API for the subprocess module  To capture the output of a program  pass the subprocess PIPE flag to the stdout keyword argument  Then access the stdout attribute of the returned CompletedProcess object   gt  gt  gt  import subprocess  gt  gt  gt  result   subprocess run   ls    -l    stdout subprocess PIPE   gt  gt  gt  result stdout b total 0 n-rw-r--r--  1 memyself  staff  0 Mar 14 11 04 files n   The return value is a bytes object  so if you want a proper string  you ll need to decode it  Assuming the called process returns a UTF-8-encoded string   gt  gt  gt  result stdout decode  utf-8    total 0 n-rw-r--r--  1 memyself  staff  0 Mar 14 11 04 files n   This can all be compressed to a one-liner if desired   gt  gt  gt  subprocess run   ls    -l    stdout subprocess PIPE  stdout decode  utf-8    total 0 n-rw-r--r--  1 memyself  staff  0 Mar 14 11 04 files n   If you want to pass input to the process s stdin  you can pass a bytes object to the input keyword argument   gt  gt  gt  cmd     awk    length  0   gt  5    gt  gt  gt  ip    foo nfoofoo n  encode  utf-8    gt  gt  gt  result   subprocess run cmd  stdout subprocess PIPE  input ip   gt  gt  gt  result stdout decode  utf-8    foofoo n   You can capture errors by passing stderr subprocess PIPE  capture to result stderr  or stderr subprocess STDOUT  capture to result stdout along with regular output   If you want run to throw an exception when the process returns a nonzero exit code  you can pass check True   Or you can check the returncode attribute of result above   When security is not a concern  you can also run more complex shell commands by passing shell True as described at the end of this answer  Later versions of Python streamline the above further  In Python 3 7   the above one-liner can be spelled like this   gt  gt  gt  subprocess run   ls    -l    capture output True  text True  stdout  total 0 n-rw-r--r--  1 memyself  staff  0 Mar 14 11 04 files n   Using run this way adds just a bit of complexity  compared to the old way of doing things  But now you can do almost anything you need to do with the run function alone  Older versions of Python  3-3 4   more about check output If you are using an older version of Python  or need modest backwards compatibility  you can use the check output function as briefly described above  It has been available since Python 2 7  subprocess check output  popenargs    kwargs     It takes takes the same arguments as Popen  see below   and returns a string containing the program s output  The beginning of this answer has a more detailed usage example  In Python 3 5   check output is equivalent to executing run with check True and stdout PIPE  and returning just the stdout attribute  You can pass stderr subprocess STDOUT to ensure that error messages are included in the returned output  When security is not a concern  you can also run more complex shell commands by passing shell True as described at the end of this answer  If you need to pipe from stderr or pass input to the process  check output won t be up to the task  See the Popen examples below in that case  Complex applications and legacy versions of Python  2 6 and below   Popen If you need deep backwards compatibility  or if you need more sophisticated functionality than check output or run provide  you ll have to work directly with Popen objects  which encapsulate the low-level API for subprocesses  The Popen constructor accepts either a single command without arguments  or a list containing a command as its first item  followed by any number of arguments  each as a separate item in the list  shlex split can help parse strings into appropriately formatted lists  Popen objects also accept a host of different arguments for process IO management and low-level configuration  To send input and capture output  communicate is almost always the preferred method  As in  output   subprocess Popen   quot mycmd quot    quot myarg quot                               stdout subprocess PIPE  communicate   0   Or  gt  gt  gt  import subprocess  gt  gt  gt  p   subprocess Popen   ls    -a    stdout subprocess PIPE                                          stderr subprocess PIPE   gt  gt  gt  out  err   p communicate    gt  gt  gt  print out      foo  If you set stdin PIPE  communicate also allows you to pass data to the process via stdin   gt  gt  gt  cmd     awk    length  0   gt  5    gt  gt  gt  p   subprocess Popen cmd  stdout subprocess PIPE                                stderr subprocess PIPE                                stdin subprocess PIPE   gt  gt  gt  out  err   p communicate  foo nfoofoo n    gt  gt  gt  print out foofoo  Note Aaron Hall s answer  which indicates that on some systems  you may need to set stdout  stderr  and stdin all to PIPE  or DEVNULL  to get communicate to work at all  In some rare cases  you may need complex  real-time output capturing  Vartec s answer suggests a way forward  but methods other than communicate are prone to deadlocks if not used carefully  As with all the above functions  when security is not a concern  you can run more complex shell commands by passing shell True  Notes 1  Running shell commands  the shell True argument Normally  each call to run  check output  or the Popen constructor executes a single program  That means no fancy bash-style pipes  If you want to run complex shell commands  you can pass shell True  which all three functions support  For example   gt  gt  gt  subprocess check output  cat books     wc   shell True  text True    1299377 17005208 101299376 n   However  doing this raises security concerns  If you re doing anything more than light scripting  you might be better off calling each process separately  and passing the output from each as an input to the next  via run cmd   stdout etc      input other output   Or Popen cmd   stdout etc      communicate other output   The temptation to directly connect pipes is strong  resist it  Otherwise  you ll likely see deadlocks or have to do hacky things like this

User · Answer

I had a slightly different flavor of the same problem with the following requirements    Capture and return STDOUT messages as they accumulate in the STDOUT buffer  i e  in realtime       vartec solved this Pythonically with his use of generators and the  yield  keyword above  Print all STDOUT lines  even if process exits before STDOUT buffer can be fully read  Don t waste CPU cycles polling the process at high-frequency Check the return code of the subprocess Print STDERR  separate from STDOUT  if we get a non-zero error return code    I ve combined and tweaked previous answers to come up with the following   import subprocess from time import sleep  def run command command       p   subprocess Popen command                           stdout subprocess PIPE                           stderr subprocess PIPE                           shell True        Read stdout from subprocess until the buffer is empty       for line in iter p stdout readline  b             if line    Don t print blank lines             yield line       This ensures the process has completed  AND sets the  returncode  attr     while p poll   is None                                                                                                                                                  sleep  1   Don t waste CPU-cycles       Empty STDERR buffer     err   p stderr read       if p returncode    0           The run command   function is responsible for logging STDERR         print  Error      str err     This code would be executed the same as previous answers   for line in run command cmd       print line

User · Answer

Your Mileage May Vary  I attempted  senderle s spin on Vartec s solution in Windows on Python 2 6 5  but I was getting errors  and no other solutions worked  My error was  WindowsError   Error 6  The handle is invalid    I found that I had to assign PIPE to every handle to get it to return the output I expected - the following worked for me   import subprocess  def run command cmd          given shell command  returns communication tuple of stdout and stderr        return subprocess Popen cmd                               stdout subprocess PIPE                               stderr subprocess PIPE                               stdin subprocess PIPE  communicate     and call like this    0  gets the first element of the tuple  stdout    run command  tracert 11 1 0 1   0    After learning more  I believe I need these pipe arguments because I m working on a custom system that uses different handles  so I had to directly control all the std s   To stop console popups  with Windows   do this   def run command cmd          given shell command  returns communication tuple of stdout and stderr          instantiate a startupinfo obj      startupinfo   subprocess STARTUPINFO         set the use show window flag  might make conditional on being in Windows      startupinfo dwFlags    subprocess STARTF USESHOWWINDOW       pass as the startupinfo keyword argument      return subprocess Popen cmd                              stdout subprocess PIPE                               stderr subprocess PIPE                               stdin subprocess PIPE                               startupinfo startupinfo  communicate    run command  tracert 11 1 0 1

User · Answer

I had the same problem but figured out a very simple way of doing this   import subprocess output   subprocess getoutput  ls -l   print output    Hope it helps out  Note  This solution is Python3 specific as subprocess getoutput   doesn t work in Python2

User · Answer

If you need to run a shell command on multiple files  this did the trick for me   import os import subprocess    Define a function for running commands and capturing stdout line by line    Modified from Vartec s solution because it wasn t printing all lines  def runProcess exe           p   subprocess Popen exe  stdout subprocess PIPE  stderr subprocess STDOUT      return iter p stdout readline  b       Get all filenames in working directory for filename in os listdir              This command will be run on each file     cmd    nm     filename        Run the command and capture the output line by line      for line in runProcess cmd split               Eliminate leading and trailing whitespace         line strip             Split the output          output   line split              Filter the output and print relevant lines         if len output   gt  2              if   output 2      set program name                     print filename                 print line   Edit  Just saw Max Persson s solution with J F  Sebastian s suggestion   Went ahead and incorporated that

User · Answer

I would like to suggest simppl as an option for consideration  It is a module that is available via pypi  pip install simppl and was runs on python3  simppl allows the user to run shell commands and read the output from the screen  The developers suggest three types of use cases    The simplest usage will look like this       from simppl simple pipeline import SimplePipeline     sp   SimplePipeline start 0  end 100       sp print and run   lt YOUR FIRST OS COMMAND gt        sp print and run   lt YOUR SECOND OS COMMAND gt             To run multiple commands concurrently use       commands      lt YOUR FIRST OS COMMAND gt      lt YOUR SECOND OS COMMAND gt        max number of processes   4     sp run parallel commands  max number of processes           Finally  if your project uses the cli module  you can run directly another command line tool as part of a pipeline  The other tool will be run from the same process  but it will appear from the logs as another command in the pipeline  This enables smoother debugging and refactoring of tools calling other tools       from example module import example tool     sp print and run clt example tool run    first number    second nmber                                        -key1    val1    -key2    val2                                       --flag          Note that the printing to STDOUT STDERR is via python s logging module   Here is a complete code to show how simppl works  import logging from logging config import dictConfig  logging config   dict      version   1      formatters              f     format                    asctime s   name -12s   levelname -8s   message s                  handlers              h     class    logging StreamHandler                  formatter    f                  level   logging DEBUG                 root              handlers     h             level   logging DEBUG               dictConfig logging config   from simppl simple pipeline import SimplePipeline sp   SimplePipeline 0  100  sp print and run  ls

User · Answer

You can use following commands to run any shell command  I have used them on ubuntu   import os os popen  your command here   read     Note  This is deprecated since python 2 6  Now you must use subprocess Popen  Below is the example  import subprocess  p   subprocess Popen  Your command   shell True  stdout subprocess PIPE  stderr subprocess PIPE  communicate   0  print p split   n

User · Answer

According to  senderle  if you use python3 6 like me   def sh cmd  input          rst   subprocess run cmd  shell True  stdout subprocess PIPE  stderr subprocess PIPE  input input encode  utf-8        assert rst returncode    0  rst stderr decode  utf-8       return rst stdout decode  utf-8     sh  ls -a     Will act exactly like you run the command in bash

User · Answer

Vartec s answer doesn t read all lines  so I made a version that did   def run command command       p   subprocess Popen command                           stdout subprocess PIPE                           stderr subprocess STDOUT      return iter p stdout readline  b      Usage is the same as the accepted answer   command    mysqladmin create test -uroot -pmysqladmin12  split   for line in run command command       print line

User · Answer

Here a solution  working if you want to print output while process is running or not   I added the current working directory also  it was useful to me more than once   Hoping the solution will help someone     import subprocess  def run command cmd and args  print constantly False  cwd None    quot  quot  quot Runs a system command    param cmd and args  the command to run with or without a Pipe       param print constantly  If True then the output is logged in continuous until the command ended   param cwd  the current working directory  the directory from which you will like to execute the command   return  - a tuple containing the return code  the stdout and the stderr of the command  quot  quot  quot  output       process   subprocess Popen cmd and args  shell True  stdout subprocess PIPE  stderr subprocess PIPE  cwd cwd   while True      next line   process stdout readline       if next line          output append str next line           if print constantly              print next line      elif not process poll            break  error   process communicate   1   return process returncode    n  join output   error

User · Answer

eg  execute  ls -ahl   differentiated three four possible returns and OS platforms    no output  but run successfully output empty line  run successfully run failed output something  run successfully   function below  def execute cmd  output True  DEBUG MODE False      Executes a bash command   cmd  output True  output  whether print shell output to screen  only affects screen display  does not affect returned values return     regardless of output True False            returns shell output as a list with each elment is a line of string  whitespace stripped both sides  from output         could be              ie  len   0 -- gt  no output                   -- gt  output empty line               None -- gt  error occured  see below          if error ocurs  returns None  ie  is None   print out the error message to screen     if not DEBUG MODE      print  Command      cmd        https   stackoverflow com a 40139101 2292993     def  execute cmd cmd           if os name     nt  or platform system       Windows                 set stdin  out  err all to PIPE to get results  other than None  after run the Popen   instance             p   subprocess Popen cmd  stdout subprocess PIPE  stderr subprocess PIPE  shell True          else                Use bash  the default is sh             p   subprocess Popen cmd  stdout subprocess PIPE  stderr subprocess PIPE  shell True  executable   bin bash              the Popen   instance starts running once instantiated                additionally  communicate    or poll   and wait process to terminate           communicate   accepts optional input as stdin to the pipe  requires setting stdin subprocess PIPE above   return out  err as tuple           if communicate    the results are buffered in memory            Read stdout from subprocess until the buffer is empty             if error occurs  the stdout is     which means the below loop is essentially skipped           A prefix of  b  or  B  is ignored in Python 2             it indicates that the literal should become a bytes literal in Python 3             e g  when code is automatically converted with 2to3             return iter p stdout readline  b            for line in iter p stdout readline  b                     Windows has  r n  Unix has  n  Old mac has  r               if line not in       n    r    r n      Don t print blank lines                 yield line         while p poll   is None                                                                                                                                                      sleep  1   Don t waste CPU-cycles           Empty STDERR buffer         err   p stderr read           if p returncode    0                responsible for logging STDERR              print  Error      str err               yield None      out          for line in  execute cmd cmd             error did not occur earlier         if line is not None                trailing comma to avoid a newline  by print itself  being printed             if output  print line              out append line strip            else                error occured earlier             out   None     return out else      print  Simulation  The command is     cmd     print

User · Answer

This is a tricky but super simple solution which works in many situations   import os os system  sample cmd  gt  tmp   print open  tmp    r   read     A temporary file here is tmp  is created with the output of the command and you can read from it your desired output   Extra note from the comments  You can remove the tmp file in the case of one-time job  If you need to do this several times  there is no need to delete the tmp   os remove  tmp

User · Answer

just wrote a small bash script to do this using curl  https   gist github com harish2704 bfb8abece94893c53ce344548ead8ba5     usr bin env bash    Usage  gdrive dl sh  lt url gt   urlBase  https   drive google com  fCookie tmpcookies  curl  curl -L -b  fCookie -c  fCookie  confirm         curl   1    grep jfk-button-action   sed -e  s   jfk-button-action  href     S        1   -e  s   amp amp    amp  g      curl -O -J    urlBase   confirm  1

User · Answer

The output can be redirected to a text file and then read it back   import subprocess import os import tempfile  def execute to file command               This function execute the command     and pass its output to a tempfile then read it back     It is usefull for process that deploy child process             temp file   tempfile NamedTemporaryFile delete False      temp file close       path   temp file name     command   command      gt      path     proc   subprocess run command  shell True  stdout subprocess PIPE  stderr subprocess PIPE  universal newlines True      if proc stderr            if command failed return         os unlink path          return     with open path   r   as f          data   f read       os unlink path      return data  if   name         main         path    Somepath      command    ecls exe  files     path     print execute command

User · Answer

This is way easier  but only works on Unix  including Cygwin  and Python2 7   import commands print commands getstatusoutput  wc -l file     It returns a tuple with the  return value  output    For a solution that works in both Python2 and Python3  use the subprocess module instead   from subprocess import Popen  PIPE output   Popen   date   stdout PIPE  response   output communicate   print response

User · Answer

Something like that    def runProcess exe           p   subprocess Popen exe  stdout subprocess PIPE  stderr subprocess STDOUT      while True             returns None while subprocess is running         retcode   p poll            line   p stdout readline           yield line         if retcode is not None              break   Note  that I m redirecting stderr to stdout  it might not be exactly what you want  but I want error messages also   This function yields line by line as they come  normally you d have to wait for subprocess to finish to get the output as a whole    For your case the usage would be   for line in runProcess  mysqladmin create test -uroot -pmysqladmin12  split         print line

User · Answer

If you use the subprocess python module  you are able to handle the STDOUT  STDERR and return code of command separately  You can see an example for the complete command caller implementation  Of course you can extend it with try  except if you want   The below function returns the STDOUT  STDERR and Return code so you can handle them in the other script   import subprocess  def command caller command None      sp   subprocess Popen command  stderr subprocess PIPE  stdout subprocess PIPE  shell False      out  err   sp communicate       if sp returncode          print               Return code    ret code s Error message    err msg s                  ret code   sp returncode   err msg   err                    return sp returncode  out  err

[python] Running shell command and capturing the output

Examples related to python

Examples related to shell

Examples related to subprocess