How to start a background process in Python

Question

I m trying to port a shell script to the much more readable python version  The original shell script starts several processes  utilities  monitors  etc   in the background with   amp    How can I achieve the same effect in python  I d like these processes not to die when the python scripts complete  I am sure it s related to the concept of a daemon somehow  but I couldn t find how to do this easily

User · Answer

You probably want to start investigating the os module for forking different threads  by opening an interactive session and issuing help os    The relevant functions are fork and any of the exec ones  To give you an idea on how to start  put something like this in a function that performs the fork  the function needs to take a list or tuple  args  as an argument that contains the program s name and its parameters  you may also want to define stdin  out and err for the new thread    try      pid   os fork   except OSError  e         some debug output     sys exit 1  if pid    0         eventually use os putenv     to set environment variables        os execv strips of args 0  for the arguments     os execv args 0   args

User · Answer

I found this here   On windows  win xp   the parent process will not finish until the longtask py has finished its work  It is not what you want in CGI-script  The problem is not specific to Python  in PHP community the problems are the same   The solution is to pass DETACHED PROCESS Process Creation Flag to the underlying CreateProcess function in win API  If you happen to have installed pywin32 you can import the flag from the win32process module  otherwise you should define it yourself   DETACHED PROCESS   0x00000008  pid   subprocess Popen  sys executable   longtask py                           creationflags DETACHED PROCESS  pid

User · Answer

Note  This answer is less current than it was when posted in 2009  Using the subprocess module shown in other answers is now recommended in the docs      Note that the subprocess module provides more powerful facilities for spawning new processes and retrieving their results  using that module is preferable to using these functions       If you want your process to start in the background you can either use system   and call it in the same way your shell script did  or you can spawn it   import os os spawnl os P DETACH   some long running command      or  alternatively  you may try the less portable os P NOWAIT flag    See the documentation here

User · Answer

While jkp s solution works  the newer way of doing things  and the way the documentation recommends  is to use the subprocess module  For simple commands its equivalent  but it offers more options if you want to do something complicated   Example for your case   import subprocess subprocess Popen   rm   -r   some file      This will run rm -r some file in the background  Note that calling  communicate   on the object returned from Popen will block until it completes  so don t do that if you want it to run in the background   import subprocess ls output subprocess Popen   sleep    30    ls output communicate      Will block for 30 seconds   See  the documentation here   Also  a point of clarification   Background  as you use it here is purely a shell concept  technically  what you mean is that you want to spawn a process without blocking while you wait for it to complete  However  I ve used  background  here to refer to shell-background-like behavior

User · Answer

Both capture output and run on background with threading  As mentioned on this answer  if you capture the output with stdout  and then try to read    then the process blocks   However  there are cases where you need this  For example  I wanted to launch two processes that talk over a port between them  and save their stdout to a log file and stdout   The threading module allows us to do that   First  have a look at how to do the output redirection part alone in this question  Python Popen  Write to stdout AND log file simultaneously  Then   main py     usr bin env python3  import os import subprocess import sys import threading  def output reader proc  file       while True          byte   proc stdout read 1          if byte              sys stdout buffer write byte              sys stdout flush               file buffer write byte          else              break  with subprocess Popen     sleep py    0    stdout subprocess PIPE  stderr subprocess PIPE  as proc1         subprocess Popen     sleep py    10    stdout subprocess PIPE  stderr subprocess PIPE  as proc2         open  log1 log    w   as file1         open  log2 log    w   as file2      t1   threading Thread target output reader  args  proc1  file1       t2   threading Thread target output reader  args  proc2  file2       t1 start       t2 start       t1 join       t2 join     sleep py     usr bin env python3  import sys import time  for i in range 4       print i   int sys argv 1        sys stdout flush       time sleep 0 5    After running     main py   stdout get updated every 0 5 seconds for every two lines to contain   0 10 1 11 2 12 3 13   and each log file contains the respective log for a given process   Inspired by  https   eli thegreenplace net 2017 interacting-with-a-long-running-child-process-in-python   Tested on Ubuntu 18 04  Python 3 6 7

User · Answer

You probably want the answer to  How to call an external command in Python    The simplest approach is to use the os system function  e g    import os os system  some command  amp      Basically  whatever you pass to the system function will be executed the same as if you d passed it to the shell in a script

User · Answer

Use subprocess Popen   with the close fds True parameter  which will allow the spawned subprocess to be detached from the Python process itself and continue running even after Python exits   https   gist github com yinjimmy d6ad0742d03d54518e9f  import os  time  sys  subprocess  if len sys argv     2      time sleep 5      print  track end      if sys platform     darwin           subprocess Popen   say    hello    else      print  main begin      subprocess Popen   python   os path realpath   file      0    close fds True      print  main end

[python] How to start a background process in Python?

Examples related to python

Examples related to process

Examples related to daemon