ProcessBuilder Forwarding stdout and stderr of started processes without blocking the main thread

Question

I m building a process in Java using ProcessBuilder as follows   ProcessBuilder pb   new ProcessBuilder            command  somecommand    arg1    arg2            redirectErrorStream true   Process p   pb start     InputStream stdOut   p getInputStream      Now my problem is the following  I would like to capture whatever is going through stdout and or stderr of that process and redirect it to System out asynchronously  I want the process and its output redirection to run in the background   So far  the only way I ve found to do this is to manually spawn a new thread that will continuously read from stdOut and then call the appropriate write   method of System out    new Thread new Runnable        public void run            byte   buffer   new byte 8192           int len   -1          while  len   stdOut read buffer    gt  0               System out write buffer  0  len                      start      While that approach kind of works  it feels a bit dirty  And on top of that  it gives me one more thread to manage and terminate correctly  Is there any better way to do this

User · Answer

A flexible solution with Java 8 lambda that lets you provide a Consumer that will process the output (eg. log it) line by line. run() is a one-liner with no checked exceptions thrown. Alternatively to implementing Runnable, it can extend Thread instead as other answers suggest.

class StreamGobbler implements Runnable {
    private InputStream inputStream;
    private Consumer<String> consumeInputLine;

    public StreamGobbler(InputStream inputStream, Consumer<String> consumeInputLine) {
        this.inputStream = inputStream;
        this.consumeInputLine = consumeInputLine;
    }

    public void run() {
        new BufferedReader(new InputStreamReader(inputStream)).lines().forEach(consumeInputLine);
    }
}

You can then use it for example like this:

public void runProcessWithGobblers() throws IOException, InterruptedException {
    Process p = new ProcessBuilder("...").start();
    Logger logger = LoggerFactory.getLogger(getClass());

    StreamGobbler outputGobbler = new StreamGobbler(p.getInputStream(), System.out::println);
    StreamGobbler errorGobbler = new StreamGobbler(p.getErrorStream(), logger::error);

    new Thread(outputGobbler).start();
    new Thread(errorGobbler).start();
    p.waitFor();
}

Here the output stream is redirected to System.out and the error stream is logged on the error level by the logger.

User · Answer

Simple java8 solution with capturing both outputs and reactive processing using CompletableFuture   static CompletableFuture lt String gt  readOutStream InputStream is        return CompletableFuture supplyAsync    - gt            try                   InputStreamReader isr   new InputStreamReader is                   BufferedReader br   new BufferedReader isr                          StringBuilder res   new StringBuilder                String inputLine              while   inputLine   br readLine       null                    res append inputLine  append System lineSeparator                               return res toString              catch  Throwable e                throw new RuntimeException  problem with executing program   e                         And the usage   Process p   Runtime getRuntime   exec cmd   CompletableFuture lt String gt  soutFut   readOutStream p getInputStream     CompletableFuture lt String gt  serrFut   readOutStream p getErrorStream     CompletableFuture lt String gt  resultFut   soutFut thenCombine serrFut   stdout  stderr  - gt                print to current stderr the stderr of process and return the stdout         System err println stderr           return stdout                 get stdout once ready  blocking String result   resultFut get

User · Answer

I too can use only Java 6  I used  EvgeniyDorofeev s thread scanner implementation  In my code  after a process finishes  I have to immediately execute two other processes that each compare the redirected output  a diff-based unit test to ensure stdout and stderr are the same as the blessed ones    The scanner threads don t finish soon enough  even if I waitFor   the process to complete  To make the code work correctly  I have to make sure the threads are joined after the process finishes   public static int runRedirect  String   args  String stdout redirect to  String stderr redirect to  throws IOException  InterruptedException       ProcessBuilder b   new ProcessBuilder   command args       Process p   b start        Thread ot   null      PrintStream out   null      if  stdout redirect to    null            out   new PrintStream new BufferedOutputStream new FileOutputStream stdout redirect to             ot   inheritIO p getInputStream    out           ot start              Thread et   null      PrintStream err   null      if  stderr redirect to    null            err   new PrintStream new BufferedOutputStream new FileOutputStream stderr redirect to             et   inheritIO p getErrorStream    err           et start              p waitFor          ensure the process finishes before proceeding     if  ot    null          ot join        ensure the thread finishes before proceeding     if  et    null          et join        ensure the thread finishes before proceeding     int rc   p exitValue        return rc     private static Thread inheritIO  final InputStream src  final PrintStream dest        return new Thread new Runnable             public void run                 Scanner sc   new Scanner src               while  sc hasNextLine                    dest println sc nextLine                 dest flush

User · Answer

Use ProcessBuilder inheritIO  it sets the source and destination for subprocess standard I O to be the same as those of the current Java process   Process p   new ProcessBuilder   inheritIO   command  command1   start      If Java 7 is not an option  public static void main String   args  throws Exception       Process p   Runtime getRuntime   exec  cmd  c dir        inheritIO p getInputStream    System out       inheritIO p getErrorStream    System err       private static void inheritIO final InputStream src  final PrintStream dest        new Thread new Runnable             public void run                 Scanner sc   new Scanner src               while  sc hasNextLine                      dest println sc nextLine                                    start        Threads will die automatically when subprocess finishes  because src will EOF

User · Answer

import java io BufferedReader  import java io InputStreamReader   public class Main        public static void main String   args  throws Exception           ProcessBuilder pb   new ProcessBuilder  quot script bat quot            pb redirectErrorStream true           Process p   pb start            BufferedReader logReader   new BufferedReader new InputStreamReader p getInputStream              String logLine   null          while    logLine   logReader readLine       null               System out println  quot Script output   quot    logLine                      By using this line  pb redirectErrorStream true   we can combine InputStream and ErrorStream

User · Answer

There is a library that provides a better ProcessBuilder  zt-exec  This library can do exactly what you are asking for and more   Here s what your code would look like with zt-exec instead of ProcessBuilder     add the dependency     lt dependency gt     lt groupId gt org zeroturnaround lt  groupId gt     lt artifactId gt zt-exec lt  artifactId gt     lt version gt 1 11 lt  version gt   lt  dependency gt    The code     new ProcessExecutor      command  somecommand    arg1    arg2      redirectOutput System out     redirectError System err     execute      Documentation of the library is here   https   github com zeroturnaround zt-exec

User · Answer

As an addition to msangel answer I would like to add the following code block    private static CompletableFuture lt Boolean gt  redirectToLogger final InputStream inputStream  final Consumer lt String gt  logLineConsumer            return CompletableFuture supplyAsync    - gt                try                   InputStreamReader inputStreamReader   new InputStreamReader inputStream                   BufferedReader bufferedReader   new BufferedReader inputStreamReader                                   String line   null                  while  line   bufferedReader readLine       null                        logLineConsumer accept line                                     return true                catch  IOException e                    return false                                    It allows to redirect the input stream  stdout  stderr  of the process to some other consumer  This might be System out  println or anything else consuming strings   Usage       Process process   processBuilder start   CompletableFuture lt Boolean gt  stdOutRes   redirectToLogger process getInputStream    System out  println   CompletableFuture lt Boolean gt  stdErrRes   redirectToLogger process getErrorStream    System out  println   System out println stdOutRes get     System out println stdErrRes get     System out println process waitFor

User · Answer

It s really surprising to me that the redirection methods in ProcessBuilder don t accept an OutputStream  only File  Yet another proof of forced boilerplate code that Java forces you to write  That said  let s look at a list of comprehensive options   If you want the process output to simply be redirected to its parent s output stream  inheritIO will do the job  If you want the process output to go to a file  use redirect  file   If you want the process output to go to a logger  you need to consume the process InputStream in a separate thread  See the answers that use a Runnable or CompletableFuture  You can also adapt the code below to do this  If you want to the process output to go to a PrintWriter  that may or may not be the stdout  very useful for testing   you can do the following   static int execute List lt String gt  args  PrintWriter out        ProcessBuilder builder   new ProcessBuilder                command args               redirectErrorStream true       Process process   null      boolean complete   false      try           process   builder start            redirectOut process getInputStream    out                   orTimeout TIMEOUT  TimeUnit SECONDS           complete   process waitFor TIMEOUT  TimeUnit SECONDS         catch  IOException e            throw new UncheckedIOException e         catch  InterruptedException e            LOG warn  quot Thread was interrupted quot   e         finally           if  process    null  amp  amp   complete                LOG warn  quot Process    didn t finish within    seconds quot   args get 0   TIMEOUT               process   process destroyForcibly                         return process    null   process exitValue     1     private static CompletableFuture lt Void gt  redirectOut InputStream in  PrintWriter out        return CompletableFuture runAsync    - gt            try                   InputStreamReader inputStreamReader   new InputStreamReader in                   BufferedReader bufferedReader   new BufferedReader inputStreamReader                          bufferedReader lines                        forEach out  println             catch  IOException e                LOG error  quot Failed to redirect process output quot   e                        Advantages of the code above over the other answers thus far   redirectErrorStream true  redirects the error stream to the output stream  so that we only have to bother with one  CompletableFuture runAsync runs from the ForkJoinPool  Note that this code doesn t block by calling get or join on the CompletableFuture but sets a timeout instead on its completion  Java 9    There s no need for CompletableFuture supplyAsync because there s nothing really to return from the method redirectOut  BufferedReader lines is simpler than using a while loop

User · Answer

Thread thread   new Thread    - gt          new BufferedReader            new InputStreamReader inputStream                                   StandardCharsets UTF 8                  lines   forEach                   thread start      Your custom code goes instead of the

User · Answer

By default  the created subprocess does not have its own terminal or console  All its standard I O  i e  stdin  stdout  stderr  operations will be redirected to the parent process  where they can be accessed via the streams obtained using the methods getOutputStream    getInputStream    and getErrorStream    The parent process uses these streams to feed input to and get output from the subprocess  Because some native platforms only provide limited buffer size for standard input and output streams  failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block  or even deadlock   https   www securecoding cert org confluence display java FIO07-J  Do not let external processes block on IO buffers

User · Answer

It s as simple as following       File logFile   new File           ProcessBuilder pb   new ProcessBuilder            command  somecommand    arg1    arg2       processBuilder redirectErrorStream true       processBuilder redirectOutput logFile     by  redirectErrorStream true  you tell process to merge error and output stream and then by  redirectOutput file  you redirect merged output to a file    Update   I did manage to do this as follows   public static void main String   args           Async part     Runnable r      - gt            ProcessBuilder pb   new ProcessBuilder   command                    Merge System err and System out         pb redirectErrorStream true              Inherit System out as redirect output stream         pb redirectOutput ProcessBuilder Redirect INHERIT           try               pb start              catch  IOException e                e printStackTrace                         new Thread r   asyncOut   start           here goes your main part     Now you re able to see both outputs from main and asyncOut threads in System out

User · Answer

To only way in Java 6 or earlier is with a so called StreamGobbler  which you are started to create    StreamGobbler errorGobbler   new StreamGobbler p getErrorStream     ERROR        any output  StreamGobbler outputGobbler   new StreamGobbler p getInputStream     OUTPUT        start gobblers outputGobbler start    errorGobbler start           private class StreamGobbler extends Thread       InputStream is      String type       private StreamGobbler InputStream is  String type            this is   is          this type   type              Override     public void run             try               InputStreamReader isr   new InputStreamReader is               BufferedReader br   new BufferedReader isr               String line   null              while   line   br readLine       null                  System out println type     gt      line                     catch  IOException ioe                ioe printStackTrace                        For Java 7  see Evgeniy Dorofeev s answer

[java] ProcessBuilder: Forwarding stdout and stderr of started processes without blocking the main thread

Examples related to java

Examples related to processbuilder