IOError Errno 32 Broken pipe Python

Question

I have a very simple Python 3 script   f1   open  a txt    r   print f1 readlines    f2   open  b txt    r   print f2 readlines    f3   open  c txt    r   print f3 readlines    f4   open  d txt    r   print f4 readlines    f1 close   f2 close   f3 close   f4 close     But it always says   IOError   Errno 32  Broken pipe   I saw on the internet all the complicated ways to fix this  but I copied this code directly  so I think that there is something wrong with the code and not Python s SIGPIPE   I am redirecting the output  so if the above script was named  open py   then my command to run would be   open py   othercommand

User · Answer

A  Broken Pipe  error occurs when you try to write to a pipe that has been closed on the other end  Since the code you ve shown doesn t involve any pipes directly  I suspect you re doing something outside of Python to redirect the standard output of the Python interpreter to somewhere else  This could happen if you re running a script like this   python foo py   someothercommand   The issue you have is that someothercommand is exiting without reading everything available on its standard input  This causes your write  via print  to fail at some point   I was able to reproduce the error with the following command on a Linux system   python -c  for i in range 1000   print i    less   If I close the less pager without scrolling through all of its input  1000 lines   Python exits with the same IOError you have reported

User · Answer

This can also occur if the read end of the output from your script dies prematurely  ie   open py   otherCommand  if otherCommand exits   and open py tries to write to stdout    I had a bad gawk script that did this lovely to me

User · Answer

The problem is due to SIGPIPE handling  You can solve this problem using the following code   from signal import signal  SIGPIPE  SIG DFL signal SIGPIPE SIG DFL     See here for background on this solution  Better answer here

User · Answer

I know this is not the  proper  way to do it  but if you are simply interested in getting rid of the error message  you could try this workaround   python your python code py 2 gt   dev null   other command

User · Answer

I haven t reproduced the issue  but perhaps this method would solve it   writing line by line to stdout rather than using print   import sys with open  a txt    r   as f1      for line in f1          sys stdout write line      You could catch the broken pipe  This writes the file to stdout line by line until the pipe is closed    import sys  errno try      with open  a txt    r   as f1          for line in f1              sys stdout write line  except IOError as e      if e errno    errno EPIPE            Handle error   You also need to make sure that othercommand is reading from the pipe before it gets too big - https   unix stackexchange com questions 11946 how-big-is-the-pipe-buffer

User · Answer

I feel obliged to point out that the method using  signal SIGPIPE  SIG DFL     is indeed dangerous  as already suggested by David Bennet in the comments  and in my case led to platform-dependent funny business when combined with multiprocessing Manager  because the standard library relies on BrokenPipeError being raised in several places   To make a long and painful story short  this is how I fixed it   First  you need to catch the IOError  Python 2  or BrokenPipeError  Python 3   Depending on your program you can try to exit early at that point or just ignore the exception   from errno import EPIPE  try      broken pipe exception   BrokenPipeError except NameError     Python 2     broken pipe exception   IOError  try      YOUR CODE GOES HERE except broken pipe exception as exc      if broken pipe exception    IOError          if exc errno    EPIPE              raise   However  this isn t enough  Python 3 may still print a message like this   Exception ignored in   lt  io TextIOWrapper name   lt stdout gt   mode  w  encoding  UTF-8  gt  BrokenPipeError   Errno 32  Broken pipe   Unfortunately getting rid of that message is not straightforward  but I finally found http   bugs python org issue11380 where Robert Collins suggests this workaround that I turned into a decorator you can wrap your main function with  yes  that s some crazy indentation    from functools import wraps from sys import exit  stderr  stdout from traceback import print exc   def suppress broken pipe msg f        wraps f      def wrapper  args    kwargs           try              return f  args    kwargs          except SystemExit              raise         except              print exc               exit 1          finally              try                  stdout flush               finally                  try                      stdout close                   finally                      try                          stderr flush                       finally                          stderr close       return wrapper    suppress broken pipe msg def main        YOUR CODE GOES HERE

User · Answer

The top answer  if e errno    errno EPIPE   here didn t really work for me   I got   AttributeError   BrokenPipeError  object has no attribute  EPIPE    However  this ought to work if all you care about is ignoring broken pipes on specific writes   I think it s safer than trapping SIGPIPE   try        writing  flushing  whatever goes here except BrokenPipeError      exit  0     You obviously have to make a decision as to whether your code is really  truly done if you hit the broken pipe  but for most purposes I think that s usually going to be true    Don t forget to close file handles  etc

User · Answer

To bring Alex L  s helpful answer   akhan s helpful answer  and Blckknght s helpful answer together with some additional information    Standard Unix signal SIGPIPE is sent to a process writing to a pipe when there s no process reading from the pipe  anymore     This is not necessarily an error condition  some Unix utilities such as head by design stop reading prematurely from a pipe  once they ve received enough data   By default - i e   if the writing process does not explicitly trap SIGPIPE - the writing process is simply terminated  and its exit code is set to 141  which is calculated as 128  to signal termination by signal in general    13  SIGPIPE s specific signal number   By design  however  Python itself traps SIGPIPE and translates it into a Python IOError instance with errno value errno EPIPE  so that a Python script can catch it  if it so chooses - see Alex L  s answer for how to do that  If a Python script does not catch it  Python outputs error message IOError   Errno 32  Broken pipe and terminates the script with exit code 1 - this is the symptom the OP saw  In many cases this is more disruptive than helpful  so reverting to the default behavior is desirable    Using the signal module allows just that  as stated in akhan s answer  signal signal   takes a signal to handle as the 1st argument and a handler as the 2nd  special handler value SIG DFL represents the system s default behavior   from signal import signal  SIGPIPE  SIG DFL signal SIGPIPE  SIG DFL

User · Answer

Closes should be done in reverse order of the opens

[python] IOError: [Errno 32] Broken pipe: Python

Examples related to python

Examples related to python-3.x

Examples related to sigpipe