How should I read a file line-by-line in Python

Question

In pre-historic times  Python 1 4  we did   fp   open  filename txt   while 1      line   fp readline       if not line          break     print line   after Python 2 1  we did   for line in open  filename txt   xreadlines        print line   before we got the convenient iterator protocol in Python 2 3  and could do   for line in open  filename txt        print line   I ve seen some examples using the more verbose   with open  filename txt   as fp      for line in fp          print line   is this the preferred method going forwards    edit  I get that the with statement ensures closing of the file    but why isn t that included in the iterator protocol for file objects

User · Answer

f   open  test txt   r   for line in f xreadlines        print line f close

User · Answer

Yes   with open  filename txt   as fp      for line in fp          print line   is the way to go   It is not more verbose  It is more safe

User · Answer

There is exactly one reason why the following is preferred   with open  filename txt   as fp      for line in fp          print line   We are all spoiled by CPython s relatively deterministic reference-counting scheme for garbage collection   Other  hypothetical implementations of Python will not necessarily close the file  quickly enough  without the with block if they use some other scheme to reclaim memory   In such an implementation  you might get a  too many files open  error from the OS if your code opens files faster than the garbage collector calls finalizers on orphaned file handles   The usual workaround is to trigger the GC immediately  but this is a nasty hack and it has to be done by every function that could encounter the error  including those in libraries   What a nightmare   Or you could just use the with block   Bonus Question   Stop reading now if are only interested in the objective aspects of the question       Why isn t that included in the iterator protocol for file objects    This is a subjective question about API design  so I have a subjective answer in two parts   On a gut level  this feels wrong  because it makes iterator protocol do two separate things   iterate over lines and close the file handle   and it s often a bad idea to make a simple-looking function do two actions   In this case  it feels especially bad because iterators relate in a quasi-functional  value-based way to the contents of a file  but managing file handles is a completely separate task   Squashing both  invisibly  into one action  is surprising to humans who read the code and makes it more difficult to reason about program behavior   Other languages have essentially come to the same conclusion   Haskell briefly flirted with so-called  lazy IO  which allows you to iterate over a file and have it automatically closed when you get to the end of the stream  but it s almost universally discouraged to use lazy IO in Haskell these days  and Haskell users have mostly moved to more explicit resource management like Conduit which behaves more like the with block in Python   On a technical level  there are some things you may want to do with a file handle in Python which would not work as well if iteration closed the file handle   For example  suppose I need to iterate over the file twice   with open  filename txt   as fp      for line in fp                  fp seek 0      for line in fp                While this is a less common use case  consider the fact that I might have just added the three lines of code at the bottom to an existing code base which originally had the top three lines   If iteration closed the file  I wouldn t be able to do that   So keeping iteration and resource management separate makes it easier to compose chunks of code into a larger  working Python program   Composability is one of the most important usability features of a language or API

User · Answer

if you re turned off by the extra line  you can use a wrapper function like so   def with iter iterable       with iterable as iter          for item in iter              yield item  for line in with iter open                    in Python 3 3  the yield from statement would make this even shorter   def with iter iterable       with iterable as iter          yield from iter

[python] How should I read a file line-by-line in Python?

Examples related to python

Examples related to python-3.x

Examples related to python-2.7