What is the python with statement designed for

Question

I came across the Python with statement for the first time today   I ve been using Python lightly for several months and didn t even know of its existence   Given its somewhat obscure status  I thought it would be worth asking    What is the Python with statement designed to be used for     What do you use it for   Are there any gotchas I need to be aware of  or common anti-patterns associated with its use   Any cases where it is better use try  finally than with  Why isn t it used more widely  Which standard library classes are compatible with it

User · Answer

An example of an antipattern might be to use the with inside a loop when it would be more efficient to have the with outside the loop

for example

for row in lines:
    with open("outfile","a") as f:
        f.write(row)

vs

with open("outfile","a") as f:
    for row in lines:
        f.write(row)

The first way is opening and closing the file for each row which may cause performance problems compared to the second way with opens and closes the file just once.

User · Answer

The Python with statement is built-in language support of the Resource Acquisition Is Initialization idiom commonly used in C    It is intended to allow safe acquisition and release of operating system resources   The with statement creates resources within a scope block  You write your code using the resources within the block  When the block exits the resources are cleanly released regardless of the outcome of the code in the block  that is whether the block exits normally or because of an exception    Many resources in the Python library that obey the protocol required by the with statement and so can used with it out-of-the-box  However anyone can make resources that can be used in a with statement by implementing the well documented protocol  PEP 0343  Use it whenever you acquire resources in your application that must be explicitly relinquished such as files  network connections  locks and the like

User · Answer

In python generally    with    statement is used to open a file  process the data present in the file  and also to close the file without calling a close   method     with    statement makes the exception handling simpler by providing cleanup activities   General form of with   with open    file name        mode     as file-var      processing statements   note  no need to close the file by calling close   upon file-var close

User · Answer

Another example for out-of-the-box support  and one that might be a bit baffling at first when you are used to the way built-in open   behaves  are connection objects of popular database modules such as    sqlite3 psycopg2 cx oracle   The connection objects are context managers and as such can be used out-of-the-box in a with-statement  however when using the above note that      When the with-block is finished  either with an exception or without  the connection is not closed  In case the with-block finishes with an exception  the transaction is rolled back  otherwise the transaction is commited    This means that the programmer has to take care to close the connection themselves  but allows to acquire a connection  and use it in multiple with-statements  as shown in the psycopg2 docs   conn   psycopg2 connect DSN   with conn      with conn cursor   as curs          curs execute SQL1   with conn      with conn cursor   as curs          curs execute SQL2   conn close     In the example above  you ll note that the cursor objects of psycopg2 also are context managers  From the relevant documentation on the behavior      When a cursor exits the with-block it is closed  releasing any resource eventually associated with it  The state of the transaction is not affected

User · Answer

See PEP 343 - The  with  statement  there is an example section at the end          new statement  with  to the Python   language to make       it possible to factor out standard uses of try finally statements

User · Answer

Again for completeness I ll add my most useful use-case for with statements   I do a lot of scientific computing and for some activities I need the Decimal library for arbitrary precision calculations   Some part of my code I need high precision and for most other parts I need less precision   I set my default precision to a low number and then use with to get a more precise answer for some sections   from decimal import localcontext  with localcontext   as ctx      ctx prec   42     Perform a high precision calculation     s   calculate something   s    s    Round the final result back to the default precision   I use this a lot with the Hypergeometric Test which requires the division of large numbers resulting form factorials   When you do genomic scale calculations you have to be careful of round-off and overflow errors

User · Answer

I believe this has already been answered by other users before me  so I only add it for the sake of completeness  the with statement simplifies exception handling by encapsulating common preparation and cleanup tasks in so-called context managers  More details can be found in PEP 343  For instance  the open statement is a context manager in itself  which lets you open a file  keep it open as long as the execution is in the context of the with statement where you used it  and close it as soon as you leave the context  no matter whether you have left it because of an exception or during regular control flow  The with statement can thus be used in ways similar to the RAII pattern in C    some resource is acquired by the with statement and released when you leave the with context  Some examples are  opening files using with open filename  as fp   acquiring locks using with lock   where lock is an instance of threading Lock   You can also construct your own context managers using the contextmanager decorator from contextlib  For instance  I often use this when I have to change the current directory temporarily and then return to where I was   from contextlib import contextmanager import os   contextmanager def working directory path       current dir   os getcwd       os chdir path      try          yield     finally          os chdir current dir   with working directory  data stuff          do something within data stuff   here I am back again in the original working directory   Here s another example that temporarily redirects sys stdin  sys stdout and sys stderr to some other file handle and restores them later   from contextlib import contextmanager import sys   contextmanager def redirected   kwds       stream names     stdin    stdout    stderr       old streams          try          for sname in stream names              stream   kwds get sname  None              if stream is not None and stream    getattr sys  sname                   old streams sname    getattr sys  sname                  setattr sys  sname  stream          yield     finally          for sname  stream in old streams iteritems                setattr sys  sname  stream   with redirected stdout open   tmp log txt    w            these print statements will go to  tmp log txt      print  Test entry 1       print  Test entry 2    back to the normal stdout print  Back to normal stdout again    And finally  another example that creates a temporary folder and cleans it up when leaving the context   from tempfile import mkdtemp from shutil import rmtree   contextmanager def temporary dir  args    kwds       name   mkdtemp  args    kwds      try          yield name     finally          shutil rmtree name   with temporary dir   as dirname        do whatever you want

User · Answer

I would suggest two interesting lectures      PEP 343 The  with  Statement   Effbot Understanding Python s  with  statement   1  The with statement is used to wrap the execution of a block with methods defined by a context manager  This allows common try   except   finally usage patterns to be encapsulated for convenient reuse   2  You could do something like   with open  foo txt   as foo file      data   foo file read     OR  from contextlib import nested with nested A    B    C    as  X  Y  Z      do something     OR  Python 3 1   with open  data   as input file  open  result    w   as output file     for line in input file       output file write parse line     OR  lock   threading Lock   with lock        Critical section of code   3   I don t see any Antipattern here  Quoting Dive into Python       try  finally is good  with is better    4  I guess it s related to programmers s habit to use try  catch  finally statement from other languages

User · Answer

The with statement works with so-called context managers   http   docs python org release 2 5 2 lib typecontextmanager html  The idea is to simplify exception handling by doing the necessary cleanup after leaving the  with  block  Some of the python built-ins already work as context managers

User · Answer

points 1  2  and 3 being reasonably well covered   4  it is relatively new  only available in python2 6   or python2 5 using from   future   import with statement

[python] What is the python "with" statement designed for?

Examples related to python

Examples related to language-features

Examples related to with-statement