Locking a file in Python

Question

I need to lock a file for writing in Python  It will be accessed from multiple Python processes at once  I have found some solutions online  but most fail for my purposes as they are often only Unix based or Windows based

User · Answer

Locking a file is usually a platform-specific operation  so you may need to allow for the possibility of running on different operating systems  For example   import os  def my lock f       if os name     posix             Unix or OS X specific locking here     elif os name     nt             Windows specific locking here     else          print  Unknown operating system  lock unavailable

User · Answer

The scenario is like that  The user requests a file to do something  Then  if the user sends the same request again  it informs the user that the second request is not done until the first request finishes  That s why  I use lock-mechanism to handle this issue   Here is my working code    from lockfile import LockFile lock   LockFile lock file path  status      if not lock is locked        lock acquire       status   lock path     is locked       print status else      status   lock path     is already locked       print status  return status

User · Answer

The other solutions cite a lot of external code bases  If you would prefer to do it yourself  here is some code for a cross-platform solution that uses the respective file locking tools on Linux   DOS systems  try        Posix based file locking  Linux  Ubuntu  MacOS  etc           Only allows locking on writable files  might cause         strange results for reading      import fcntl  os     def lock file f           if f writable    fcntl lockf f  fcntl LOCK EX      def unlock file f           if f writable    fcntl lockf f  fcntl LOCK UN  except ModuleNotFoundError        Windows file locking     import msvcrt  os     def file size f           return os path getsize  os path realpath f name        def lock file f           msvcrt locking f fileno    msvcrt LK RLCK  file size f       def unlock file f           msvcrt locking f fileno    msvcrt LK UNLCK  file size f       Class for ensuring that all file operations are atomic  treat   initialization like a standard call to  open  that happens to be atomic    This file opener  must  be used in a  quot with quot  block  class AtomicOpen        Open the file with arguments provided by user  Then acquire       a lock on that file object  WARNING  Advisory locking       def   init   self  path   args    kwargs             Open the file and acquire a lock on the file before operating         self file   open path  args    kwargs            Lock the opened file         lock file self file         Return the opened file object  knowing a lock has been obtained       def   enter   self   args    kwargs   return self file        Unlock the file and close the file object      def   exit   self  exc type None  exc value None  traceback None                     Flush to make sure all buffered contents are written to file          self file flush           os fsync self file fileno              Release the lock on the file          unlock file self file          self file close             Handle exceptions that may have come up during execution  by           default any exceptions are raised to the user          if  exc type    None   return False         else                   return True          Now  AtomicOpen can be used in a with block where one would normally use an open statement  WARNINGS   If running on Windows and Python crashes before exit is called  I m not sure what the lock behavior would be  The locking provided here is advisory  not absolute  All potentially competing processes must use the  quot AtomicOpen quot  class  As of  Nov 9th  2020  this code only locks writable files on Posix systems  At some point after the posting and before this date  it became illegal to use the fcntl lock on read-only files

User · Answer

I prefer lockfile     Platform-independent file locking

User · Answer

Locking is platform and device specific  but generally  you have a few options    Use flock    or equivalent  if your os supports it    This is advisory locking  unless you check for the lock  its ignored  Use a lock-copy-move-unlock methodology  where you copy the file  write the new data  then move it  move  not copy - move is an atomic operation in Linux -- check your OS   and you check for the existence of the lock file  Use a directory as a  lock    This is necessary if you re writing to NFS  since NFS doesn t support flock    There s also the possibility of using shared memory between the processes  but I ve never tried that  it s very OS-specific    For all these methods  you ll have to use a spin-lock  retry-after-failure  technique for acquiring and testing the lock   This does leave a small window for mis-synchronization  but its generally small enough to not be an major issue   If you re looking for a solution that is cross platform  then you re better off logging to another system via some other mechanism  the next best thing is the NFS technique above      Note that sqlite is subject to the same constraints over NFS that normal files are  so you can t write to an sqlite database on a network share and get synchronization for free

User · Answer

Locking is platform and device specific  but generally  you have a few options    Use flock    or equivalent  if your os supports it    This is advisory locking  unless you check for the lock  its ignored  Use a lock-copy-move-unlock methodology  where you copy the file  write the new data  then move it  move  not copy - move is an atomic operation in Linux -- check your OS   and you check for the existence of the lock file  Use a directory as a  lock    This is necessary if you re writing to NFS  since NFS doesn t support flock    There s also the possibility of using shared memory between the processes  but I ve never tried that  it s very OS-specific    For all these methods  you ll have to use a spin-lock  retry-after-failure  technique for acquiring and testing the lock   This does leave a small window for mis-synchronization  but its generally small enough to not be an major issue   If you re looking for a solution that is cross platform  then you re better off logging to another system via some other mechanism  the next best thing is the NFS technique above      Note that sqlite is subject to the same constraints over NFS that normal files are  so you can t write to an sqlite database on a network share and get synchronization for free

User · Answer

Coordinating access to a single file at the OS level is fraught with all kinds of issues that you probably don t want to solve   Your best bet is have a separate process that coordinates read write access to that file

User · Answer

There is a cross-platform file locking module here  Portalocker  Although as Kevin says  writing to a file from multiple processes at once is something you want to avoid if at all possible   If you can shoehorn your problem into a database  you could use SQLite   It supports concurrent access and handles its own locking

User · Answer

Alright  so I ended up going with the code I wrote here  on my website link is dead  view on archive org  also available on GitHub   I can use it in the following fashion   from filelock import FileLock  with FileLock  myfile txt lock        print  Lock acquired        with open  myfile txt              work with the file as it is now locked

User · Answer

Alright  so I ended up going with the code I wrote here  on my website link is dead  view on archive org  also available on GitHub   I can use it in the following fashion   from filelock import FileLock  with FileLock  myfile txt lock        print  Lock acquired        with open  myfile txt              work with the file as it is now locked

User · Answer

Alright  so I ended up going with the code I wrote here  on my website link is dead  view on archive org  also available on GitHub   I can use it in the following fashion   from filelock import FileLock  with FileLock  myfile txt lock        print  Lock acquired        with open  myfile txt              work with the file as it is now locked

User · Answer

Coordinating access to a single file at the OS level is fraught with all kinds of issues that you probably don t want to solve   Your best bet is have a separate process that coordinates read write access to that file

User · Answer

Coordinating access to a single file at the OS level is fraught with all kinds of issues that you probably don t want to solve   Your best bet is have a separate process that coordinates read write access to that file

User · Answer

You may find pylocker very useful  It can be used to lock a file or for locking mechanisms in general and can be accessed from multiple Python processes at once    If you simply  want to lock a file here s how it works    import uuid from pylocker import Locker     create a unique lock pass  This can be any string  lpass   str uuid uuid1       create locker instance  FL   Locker filePath  myfile txt   lockPass lpass  mode  w      aquire the lock with FL as r        get the result     acquired  code  fd    r        check if aquired      if fd is not None          print fd         fd write  I have succesfuly aquired the lock        no need to release anything or to close the file descriptor     with statement takes care of that  let s print fd and verify that  print fd

User · Answer

Locking a file is usually a platform-specific operation  so you may need to allow for the possibility of running on different operating systems  For example   import os  def my lock f       if os name     posix             Unix or OS X specific locking here     elif os name     nt             Windows specific locking here     else          print  Unknown operating system  lock unavailable

User · Answer

I prefer lockfile     Platform-independent file locking

User · Answer

Locking a file is usually a platform-specific operation  so you may need to allow for the possibility of running on different operating systems  For example   import os  def my lock f       if os name     posix             Unix or OS X specific locking here     elif os name     nt             Windows specific locking here     else          print  Unknown operating system  lock unavailable

User · Answer

There is a cross-platform file locking module here  Portalocker  Although as Kevin says  writing to a file from multiple processes at once is something you want to avoid if at all possible   If you can shoehorn your problem into a database  you could use SQLite   It supports concurrent access and handles its own locking

User · Answer

There is a cross-platform file locking module here  Portalocker  Although as Kevin says  writing to a file from multiple processes at once is something you want to avoid if at all possible   If you can shoehorn your problem into a database  you could use SQLite   It supports concurrent access and handles its own locking

User · Answer

I have been working on a situation like this where I run multiple copies of the same program from within the same directory folder and logging errors   My approach was to write a  lock file  to the disc before opening the log file  The program checks for the presence of the  lock file  before proceeding  and waits for its turn if the  lock file  exists   Here is the code   def errlogger error        while True          if not exists  errloglock                lock   open  errloglock    w               if exists  errorlog    log   open  errorlog    a               else  log   open  errorlog    w               log write str datetime utcnow    0 -7          error     n               log close               remove  errloglock               return         else              check   stat  errloglock               if time   - check st ctime  gt  0 01  remove  errloglock               print  waiting my turn     EDIT--- After thinking over some of the comments about stale locks above I edited the code to add a check for staleness of the  lock file    Timing several thousand iterations of this function on my system gave and average of 0 002066    seconds from just before   lock   open  errloglock    w     to just after   remove  errloglock     so I figured I will start with 5 times that amount to indicate staleness and monitor the situation for problems   Also  as I was working with the timing  I realized that I had a bit of code  that was not really necessary   lock close     which I had immediately following the open statement  so I have removed it in this edit

User · Answer

this worked for me  Do not occupy large files  distribute in several small ones you create file Temp  delete file A and then rename file Temp to A  import os import json  def Server        i   0     while i    0          try                          with open File Temp   quot w quot   as file                      json dump DATA  file  indent 2                  if os path exists File A                       os remove File A                  os rename File Temp  File A                  i   1         except OSError as e                  print   quot file locked   quot   str e                   time sleep 1                            def Clients        i   0     while i    0          try              if os path exists File A                   with open File A  quot r quot   as file                      DATA Temp   file read               DATA   json loads DATA Temp              i   1         except OSError as e              print  str e               time sleep 1

User · Answer

I found a simple and worked    implementation from grizzled-python   Simple use os open      O EXCL    os close   didn t work on windows

User · Answer

Locking a file is usually a platform-specific operation  so you may need to allow for the possibility of running on different operating systems  For example   import os  def my lock f       if os name     posix             Unix or OS X specific locking here     elif os name     nt             Windows specific locking here     else          print  Unknown operating system  lock unavailable

User · Answer

I have been looking at several solutions to do that and my choice has been oslo concurrency  It s powerful and relatively well documented  It s based on fasteners   Other solutions    Portalocker  requires pywin32  which is an exe installation  so not possible via pip fasteners  poorly documented lockfile  deprecated flufl lock  NFS-safe file locking for POSIX systems  simpleflock   Last update 2013-07 zc lockfile   Last update 2016-06  as of 2017-03  lock file   Last update in 2007-10

User · Answer

Locking is platform and device specific  but generally  you have a few options    Use flock    or equivalent  if your os supports it    This is advisory locking  unless you check for the lock  its ignored  Use a lock-copy-move-unlock methodology  where you copy the file  write the new data  then move it  move  not copy - move is an atomic operation in Linux -- check your OS   and you check for the existence of the lock file  Use a directory as a  lock    This is necessary if you re writing to NFS  since NFS doesn t support flock    There s also the possibility of using shared memory between the processes  but I ve never tried that  it s very OS-specific    For all these methods  you ll have to use a spin-lock  retry-after-failure  technique for acquiring and testing the lock   This does leave a small window for mis-synchronization  but its generally small enough to not be an major issue   If you re looking for a solution that is cross platform  then you re better off logging to another system via some other mechanism  the next best thing is the NFS technique above      Note that sqlite is subject to the same constraints over NFS that normal files are  so you can t write to an sqlite database on a network share and get synchronization for free

User · Answer

Alright  so I ended up going with the code I wrote here  on my website link is dead  view on archive org  also available on GitHub   I can use it in the following fashion   from filelock import FileLock  with FileLock  myfile txt lock        print  Lock acquired        with open  myfile txt              work with the file as it is now locked

User · Answer

I found a simple and worked    implementation from grizzled-python   Simple use os open      O EXCL    os close   didn t work on windows

User · Answer

To add on to Evan Fossmark s answer  here s an example of how to use filelock   from filelock import FileLock  lockfile   r c  scr txt  lock   FileLock lockfile     lock   with lock      file   open path   w       file write  123       file close     Any code within the with lock  block is thread-safe  meaning that it will be finished before another process has access to the file

User · Answer

There is a cross-platform file locking module here  Portalocker  Although as Kevin says  writing to a file from multiple processes at once is something you want to avoid if at all possible   If you can shoehorn your problem into a database  you could use SQLite   It supports concurrent access and handles its own locking

User · Answer

this worked for me  Do not occupy large files  distribute in several small ones you create file Temp  delete file A and then rename file Temp to A  import os import json  def Server        i   0     while i    0          try                          with open File Temp   quot w quot   as file                      json dump DATA  file  indent 2                  if os path exists File A                       os remove File A                  os rename File Temp  File A                  i   1         except OSError as e                  print   quot file locked   quot   str e                   time sleep 1                            def Clients        i   0     while i    0          try              if os path exists File A                   with open File A  quot r quot   as file                      DATA Temp   file read               DATA   json loads DATA Temp              i   1         except OSError as e              print  str e               time sleep 1

User · Answer

The other solutions cite a lot of external code bases  If you would prefer to do it yourself  here is some code for a cross-platform solution that uses the respective file locking tools on Linux   DOS systems  try        Posix based file locking  Linux  Ubuntu  MacOS  etc           Only allows locking on writable files  might cause         strange results for reading      import fcntl  os     def lock file f           if f writable    fcntl lockf f  fcntl LOCK EX      def unlock file f           if f writable    fcntl lockf f  fcntl LOCK UN  except ModuleNotFoundError        Windows file locking     import msvcrt  os     def file size f           return os path getsize  os path realpath f name        def lock file f           msvcrt locking f fileno    msvcrt LK RLCK  file size f       def unlock file f           msvcrt locking f fileno    msvcrt LK UNLCK  file size f       Class for ensuring that all file operations are atomic  treat   initialization like a standard call to  open  that happens to be atomic    This file opener  must  be used in a  quot with quot  block  class AtomicOpen        Open the file with arguments provided by user  Then acquire       a lock on that file object  WARNING  Advisory locking       def   init   self  path   args    kwargs             Open the file and acquire a lock on the file before operating         self file   open path  args    kwargs            Lock the opened file         lock file self file         Return the opened file object  knowing a lock has been obtained       def   enter   self   args    kwargs   return self file        Unlock the file and close the file object      def   exit   self  exc type None  exc value None  traceback None                     Flush to make sure all buffered contents are written to file          self file flush           os fsync self file fileno              Release the lock on the file          unlock file self file          self file close             Handle exceptions that may have come up during execution  by           default any exceptions are raised to the user          if  exc type    None   return False         else                   return True          Now  AtomicOpen can be used in a with block where one would normally use an open statement  WARNINGS   If running on Windows and Python crashes before exit is called  I m not sure what the lock behavior would be  The locking provided here is advisory  not absolute  All potentially competing processes must use the  quot AtomicOpen quot  class  As of  Nov 9th  2020  this code only locks writable files on Posix systems  At some point after the posting and before this date  it became illegal to use the fcntl lock on read-only files

User · Answer

The scenario is like that  The user requests a file to do something  Then  if the user sends the same request again  it informs the user that the second request is not done until the first request finishes  That s why  I use lock-mechanism to handle this issue   Here is my working code    from lockfile import LockFile lock   LockFile lock file path  status      if not lock is locked        lock acquire       status   lock path     is locked       print status else      status   lock path     is already locked       print status  return status

User · Answer

You may find pylocker very useful  It can be used to lock a file or for locking mechanisms in general and can be accessed from multiple Python processes at once    If you simply  want to lock a file here s how it works    import uuid from pylocker import Locker     create a unique lock pass  This can be any string  lpass   str uuid uuid1       create locker instance  FL   Locker filePath  myfile txt   lockPass lpass  mode  w      aquire the lock with FL as r        get the result     acquired  code  fd    r        check if aquired      if fd is not None          print fd         fd write  I have succesfuly aquired the lock        no need to release anything or to close the file descriptor     with statement takes care of that  let s print fd and verify that  print fd

User · Answer

To add on to Evan Fossmark s answer  here s an example of how to use filelock   from filelock import FileLock  lockfile   r c  scr txt  lock   FileLock lockfile     lock   with lock      file   open path   w       file write  123       file close     Any code within the with lock  block is thread-safe  meaning that it will be finished before another process has access to the file

User · Answer

Locking is platform and device specific  but generally  you have a few options    Use flock    or equivalent  if your os supports it    This is advisory locking  unless you check for the lock  its ignored  Use a lock-copy-move-unlock methodology  where you copy the file  write the new data  then move it  move  not copy - move is an atomic operation in Linux -- check your OS   and you check for the existence of the lock file  Use a directory as a  lock    This is necessary if you re writing to NFS  since NFS doesn t support flock    There s also the possibility of using shared memory between the processes  but I ve never tried that  it s very OS-specific    For all these methods  you ll have to use a spin-lock  retry-after-failure  technique for acquiring and testing the lock   This does leave a small window for mis-synchronization  but its generally small enough to not be an major issue   If you re looking for a solution that is cross platform  then you re better off logging to another system via some other mechanism  the next best thing is the NFS technique above      Note that sqlite is subject to the same constraints over NFS that normal files are  so you can t write to an sqlite database on a network share and get synchronization for free

User · Answer

I have been working on a situation like this where I run multiple copies of the same program from within the same directory folder and logging errors   My approach was to write a  lock file  to the disc before opening the log file  The program checks for the presence of the  lock file  before proceeding  and waits for its turn if the  lock file  exists   Here is the code   def errlogger error        while True          if not exists  errloglock                lock   open  errloglock    w               if exists  errorlog    log   open  errorlog    a               else  log   open  errorlog    w               log write str datetime utcnow    0 -7          error     n               log close               remove  errloglock               return         else              check   stat  errloglock               if time   - check st ctime  gt  0 01  remove  errloglock               print  waiting my turn     EDIT--- After thinking over some of the comments about stale locks above I edited the code to add a check for staleness of the  lock file    Timing several thousand iterations of this function on my system gave and average of 0 002066    seconds from just before   lock   open  errloglock    w     to just after   remove  errloglock     so I figured I will start with 5 times that amount to indicate staleness and monitor the situation for problems   Also  as I was working with the timing  I realized that I had a bit of code  that was not really necessary   lock close     which I had immediately following the open statement  so I have removed it in this edit

User · Answer

I have been looking at several solutions to do that and my choice has been oslo concurrency  It s powerful and relatively well documented  It s based on fasteners   Other solutions    Portalocker  requires pywin32  which is an exe installation  so not possible via pip fasteners  poorly documented lockfile  deprecated flufl lock  NFS-safe file locking for POSIX systems  simpleflock   Last update 2013-07 zc lockfile   Last update 2016-06  as of 2017-03  lock file   Last update in 2007-10

User · Answer

Coordinating access to a single file at the OS level is fraught with all kinds of issues that you probably don t want to solve   Your best bet is have a separate process that coordinates read write access to that file

[python] Locking a file in Python

Examples related to python

Examples related to file-locking