How to search file text for a pattern and replace it with a given value

Question

I m looking for a script to search a file  or list of files  for a pattern and  if found  replace that pattern with a given value   Thoughts

User · Answer

Here s a solution for find replace in all files of a given directory  Basically I took the answer provided by sepp2k and expanded it     First set the files to search replace in files   Dir glob   PATH        Then set the variables for find replace  original string or regex    REGEX   replacement string    STRING   files each do  file name    text   File read file name    replace   text gsub   original string or regex   replacement string    File open file name   w      file  file puts replace   end

User · Answer

Disclaimer  This approach is a naive illustration of Ruby s capabilities  and not a production-grade solution for replacing strings in files  It s prone to various failure scenarios  such as data loss in case of a crash  interrupt  or disk being full  This code is not fit for anything beyond a quick one-off script where all the data is backed up  For that reason  do NOT copy this code into your programs   Here s a quick short way to do it   file names     foo txt    bar txt    file names each do  file name    text   File read file name    new contents   text gsub  search regexp    replacement string        To merely print the contents of the file  use    puts new contents      To write changes to the file  use    File open file name   w     file  file puts new contents   end

User · Answer

Here an alternative to the one liner from jim  this time in a script  ARGV 0  -3  each  f  File write f  File read f  gsub ARGV -2  ARGV -1       Save it in a script  eg replace rb  You start in on the command line with  replace rb   txt  lt string to replace gt   lt replacement gt      txt can be replaced with another selection or with some filenames or paths  broken down so that I can explain what s happening but still executable    ARGV is an array of the arguments passed to the script  ARGV 0  -3  each do  f    enumerate the arguments of this script from the first to the last  -1  minus 2   File write f     open the argument    filename  for writing     File read f    open the argument    filename  for reading      gsub ARGV -2  ARGV -1      and replace all occurances of the beforelast with the last argument  string  end   EDIT  if you want to use a regular expression use this instead Obviously  this is only for handling relatively small text files  no Gigabyte monsters  ARGV 0  -3  each  f  File write f  File read f  gsub    ARGV -2    ARGV -1

User · Answer

Keep in mind that  when you do this  the filesystem could be out of space and you may create a zero-length file   This is catastrophic if you re doing something like writing out  etc passwd files as part of system configuration management  Note that in-place file editing like in the accepted answer will always truncate the file and write out the new file sequentially   There will always be a race condition where concurrent readers will see a truncated file   If the process is aborted for any reason  ctrl-c  OOM killer  system crash  power outage  etc  during the write then the truncated file will also be left over  which can be catastrophic   This is the kind of dataloss scenario which developers MUST consider because it will happen   For that reason  I think the accepted answer should most likely not be the accepted answer  At a bare minimum write to a tempfile and move rename the file into place like the  quot simple quot  solution at the end of this answer  You need to use an algorithm that   Reads the old file and writes out to the new file   You need to be careful about slurping entire files into memory    Explicitly closes the new temporary file  which is where you may throw an exception because the file buffers cannot be written to disk because there is no space   Catch this and cleanup the temporary file if you like  but you need to rethrow something or fail fairly hard at this point   Fixes the file permissions and modes on the new file   Renames the new file and drops it into place    With ext3 filesystems you are guaranteed that the metadata write to move the file into place will not get rearranged by the filesystem and written before the data buffers for the new file are written  so this should either succeed or fail   The ext4 filesystem has also been patched to support this kind of behavior   If you are very paranoid you should call the fdatasync   system call as a step 3 5 before moving the file into place  Regardless of language  this is best practice   In languages where calling close   does not throw an exception  Perl or C  you must explicitly check the return of close   and throw an exception if it fails  The suggestion above to simply slurp the file into memory  manipulate it and write it out to the file will be guaranteed to produce zero-length files on a full filesystem   You need to always use FileUtils mv to move a fully-written temporary file into place  A final consideration is the placement of the temporary file   If you open a file in  tmp then you have to consider a few problems   If  tmp is mounted on a different file system you may run  tmp out of space before you ve written out the file that would otherwise be deployable to the destination of the old file   Probably more importantly  when you try to mv the file across a device mount you will transparently get converted to cp behavior  The old file will be opened  the old files inode will be preserved and reopened and the file contents will be copied   This is most likely not what you want  and you may run into  quot text file busy quot  errors if you try to edit the contents of a running file   This also defeats the purpose of using the filesystem mv commands and you may run the destination filesystem out of space with only a partially written file  This also has nothing to do with Ruby s implementation   The system mv and cp commands behave similarly    What is more preferable is to open a Tempfile in the same directory as the old file   This ensures that there will be no cross-device move issues   The mv itself should never fail  and you should always get a complete and untruncated file   Any failures  such as device out of space  permission errors  etc   should be encountered during writing the Tempfile out  The only downsides to the approach of creating the Tempfile in the destination directory are   Sometimes you may not be able to open a Tempfile there  such as if you are trying to  edit  a file in  proc for example   For that reason you might want to fall back and try  tmp if opening the file in the destination directory fails  You must have enough space on the destination partition in order to hold both the complete old file and the new file   However  if you have insufficient space to hold both copies then you are probably short on disk space and the actual risk of writing a truncated file is much higher  so I would argue this is a very poor tradeoff outside of some exceedingly narrow  and well-monitored  edge cases   Here s some code that implements the full-algorithm  windows code is untested and unfinished      usr bin env ruby  require  tempfile   def file edit filename  regexp  replacement    tempdir   File dirname filename    tempprefix   File basename filename    tempprefix prepend      unless RUBY PLATFORM     mswin mingw windows    tempfile       begin       Tempfile new tempprefix  tempdir      rescue       Tempfile new tempprefix      end   File open filename  each do  line      tempfile puts line gsub regexp  replacement    end   tempfile fdatasync unless RUBY PLATFORM     mswin mingw windows    tempfile close   unless RUBY PLATFORM     mswin mingw windows      stat   File stat filename      FileUtils chown stat uid  stat gid  tempfile path     FileUtils chmod stat mode  tempfile path   else       FIXME  apply perms on windows   end   FileUtils mv tempfile path  filename end  file edit   tmp foo    foo    quot baz quot    And here is a slightly tighter version that doesn t worry about every possible edge case  if you are on Unix and don t care about writing to  proc      usr bin env ruby  require  tempfile   def file edit filename  regexp  replacement    Tempfile open  quot    File basename filename   quot   File dirname filename   do  tempfile      File open filename  each do  line        tempfile puts line gsub regexp  replacement      end     tempfile fdatasync     tempfile close     stat   File stat filename      FileUtils chown stat uid  stat gid  tempfile path     FileUtils chmod stat mode  tempfile path     FileUtils mv tempfile path  filename   end end  file edit   tmp foo    foo    quot baz quot    The really simple use-case  for when you don t care about file system permissions  either you re not running as root  or you re running as root and the file is root owned      usr bin env ruby  require  tempfile   def file edit filename  regexp  replacement    Tempfile open  quot    File basename filename   quot   File dirname filename   do  tempfile      File open filename  each do  line        tempfile puts line gsub regexp  replacement      end     tempfile close     FileUtils mv tempfile path  filename   end end  file edit   tmp foo    foo    quot baz quot    TL DR  That should be used instead of the accepted answer at a minimum  in all cases  in order to ensure the update is atomic and concurrent readers will not see truncated files   As I mentioned above  creating the Tempfile in the same directory as the edited file is important here to avoid cross device mv operations being translated into cp operations if  tmp is mounted on a different device   Calling fdatasync is an added layer of paranoia  but it will incur a performance hit  so I omitted it from this example since it is not commonly practiced

User · Answer

require  trollop   opts   Trollop  options do   opt  output   Output file    type   gt  String   opt  input   Input file    type   gt  String   opt  ss   String to search    type   gt  String   opt  rs   String to replace    type   gt  String end  text   File read opts input  text gsub  opts ss  opts rs  File open opts output   w      f  f write text

User · Answer

In my case I am using tty-gem ruby gem for that  I also needed appending  prepending  on a given text regex inside the file   diffs  and others  The gem includes all that and a fairly clear documentation

User · Answer

If you need to do substitutions across line boundaries  then using ruby -pi -e won t work because the p processes one line at a time  Instead  I recommend the following  although it could fail with a multi-GB file   ruby -e  file  translation ja yml   IO write file   IO read file  gsub   s       q          The is looking for white space  potentially including new lines  following by a quote  in which case it gets rid of the whitespace  The  q   is just a fancy way of quoting the quote character

User · Answer

Another approach is to use inplace editing inside Ruby  not from the command line       usr bin ruby  def inplace edit file  bak   amp block      old stdout    stdout     argf   ARGF clone      argf argv replace  file      argf inplace mode   bak     argf each line do  line          yield line     end     argf close       stdout   old stdout end  inplace edit  test txt     bak  do  line      line   line gsub  search1   replace1       line   line gsub  search2   replace2       print line unless line match  something   end   If you don t want to create a backup then change   bak  to

User · Answer

This works for me   filename    foo  text   File read filename   content   text gsub  search regexp    replacestring   File open filename   w      file  file  lt  lt  content

User · Answer

Actually  Ruby does have an in-place editing feature   Like Perl  you can say  ruby -pi bak -e  gsub  oldtext    newtext      txt   This will apply the code in double-quotes to all files in the current directory whose names end with   txt    Backup copies of edited files will be created with a   bak  extension   foobar txt bak  I think    NOTE  this does not appear to work for multiline searches   For those  you have to do it the other less pretty way  with a wrapper script around the regex

User · Answer

There isn t really a way to edit files in-place  What you usually do when you can get away with it  i e  if the files are not too big  is  you read the file into memory  File read   perform your substitutions on the read string  String gsub  and then write the changed string back to the file  File open  File write    If the files are big enough for that to be unfeasible  what you need to do  is read the file in chunks  if the pattern you want to replace won t span multiple lines then one chunk usually means one line - you can use File foreach to read a file line by line   and for each chunk perform the substitution on it and append it to a temporary file  When you re done iterating over the source file  you close it and use FileUtils mv to overwrite it with the temporary file

[ruby] How to search file text for a pattern and replace it with a given value

Examples related to ruby

Examples related to file-io