How to delete from a text file all lines that contain a specific string

Question

How would I use sed to delete all lines in a text file that contain a specific string

User · Answer

I was struggling with this on Mac  Plus  I needed to do it using variable replacement   So I used   sed -i       pattern d   file  where  file is the file where deletion is needed and  pattern is the pattern to be matched for deletion   I picked the    from this comment   The thing to note here is use of double quotes in    pattern d   Variable won t work when we use single quotes

User · Answer

You can use sed to replace lines in place in a file  However  it seems to be much slower than using grep for the inverse into a second file and then moving the second file over the original   e g   sed -i   pattern d  filename         or  grep -v  pattern  filename  gt  filename2  mv filename2 filename   The first command takes 3 times longer on my machine anyway

User · Answer

Curiously enough  the accepted answer does not actually answer the question directly  The question asks about using sed to replace a string  but the answer seems to presuppose knowledge of how to convert an arbitrary string into a regex  Many programming language libraries have a function to perform such a transformation  e g  python  re escape STRING  ruby  Regexp escape STRING  java   Pattern quote STRING   But how to do it on the command line  Since this is a sed-oriented question  one approach would be to use sed itself  sed  s                      1 g   So given an arbitrary string  STRING we could write something like  re   sed  s                     1 g   lt  lt  lt   quot  STRING quot   sed  quot   re d quot  FILE  or as a one-liner   sed  quot    sed  s                      1 g   lt  lt  lt   quot  STRING quot   d quot    with variations as described elsewhere on this page

User · Answer

to show the treated text in console  cat filename   sed   text to remove d     to save treated text into a file   cat filename   sed   text to remove d   gt  newfile   to append treated text info an existing file  cat filename   sed   text to remove d   gt  gt  newfile   to treat already treated text  in this case remove more lines of what has been removed  cat filename   sed   text to remove d    sed   remove this too d    more   the   more will show text in chunks of one page at a time

User · Answer

You can also use this    grep -v  pattern  filename   Here -v will print only other than your pattern  that means invert match

User · Answer

SED      James  John d  -n   James  John  p    AWK       James John    James John   next    print    GREP    -v  James  John

User · Answer

You can use good old ed to edit a file in a similar fashion to the answer that uses ex  The big difference in this case is that ed takes its commands via standard input  not as command line arguments like ex can  When using it in a script  the usual way to accomodate this is to use printf to pipe commands to it   printf   s n   g pattern d  w   ed -s filename   or with a heredoc   ed -s filename  lt  lt EOF g pattern d w EOF

User · Answer

There are many other ways to delete lines with specific string besides sed   AWK  awk    pattern   file  gt  temp  amp  amp  mv temp file   Ruby  1 9    ruby -i bak -ne  print if not  test   file   Perl  perl -ni bak -e  print unless  pattern   file   Shell  bash 3 2 and later   while read -r line do         line    pattern     amp  amp  echo   line  done  lt file  gt  o mv o file   GNU grep  grep -v  pattern  file  gt  temp  amp  amp  mv temp file   And of course sed  printing the inverse is faster than actual deletion    sed -n   pattern  p  file

User · Answer

You may consider using ex  which is a standard Unix command-based editor    ex  g match d -cwq file   where      executes given Ex command  man ex   same as -c which executes wq  write and quit  g match d - Ex command to delete lines with given match  see  Power of g   The above example is a POSIX-compliant method for in-place editing a file as per this post at Unix SE and POSIX specifications for ex     The difference with sed is that      sed is a Stream EDitor  not a file editor BashFAQ   Unless you enjoy unportable code  I O overhead and some other bad side effects  So basically some parameters  such as in-place -i  are non-standard FreeBSD extensions and may not be available on other operating systems

User · Answer

To remove the line and print the output to standard out   sed   pattern to match d    infile   To directly modify the file     does not work with BSD sed   sed -i   pattern to match d    infile   Same  but for BSD sed  Mac OS X and FreeBSD      does not work with GNU sed   sed -i      pattern to match d    infile   To directly modify the file  and create a backup      works with BSD and GNU sed   sed -i bak   pattern to match d    infile

User · Answer

echo -e   thing to delete ndd 033 x n    vim file to edit txt

User · Answer

I have made a small benchmark with a file which contains approximately 345 000 lines  The way with grep seems to be around 15 times faster than the sed method in this case   I have tried both with and without the setting LC ALL C  it does not seem change the timings significantly  The search string  CDGA 00004 pdbqt gz tar  is somewhere in the middle of the file   Here are the commands and the timings   time sed -i   CDGA 00004 pdbqt gz tar d   tmp input txt  real    0m0 711s user    0m0 179s sys     0m0 530s  time perl -ni -e  print unless  CDGA 00004 pdbqt gz tar    tmp input txt  real    0m0 105s user    0m0 088s sys     0m0 016s  time  grep -v CDGA 00004 pdbqt gz tar  tmp input txt  gt   tmp input tmp  mv  tmp input tmp  tmp input txt    real    0m0 046s user    0m0 014s sys     0m0 019s

User · Answer

To get a inplace like result with grep you can do this   echo    grep -v  pattern  filename    gt filename

User · Answer

perl -i    -nle  regexp   print  file1 file2 file3 perl -i bk -nle  regexp   print  file1 file2 file3   The first command edits the file s  inplace  -i    The second command does the same thing but keeps a copy or backup of the original file s  by adding  bk to the file names   bk can be changed to anything

User · Answer

Delete lines from all files that match the match grep -rl  text to search      xargs sed -i   text to search d

User · Answer

The easy way to do it  with GNU sed   sed --in-place   some string here d  yourfile

User · Answer

You can also delete a range of lines in a file  For example to delete stored procedures in a SQL file  sed   CREATE PROCEDURE     END   d  sqllines sql This will remove all lines between CREATE PROCEDURE and END    I have cleaned up many sql files withe this sed command

User · Answer

Just in case someone wants to do it for exact matches of strings  you can use the -w flag in grep - w for whole  That is  for example if you want to delete the lines that have number 11  but keep the lines with number 111   -bash-4 1  head file 1 11 111  -bash-4 1  grep -v  11  file 1  -bash-4 1  grep -w -v  11  file 1 111   It also works with the -f flag if you want to exclude several exact patterns at once  If  blacklist  is a file with several patterns on each line that you want to delete from  file    grep -w -v -f blacklist file

User · Answer

cat filename   grep -v  pattern   gt  filename 1 mv filename 1 filename

[shell] How to delete from a text file, all lines that contain a specific string?

Examples related to shell

Examples related to sed

Examples related to text-parsing

Examples related to in-place