[bash] find difference between two text files with one item per line

I have two files:

file 1

dsf
sdfsd
dsfsdf

file 2

ljljlj 
lkklk 
dsf
sdfsd
dsfsdf

I want to display what is in file 2 but not in file 1, so file 3 should look like

ljljlj 
lkklk 

This question is related to bash file scripting sed awk

The answer is


if you are expecting them in a certain order, you can just use diff

diff file1 file2 | grep ">"


grep -Fxvf file1 file2

What the flags mean:

-F, --fixed-strings
              Interpret PATTERN as a list of fixed strings, separated by newlines, any of which is to be matched.    
-x, --line-regexp
              Select only those matches that exactly match the whole line.
-v, --invert-match
              Invert the sense of matching, to select non-matching lines.
-f FILE, --file=FILE
              Obtain patterns from FILE, one per line.  The empty file contains zero patterns, and therefore matches nothing.

I successfully used

diff "${file1}" "${file2}" | grep "<" | sed 's/^<//g' > "${diff_file}"

Outputting the difference to a file.


an awk answer:

awk 'NR == FNR {file1[$0]++; next} !($0 in file1)' file1 file2


With GNU sed:

sed 's#[^^]#[&]#g;s#\^#\\^#g;s#^#/^#;s#$#$/d#' file1 | sed -f- file2

How it works:

The first sed produces an output like this:

/^[d][s][f]$/d
/^[s][d][f][s][d]$/d
/^[d][s][f][s][d][f]$/d

Then it is used as a sed script by the second sed.


If you want to use loops You can try like this: (diff and cmp are much more efficient. )

while read line
do
    flag = 0
    while read line2
    do
       if ( "$line" = "$line2" )
        then
            flag = 1
        fi
     done < file1 
     if ( flag -eq 0 )
     then
         echo $line > file3
     fi
done < file2

Note: The program is only to provide a basic insight into what can be done if u dont want to use system calls such as diff n comm..


A tried a slight variation on Luca's answer and it worked for me.

diff file1 file2 | grep ">" | sed 's/^> //g' > diff_file

Note that the searched pattern in sed is a > followed by a space.


You can use the comm command to compare two sorted files

comm -13 <(sort file1) <(sort file2)

join -v 2 <(sort file1) <(sort file2)

file1 
m1
m2
m3

file2 
m2
m4
m5

>awk 'NR == FNR {file1[$0]++; next} !($0 in file1)' file1 file2
m4
m5

>awk 'NR == FNR {file1[$0]++; next} ($0 in file1)' file1 file2
m2

> What's awk command to get 'm1 and m3' ??  as in file1 and not in file2? 
m1
m3

Examples related to bash

Comparing a variable with a string python not working when redirecting from bash script Zipping a file in bash fails How do I prevent Conda from activating the base environment by default? Get first line of a shell command's output Fixing a systemd service 203/EXEC failure (no such file or directory) /bin/sh: apt-get: not found VSCode Change Default Terminal Run bash command on jenkins pipeline How to check if the docker engine and a docker container are running? How to switch Python versions in Terminal?

Examples related to file

Gradle - Move a folder from ABC to XYZ Difference between opening a file in binary vs text Angular: How to download a file from HttpClient? Python error message io.UnsupportedOperation: not readable java.io.FileNotFoundException: class path resource cannot be opened because it does not exist Writing JSON object to a JSON file with fs.writeFileSync How to read/write files in .Net Core? How to write to a CSV line by line? Writing a dictionary to a text file? What are the pros and cons of parquet format compared to other formats?

Examples related to scripting

What does `set -x` do? Creating an array from a text file in Bash Windows batch - concatenate multiple text files into one Raise error in a Bash script How do I assign a null value to a variable in PowerShell? Difference between ${} and $() in Bash Using a batch to copy from network drive to C: or D: drive Check if a string matches a regex in Bash script How to run a script at a certain time on Linux? How to make an "alias" for a long path?

Examples related to sed

Retrieve last 100 lines logs How to replace multiple patterns at once with sed? Insert multiple lines into a file after specified pattern using shell script Linux bash script to extract IP address Ansible playbook shell output remove white space from the end of line in linux bash, extract string before a colon invalid command code ., despite escaping periods, using sed RE error: illegal byte sequence on Mac OS X How to use variables in a command in sed?

Examples related to awk

What are NR and FNR and what does "NR==FNR" imply? awk - concatenate two string variable and assign to a third Printing column separated by comma using Awk command line Insert multiple lines into a file after specified pattern using shell script cut or awk command to print first field of first row How to run an awk commands in Windows? Linux bash script to extract IP address Print line numbers starting at zero using awk Trim leading and trailing spaces from a string in awk Use awk to find average of a column