[linux] Find the files existing in one directory but not in the other

I'm trying to find the files existing in one directory but not in the other, I tried to use this command:

diff -q dir1 dir2

The problem with the above command that it finds both the files in dir1 but not in dir2 as well as the files in dir2 but not in dir1,

I am trying to find the files in dir1 but not in dir2 only.

Here's a small sample of what my data looks like

dir1    dir2    dir3
1.txt   1.txt   1.txt
2.txt   3.txt   3.txt
5.txt   4.txt   5.txt
6.txt   7.txt   8.txt

Another question on my mind is how can I find the files in dir1 but not in dir2 or dir3 in a single command?

This question is related to linux bash diff

The answer is


diff -r dir1 dir2 | grep dir1 | awk '{print $4}' > difference1.txt

Explanation:

  • diff -r dir1 dir2 shows which files are only in dir1 and those only in dir2 and also the changes of the files present in both directories if any.

  • diff -r dir1 dir2 | grep dir1 shows which files are only in dir1

  • awk to print only filename.


vim's DirDiff plugin is another very useful tool for comparing directories.

vim -c "DirDiff dir1 dir2"

It not only lists which files are different between the directories, but also allows you to inspect/modify with vimdiff the files that are different.


A good way to do this comparison is to use find with md5sum, then a diff.

Example:

Use find to list all the files in the directory then calculate the md5 hash for each file and pipe it to a file:

find /dir1/ -type f -exec md5sum {} \; > dir1.txt

Do the same procedure to the another directory:

find /dir2/ -type f -exec md5sum {} \; > dir2.txt

Then compare the result two files with "diff":

diff dir1.txt dir2.txt

This strategy is very useful when the two directories to be compared are not in the same machine and you need to make sure that the files are equal in both directories.

Another good way to do the job is using git

git diff --no-index dir1/ dir2/

Best regards!


This should do the job:

diff -rq dir1 dir2

Options explained (via diff(1) man page):

  • -r - Recursively compare any subdirectories found.
  • -q - Output only whether files differ.

Unsatisfied with all the replies, since most of them work very slowly and produce unnecessarily long output for large directories, I wrote my own Python script to compare two folders.

Unlike many other solutions, it doesn't compare contents of the files. Also it doesn't go inside subdirectories which are missing in another directory. So the output is quite concise and the script works fast.

#!/usr/bin/env python3

import os, sys

def compare_dirs(d1: "old directory name", d2: "new directory name"):
    def print_local(a, msg):
        print('DIR ' if a[2] else 'FILE', a[1], msg)
    # ensure validity
    for d in [d1,d2]:
        if not os.path.isdir(d):
            raise ValueError("not a directory: " + d)
    # get relative path
    l1 = [(x,os.path.join(d1,x)) for x in os.listdir(d1)]
    l2 = [(x,os.path.join(d2,x)) for x in os.listdir(d2)]
    # determine type: directory or file?
    l1 = sorted([(x,y,os.path.isdir(y)) for x,y in l1])
    l2 = sorted([(x,y,os.path.isdir(y)) for x,y in l2])
    i1 = i2 = 0
    common_dirs = []
    while i1<len(l1) and i2<len(l2):
        if l1[i1][0] == l2[i2][0]:      # same name
            if l1[i1][2] == l2[i2][2]:  # same type
                if l1[i1][2]:           # remember this folder for recursion
                    common_dirs.append((l1[i1][1], l2[i2][1]))
            else:
                print_local(l1[i1],'type changed')
            i1 += 1
            i2 += 1
        elif l1[i1][0]<l2[i2][0]:
            print_local(l1[i1],'removed')
            i1 += 1
        elif l1[i1][0]>l2[i2][0]:
            print_local(l2[i2],'added')
            i2 += 1
    while i1<len(l1):
        print_local(l1[i1],'removed')
        i1 += 1
    while i2<len(l2):
        print_local(l2[i2],'added')
        i2 += 1
    # compare subfolders recursively
    for sd1,sd2 in common_dirs:
        compare_dirs(sd1, sd2)

if __name__=="__main__":
    compare_dirs(sys.argv[1], sys.argv[2])

Sample usage:

user@laptop:~$ python3 compare_dirs.py dir1/ dir2/
DIR  dir1/out/flavor-domino removed
DIR  dir2/out/flavor-maxim2 added
DIR  dir1/target/vendor/flavor-domino removed
DIR  dir2/target/vendor/flavor-maxim2 added
FILE dir1/tmp/.kconfig-flavor_domino removed
FILE dir2/tmp/.kconfig-flavor_maxim2 added
DIR  dir2/tools/tools/LiveSuit_For_Linux64 added

Or if you want to see only files from the first directory:

user@laptop:~$ python3 compare_dirs.py dir2/ dir1/ | grep dir1
DIR  dir1/out/flavor-domino added
DIR  dir1/target/vendor/flavor-domino added
FILE dir1/tmp/.kconfig-flavor_domino added

P.S. If you need to compare file sizes and file hashes for potential changes, I published an updated script here: https://gist.github.com/amakukha/f489cbde2afd32817f8e866cf4abe779


A simplified way to compare 2 directories using the DIFF command

diff filename.1 filename.2 > filename.dat >>Enter

open filename.dat after the run is complete

and you will see: Only in filename.1: filename.2 Only in: directory_name: name_of_file1 Only in: directory_Name: name_of_file2


This is the bash script to print commands for syncing two directories

dir1=/tmp/path_to_dir1
dir2=/tmp/path_to_dir2
diff -rq $dir1 $dir2 | sed -e "s|Only in $dir2\(.*\): \(.*\)|cp -r $dir2\1/\2 $dir1\1|" |  sed -e "s|Only in $dir1\(.*\): \(.*\)|cp -r $dir1\1/\2 $dir2\1|" 

The accepted answer will also list the files that exist in both directories, but have different content. To list ONLY the files that exist in dir1 you can use:

diff -r dir1 dir2 | grep 'Only in' | grep dir1 | awk '{print $4}' > difference1.txt

Explanation:

  • diff -r dir1 dir2 : compare
  • grep 'Only in': get lines that contain 'Only in'
  • grep dir1 : get lines that contain dir

This is a bit late but may help someone. Not sure if diff or rsync spit out just filenames in a bare format like this. Thanks to plhn for giving that nice solution which I expanded upon below.

If you want just the filenames so it's easy to just copy the files you need in a clean format, you can use the find command.

comm -23 <(find dir1 | sed 's/dir1/\//'| sort) <(find dir2 | sed 's/dir2/\//'| sort) | sed 's/^\//dir1/'

This assumes that both dir1 and dir2 are in the same parent folder. sed just removes the parent folder so you can compare apples with apples. The last sed just puts the dir1 name back.

If you just want files:

comm -23 <(find dir1 -type f | sed 's/dir1/\//'| sort) <(find dir2 -type f | sed 's/dir2/\//'| sort) | sed 's/^\//dir1/'

Similarly for directories:

comm -23 <(find dir1 -type d | sed 's/dir1/\//'| sort) <(find dir2 -type d | sed 's/dir2/\//'| sort) | sed 's/^\//dir1/'

GNU grep can inverse the search with the option -v. This makes grep reporting the lines, which do not match. By this you can remove the files in dir2 from the list of files in dir1.

grep -v -F -x -f <(find dir2 -type f -printf '%P\n') <(find dir1 -type f -printf '%P\n')

The options -F -x tell grep to perform a string search on the whole line.


Meld (http://meldmerge.org/) does a great job at comparing directories and the files within.

Meld comparing directories


comm -23 <(ls dir1 |sort) <(ls dir2|sort)

This command will give you files those are in dir1 and not in dir2.

About <( ) sign, you can google it as 'process substitution'.


Another (maybe faster for large directories) approach:

$ find dir1 | sed 's,^[^/]*/,,' | sort > dir1.txt && find dir2 | sed 's,^[^/]*/,,' | sort > dir2.txt
$ diff dir1.txt dir2.txt

The sed command removes the first directory component thanks to Erik`s post)


This answer optimizes one of the suggestions from @Adail-Junior by adding the -D option, which is helpful when neither of the directories being compared are git repositories:

git diff -D --no-index dir1/ dir2/

If you use -D then you won't see comparisons to /dev/null: text Binary files a/whatever and /dev/null differ


Examples related to linux

grep's at sign caught as whitespace How to prevent Google Colab from disconnecting? "E: Unable to locate package python-pip" on Ubuntu 18.04 How to upgrade Python version to 3.7? Install Qt on Ubuntu Get first line of a shell command's output Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running? Run bash command on jenkins pipeline How to uninstall an older PHP version from centOS7 How to update-alternatives to Python 3 without breaking apt?

Examples related to bash

Comparing a variable with a string python not working when redirecting from bash script Zipping a file in bash fails How do I prevent Conda from activating the base environment by default? Get first line of a shell command's output Fixing a systemd service 203/EXEC failure (no such file or directory) /bin/sh: apt-get: not found VSCode Change Default Terminal Run bash command on jenkins pipeline How to check if the docker engine and a docker container are running? How to switch Python versions in Terminal?

Examples related to diff

Create patch or diff file from git repository and apply it to another different git repository Comparing the contents of two files in Sublime Text Git diff between current branch and master but not including unmerged master commits Fast way of finding lines in one file that are not in another? Python - difference between two strings How to see the changes in a Git commit? unix diff side-to-side results? Find the files existing in one directory but not in the other git diff between two different files How to get the difference (only additions) between two files in linux