[linux] Find unique lines

How can I find the unique lines and remove all duplicates from a file? My input file is

1
1
2
3
5
5
7
7

I would like the result to be:

2
3

sort file | uniq will not do the job. Will show all values 1 time

This question is related to linux sorting unique uniq

The answer is


This was the first i tried

skilla:~# uniq -u all.sorted  

76679787
76679787 
76794979
76794979 
76869286
76869286 
......

After doing a cat -e all.sorted

skilla:~# cat -e all.sorted 
$
76679787$
76679787 $
76701427$
76701427$
76794979$
76794979 $
76869286$
76869286 $

Every second line has a trailing space :( After removing all trailing spaces it worked!

thank you


uniq should do fine if you're file is/can be sorted, if you can't sort the file for some reason you can use awk:

awk '{a[$0]++}END{for(i in a)if(a[i]<2)print i}'


While sort takes O(n log(n)) time, I prefer using

awk '!seen[$0]++'

awk '!seen[$0]++' is an abbreviation for awk '!seen[$0]++ {print}', print line(=$0) if seen[$0] is not zero. It take more space but only O(n) time.


uniq -u has been driving me crazy because it did not work.

So instead of that, if you have python (most Linux distros and servers already have it):

Assuming you have the data file in notUnique.txt

#Python
#Assuming file has data on different lines
#Otherwise fix split() accordingly.

uniqueData = []
fileData = open('notUnique.txt').read().split('\n')

for i in fileData:
  if i.strip()!='':
    uniqueData.append(i)

print uniqueData

###Another option (less keystrokes):
set(open('notUnique.txt').read().split('\n'))

Note that due to empty lines, the final set may contain '' or only-space strings. You can remove that later. Or just get away with copying from the terminal ;)

#

Just FYI, From the uniq Man page:

"Note: 'uniq' does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use 'sort -u' without 'uniq'. Also, comparisons honor the rules specified by 'LC_COLLATE'."

One of the correct ways, to invoke with: # sort nonUnique.txt | uniq

Example run:

$ cat x
3
1
2
2
2
3
1
3

$ uniq x
3
1
2
3
1
3

$ uniq -u x
3
1
3
1
3

$ sort x | uniq
1
2
3

Spaces might be printed, so be prepared!


uniq has the option you need:

   -u, --unique
          only print unique lines
$ cat file.txt
1
1
2
3
5
5
7
7
$ uniq -u file.txt
2
3

you can use:

sort data.txt| uniq -u

this sort data and filter by unique values


You could also print out the unique value in "file" using the cat command by piping to sort and uniq

cat file | sort | uniq -u


Use as follows:

sort < filea | uniq > fileb

sort -d "file name" | uniq -u

this worked for me for a similar one. Use this if it is not arranged. You can remove sort if it is arranged


I find this easier.

sort -u input_filename > output_filename

-u stands for unique.


uniq -u < file will do the job.


Examples related to linux

grep's at sign caught as whitespace How to prevent Google Colab from disconnecting? "E: Unable to locate package python-pip" on Ubuntu 18.04 How to upgrade Python version to 3.7? Install Qt on Ubuntu Get first line of a shell command's output Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running? Run bash command on jenkins pipeline How to uninstall an older PHP version from centOS7 How to update-alternatives to Python 3 without breaking apt?

Examples related to sorting

Sort Array of object by object field in Angular 6 Sorting a list with stream.sorted() in Java How to sort dates from Oldest to Newest in Excel? how to sort pandas dataframe from one column Reverse a comparator in Java 8 Find the unique values in a column and then sort them pandas groupby sort within groups pandas groupby sort descending order Efficiently sorting a numpy array in descending order? Swift: Sort array of objects alphabetically

Examples related to unique

Count unique values with pandas per groups Find the unique values in a column and then sort them How can I check if the array of objects have duplicate property values? Firebase: how to generate a unique numeric ID for key? pandas unique values multiple columns Select unique values with 'select' function in 'dplyr' library Generate 'n' unique random numbers within a range SQL - select distinct only on one column Can I use VARCHAR as the PRIMARY KEY? Count unique values in a column in Excel

Examples related to uniq

Find unique lines Sort & uniq in Linux shell Is there a way to 'uniq' by column?