Getting the count of unique values in a column in bash

Question

I have tab delimited files with several columns  I want to count the frequency of occurrence of the different values in a column for all the files in a folder and sort them in decreasing order of count  highest count first   How would I accomplish this in a Linux command line environment   It can use any common command line language like awk  perl  python etc

User · Answer

Here is a way to do it in the shell   FIELD 2 cut -f  FIELD     sort  uniq -c  sort -nr   This is the sort of thing bash is great at

User · Answer

The GNU site suggests this nice awk script  which prints both the words and their frequency    Possible changes    You can pipe through sort -nr  and reverse word and freq word   to see the result in descending order  If you want a specific column  you can omit the for loop and simply write freq 3    - replace 3 with the column number    Here goes      wordfreq awk --- print list of word frequencies           0   tolower  0       remove case distinctions        remove punctuation      gsub      alnum     blank           0       for  i   1  i  lt   NF  i             freq  i         END        for  word in freq           printf   s t d n   word  freq word

User · Answer

Perl  This code computes the occurrences of all columns  and prints a sorted report for each of them     columnvalues pl while   lt  gt          Fields   split   s        for  i   0      Fields              result  i   Fields  i              for  j   0      result         print  column  j  n        values   keys    result  j         sorted   sort    result  j   b   lt   gt   result  j   a        a cmp  b    values      for  k    sorted             print    k  result  j   k  n            Save the text as columnvalues pl Run it as   perl columnvalues pl files   Explanation  In the top-level while loop     Loop over each line of the combined input files    Split the line into the  Fields array    For every column  increment the result array-of-hashes data structure    In the top-level for loop     Loop over the result array    Print the column number    Get the values used in that column    Sort the values by the number of occurrences    Secondary sort based on the value  for example b vs g vs m vs z     Iterate through the result hash  using the sorted list    Print the value and number of each occurrence    Results based on the sample input files provided by  Dennis  column 0   a 3  z 3  t 1  v 1  w 1 column 1   d 3  r 2  b 1  g 1  m 1  z 1 column 2   c 4  a 3  e 2    csv input  If your input files are  csv  change   s   to      Obfuscation  In an ugly contest  Perl is particularly well equipped  This one-liner does the same   perl -lane  for  i  0    F   g  i   F  i      END for  j  0    g  print   j   for  k  sort  g  j   b  lt   gt  g  j   a    a cmp  b  keys    g  j    print    k  g  j   k       files

User · Answer

To see a frequency count for column two  for example    awk -F   t    print  2       sort   uniq -c   sort -nr   fileA txt  z    z    a a    b    c w    d    e   fileB txt  t    r    e z    d    a a    g    c   fileC txt  z    r    a v    d    c a    m    c   Result     3 d   2 r   1 z   1 m   1 g   1 b

User · Answer

Ruby 1 9       usr bin env ruby Dir      each do  file      h Hash new 0      open file  each do  row          row chomp split   t   each do  w              h  w      1         end     end     h sort  a b  b 1  lt   gt a 1    each  x y  print    x    y  n    end

[bash] Getting the count of unique values in a column in bash

Examples related to bash

Examples related to command-line

Examples related to frequency