How do I remove duplicate items from an array in Perl

Question

I have an array in Perl   my  my array     one   two   three   two   three      How do I remove the duplicates from the array

User · Answer

The Perl documentation comes with a nice collection of FAQs. Your question is frequently asked:

% perldoc -q duplicate

The answer, copy and pasted from the output of the command above, appears below:

Found in /usr/local/lib/perl5/5.10.0/pods/perlfaq4.pod
 How can I remove duplicate elements from a list or array?
   (contributed by brian d foy)

   Use a hash. When you think the words "unique" or "duplicated", think
   "hash keys".

   If you don't care about the order of the elements, you could just
   create the hash then extract the keys. It's not important how you
   create that hash: just that you use "keys" to get the unique elements.

       my %hash   = map { $_, 1 } @array;
       # or a hash slice: @hash{ @array } = ();
       # or a foreach: $hash{$_} = 1 foreach ( @array );

       my @unique = keys %hash;

   If you want to use a module, try the "uniq" function from
   "List::MoreUtils". In list context it returns the unique elements,
   preserving their order in the list. In scalar context, it returns the
   number of unique elements.

       use List::MoreUtils qw(uniq);

       my @unique = uniq( 1, 2, 3, 4, 4, 5, 6, 5, 7 ); # 1,2,3,4,5,6,7
       my $unique = uniq( 1, 2, 3, 4, 4, 5, 6, 5, 7 ); # 7

   You can also go through each element and skip the ones you've seen
   before. Use a hash to keep track. The first time the loop sees an
   element, that element has no key in %Seen. The "next" statement creates
   the key and immediately uses its value, which is "undef", so the loop
   continues to the "push" and increments the value for that key. The next
   time the loop sees that same element, its key exists in the hash and
   the value for that key is true (since it's not 0 or "undef"), so the
   next skips that iteration and the loop goes to the next element.

       my @unique = ();
       my %seen   = ();

       foreach my $elem ( @array )
       {
         next if $seen{ $elem }++;
         push @unique, $elem;
       }

   You can write this more briefly using a grep, which does the same
   thing.

       my %seen = ();
       my @unique = grep { ! $seen{ $_ }++ } @array;

User · Answer

Using concept of unique hash keys    my  array      a   b   c   b   a   d   c   a   d    my  hash     map        gt  1    array  my  unique   keys  hash  print   unique    n     Output  a c b d

User · Answer

Install List  MoreUtils from CPAN  Then in your code   use strict  use warnings  use List  MoreUtils qw uniq    my  dup list   qw 1 1 1 2 3 4 4    my  uniq list   uniq  dup list

User · Answer

The variable  array is the list with duplicate elements     seen      unique   grep      seen           array

User · Answer

Method 1  Use a hash  Logic  A hash can have only unique keys  so iterate over array  assign any value to each element of array  keeping element as key of that hash  Return keys of the hash  its your unique array   my  unique   keys  map       gt  1   array     Method 2  Extension of method 1 for reusability  Better to make a subroutine if we are supposed to use this functionality multiple times in our code   sub get unique       my  seen      grep   seen              my  unique   get unique  array     Method 3  Use module List  MoreUtils  use List  MoreUtils qw uniq   my  unique   uniq  array

User · Answer

You can do something like this as demonstrated in perlfaq4   sub uniq       my  seen      grep   seen               my  array   qw one two three two three   my  filtered   uniq  array    print   filtered n     Outputs   one two three   If you want to use a module  try the uniq function from List  MoreUtils

User · Answer

My usual way of doing this is   my  unique       foreach my  item   myarray         unique  item        my  myuniquearray   keys  unique    If you use a hash and add the items to the hash  You also have the bonus of knowing how many times each item appears in the list

User · Answer

Can be done with a simple Perl one liner   my  in qw 1 3 4  6 2 4  3 2 6  3 2 3 4 4 3 2 5 5 32 3    Sample data  my  out keys     map     gt 1  in      Perform PFM print join      sort  a lt   gt  b   out   Print data back out sorted and in order    The PFM block does this   Data in  in is fed into MAP   MAP builds an anonymous hash   Keys are extracted from the hash and feed into  out

User · Answer

Try this  seems the uniq function needs a sorted list to work properly   use strict     Helper function to remove duplicates in a list  sub uniq     my  seen    grep   seen               my  teststrings     one    two    three    one     my  filtered   uniq  teststrings  print  uniq   filtered n   my  sorted   sort  teststrings  print  sort   sorted n   my  sortedfiltered   uniq sort  teststrings  print  uniq sort    sortedfiltered n

User · Answer

That last one was pretty good  I d just tweak it a bit   my  arr  my  uniqarr   foreach my  var    arr      if     grep    var    uniqarr           push   uniqarr   var            I think this is probably the most readable way to do it

User · Answer

Previous answers pretty much summarize the possible ways of accomplishing this task   However  I suggest a modification for those who don t care about counting the duplicates  but do care about order   my  record   qw  yeah I mean uh right right uh yeah so well right I maybe    my  record  print grep   record      amp  amp     record       record    Note that the previously suggested grep   seen           increments  seen     before negating  so the increment occurs regardless of whether it has already been  seen or not  The above  however  short-circuits when  record     is true  leaving what s been heard once  off the  record    You could also go for this ridiculousness  which takes advantage of autovivification and existence of hash keys       grep   exists  record        undef  record        record    That  however  might lead to some confusion   And if you care about neither order or duplicate count  you could for another hack using hash slices and the trick I just mentioned       undef  record  record   keys  record    your record  now probably scrambled but at least deduped

[perl] How do I remove duplicate items from an array in Perl?

Examples related to perl

Examples related to arrays

Examples related to unique

Examples related to duplicates