How do I remove duplicate items from an array in Perl

Question

I have an array in Perl   my  my array     one   two   three   two   three      How do I remove the duplicates from the array

User · Answer

Can be done with a simple Perl one liner.

my @in=qw(1 3 4  6 2 4  3 2 6  3 2 3 4 4 3 2 5 5 32 3); #Sample data 
my @out=keys %{{ map{$_=>1}@in}}; # Perform PFM
print join ' ', sort{$a<=>$b} @out;# Print data back out sorted and in order.

The PFM block does this:

Data in @in is fed into MAP. MAP builds an anonymous hash. Keys are extracted from the hash and feed into @out

User · Answer

Previous answers pretty much summarize the possible ways of accomplishing this task   However  I suggest a modification for those who don t care about counting the duplicates  but do care about order   my  record   qw  yeah I mean uh right right uh yeah so well right I maybe    my  record  print grep   record      amp  amp     record       record    Note that the previously suggested grep   seen           increments  seen     before negating  so the increment occurs regardless of whether it has already been  seen or not  The above  however  short-circuits when  record     is true  leaving what s been heard once  off the  record    You could also go for this ridiculousness  which takes advantage of autovivification and existence of hash keys       grep   exists  record        undef  record        record    That  however  might lead to some confusion   And if you care about neither order or duplicate count  you could for another hack using hash slices and the trick I just mentioned       undef  record  record   keys  record    your record  now probably scrambled but at least deduped

User · Answer

Method 1  Use a hash  Logic  A hash can have only unique keys  so iterate over array  assign any value to each element of array  keeping element as key of that hash  Return keys of the hash  its your unique array   my  unique   keys  map       gt  1   array     Method 2  Extension of method 1 for reusability  Better to make a subroutine if we are supposed to use this functionality multiple times in our code   sub get unique       my  seen      grep   seen              my  unique   get unique  array     Method 3  Use module List  MoreUtils  use List  MoreUtils qw uniq   my  unique   uniq  array

User · Answer

My usual way of doing this is   my  unique       foreach my  item   myarray         unique  item        my  myuniquearray   keys  unique    If you use a hash and add the items to the hash  You also have the bonus of knowing how many times each item appears in the list

User · Answer

That last one was pretty good  I d just tweak it a bit   my  arr  my  uniqarr   foreach my  var    arr      if     grep    var    uniqarr           push   uniqarr   var            I think this is probably the most readable way to do it

User · Answer

Using concept of unique hash keys    my  array      a   b   c   b   a   d   c   a   d    my  hash     map        gt  1    array  my  unique   keys  hash  print   unique    n     Output  a c b d

User · Answer

You can do something like this as demonstrated in perlfaq4   sub uniq       my  seen      grep   seen               my  array   qw one two three two three   my  filtered   uniq  array    print   filtered n     Outputs   one two three   If you want to use a module  try the uniq function from List  MoreUtils

User · Answer

Try this  seems the uniq function needs a sorted list to work properly   use strict     Helper function to remove duplicates in a list  sub uniq     my  seen    grep   seen               my  teststrings     one    two    three    one     my  filtered   uniq  teststrings  print  uniq   filtered n   my  sorted   sort  teststrings  print  sort   sorted n   my  sortedfiltered   uniq sort  teststrings  print  uniq sort    sortedfiltered n

User · Answer

The variable  array is the list with duplicate elements     seen      unique   grep      seen           array

User · Answer

Install List  MoreUtils from CPAN  Then in your code   use strict  use warnings  use List  MoreUtils qw uniq    my  dup list   qw 1 1 1 2 3 4 4    my  uniq list   uniq  dup list

User · Answer

The Perl documentation comes with a nice collection of FAQs   Your question is frequently asked     perldoc -q duplicate   The answer  copy and pasted from the output of the command above  appears below   Found in  usr local lib perl5 5 10 0 pods perlfaq4 pod  How can I remove duplicate elements from a list or array      contributed by brian d foy      Use a hash  When you think the words  unique  or  duplicated   think     hash keys       If you don t care about the order of the elements  you could just    create the hash then extract the keys  It s not important how you    create that hash  just that you use  keys  to get the unique elements          my  hash     map       1    array           or a hash slice   hash   array                  or a foreach   hash       1 foreach    array            my  unique   keys  hash      If you want to use a module  try the  uniq  function from     List  MoreUtils   In list context it returns the unique elements     preserving their order in the list  In scalar context  it returns the    number of unique elements          use List  MoreUtils qw uniq           my  unique   uniq  1  2  3  4  4  5  6  5  7      1 2 3 4 5 6 7        my  unique   uniq  1  2  3  4  4  5  6  5  7      7     You can also go through each element and skip the ones you ve seen    before  Use a hash to keep track  The first time the loop sees an    element  that element has no key in  Seen  The  next  statement creates    the key and immediately uses its value  which is  undef   so the loop    continues to the  push  and increments the value for that key  The next    time the loop sees that same element  its key exists in the hash and    the value for that key is true  since it s not 0 or  undef    so the    next skips that iteration and the loop goes to the next element          my  unique              my  seen                 foreach my  elem    array                     next if  seen   elem               push  unique   elem               You can write this more briefly using a grep  which does the same    thing          my  seen              my  unique   grep      seen            array

[perl] How do I remove duplicate items from an array in Perl?

Examples related to perl

Examples related to arrays

Examples related to unique

Examples related to duplicates