Identify duplicates in a List

Question

I have a List of type Integer eg    1  1  2  3  3  3    I would like a method to return all the duplicates eg    1  3    What is the best way to do this

User · Answer

Lambas might be a solution  Integer   nums    new Integer    1  1  2  3  3  3   List lt Integer gt  list   Arrays asList nums    List lt Integer gt  dps   list stream   distinct   filter entry - gt  Collections frequency list  entry   gt  1  collect Collectors toList

User · Answer

This is a problem where functional techniques shine  For example  the following F  solution is both clearer and less bug prone than the best imperative Java solution  and I work daily with both Java and F      1 1 2 3 3 3     gt  Seq countBy id    gt  Seq choose  fun  key count  - gt  if count  gt  1 then Some key  else None    Of course  this question is about Java  So my suggestion is to adopt a library which brings functional features to Java  For example  it could be solved using my own library as follows  and there are several others out there worth looking at too    Seq of 1 1 2 3 3 3   groupBy new Func1 lt Integer Integer gt          public Integer call Integer key            return key           filter new Predicate lt Grouping lt Integer Integer gt  gt         public Boolean call Grouping lt Integer  Integer gt  grouping            return grouping getGrouping   count    gt  1          map new Func1 lt Grouping lt Integer Integer gt  Integer gt          public Integer call Grouping lt Integer  Integer gt  grouping            return grouping getKey

User · Answer

A thread-safe alternative is this          Returns all duplicates that are in the list as a new   link Set  thread-safe      lt p gt     Usually the Set will contain only the last duplicate  however the decision    what elements are equal depends on the implementation of the   link List   An    exotic implementation of   link List  might decide two elements are  equal      in this case multiple duplicates might be returned          param  lt X gt   The type of element to compare      param list The list that contains the elements  never  lt code gt null lt  code gt       return A set of all duplicates in the list  Returns only the last duplicate      public  lt X extends Object gt  Set lt X gt  findDuplicates List lt X gt  list        Set lt X gt  dups   new LinkedHashSet lt  gt  list size         synchronized  list            for  X x   list                if  list indexOf x     list lastIndexOf x                     dups add x                                     return dups

User · Answer

If you use Eclipse Collections  this will work   MutableList lt Integer gt  list   Lists mutable with 1  1  2  3  3  3   Set lt Integer gt  dupes   list toBag   selectByOccurrences i - gt  i  gt  1  toSet    Assert assertEquals Sets mutable with 1  3   dupes     Update  As of Eclipse Collections 9 2 you can now use selectDuplicates  MutableList lt Integer gt  list   Lists mutable with 1  1  2  3  3  3   Set lt Integer gt  dupes   list toBag   selectDuplicates   toSet    Assert assertEquals Sets mutable with 1  3   dupes     You can also use primitive collections to accomplish this   IntList list   IntLists mutable with 1  1  2  3  3  3   IntSet dupes   list toBag   selectDuplicates   toSet    Assert assertEquals IntSets mutable with 1  3   dupes     Note  I am a committer for Eclipse Collections

User · Answer

The method add of Set returns a boolean whether a value already exists  true if it does not exist  false if it already exists  see Set documentation    So just iterate through all the values   public Set lt Integer gt  findDuplicates List lt Integer gt  listContainingDuplicates       final Set lt Integer gt  setToReturn   new HashSet lt  gt        final Set lt Integer gt  set1   new HashSet lt  gt        for  Integer yourInt   listContainingDuplicates         if   set1 add yourInt            setToReturn add yourInt              return setToReturn

User · Answer

Just try this     Example if List values are   1  2  3  4  5  6  4  3  7  8  duplicate item  3  4    Collections sort list           List lt Integer gt  dup   new ArrayList lt  gt             for  int i   0  i  lt  list size   - 1  i                  if  list get i     list get i   1                     if   dup contains list get i   1                          dup add list get i   1                                                      System out println  duplicate item     dup

User · Answer

This also works   public static Set lt Integer gt  findDuplicates List lt Integer gt  input        List lt Integer gt  copy   new ArrayList lt Integer gt  input       for  Integer value   new HashSet lt Integer gt  input             copy remove value             return new HashSet lt Integer gt  copy

User · Answer

How about this code -   public static void main String   args           Lets say we have a elements in array     int   a    13 65 13 67 88 65 88 23 65 88 92        List lt Integer gt  ls1   new ArrayList lt  gt         List lt Integer gt  ls2   new ArrayList lt  gt         Set lt Integer gt  ls3   new TreeSet lt  gt            Adding each element of the array in the list           for int i 0 i lt a length i                 ls1 add a i                       Iterating each element in the arrary     for  Integer eachInt   ls1           If the list2 contains the iterating element  then add that into set lt  gt   as this would be a duplicate element          if ls2 contains eachInt                 ls3 add eachInt                     else  ls2 add eachInt                System out println  Elements in array or ls1  ls1        System out println  Duplicate Elements in Set ls3  ls3

User · Answer

import java util Scanner   public class OnlyDuplicates       public static void main String   args            System out print   Enter a set of 10 numbers              int   numbers   new int 10           Scanner input   new Scanner System in           for  int i   0  i  lt  numbers length  i                  numbers i    input nextInt                      numbers   onlyDuplicates numbers           System out print   The numbers are              for  int i   0  i  lt  numbers length  i                  System out print numbers i                              public static int   onlyDuplicates int   list            boolean flag   true          int   array   new int 0           array   add2Array array  list 0            for  int i   0  i  lt  list length  i                  for  int j   0  j  lt  array length  j                      if  list i     array j                         flag   false                      break                                              if  flag                    array   add2Array array  list i                              flag   true                    return array               Copy numbers1 to numbers2        If the length of numbers2 is less then numbers2  return false     public static boolean copyArray int   source  int   dest            if  source length  gt  dest length                return false                     for  int i   0  i  lt  source length  i                  dest i    source i                     return true               Increase array size by one and add integer to the end of the array     public static int   add2Array int   source  int data            int   dest   new int source length   1           copyArray source  dest           dest source length    data          return dest

User · Answer

create a Map lt Integer Integer gt   iterate the list  if an element is in the map  increase it s value  otherwise add it to the map with key 1 iterate the map  and add to the lists all elements with key  2  public static void main String   args            List lt Integer gt  list   new LinkedList lt Integer gt             list add 1           list add 1           list add 1           list add 2           list add 3           list add 3           Map lt Integer Integer gt  map   new HashMap lt Integer  Integer gt             for  Integer x   list                 Integer val   map get x               if  val    null                     map put x 1                 else                   map remove x                   map put x val 1                                   List lt Integer gt  result   new LinkedList lt Integer gt             for  Entry lt Integer  Integer gt  entry   map entrySet                  if  entry getValue    gt  1                    result add entry getKey                                     for  Integer x   result                 System out println x

User · Answer

java 8 base solution   List duplicates       list stream   collect Collectors groupingBy Function identity          entrySet        stream        filter e - gt  e getValue   size    gt  1       map Map Entry  getKey       collect Collectors toList

User · Answer

And version which uses commons-collections CollectionUtils getCardinalityMap method   final List lt Integer gt  values   Arrays asList 1  1  2  3  3  3   final Map lt Integer  Integer gt  cardinalityMap   CollectionUtils getCardinalityMap values   System out println cardinalityMap              entrySet                stream   filter e - gt  e getValue    gt  1               map e - gt  e getKey                 collect Collectors toList

User · Answer

Try this to find duplicates items in list    ArrayList lt String gt  arrayList1   new ArrayList lt String gt       arrayList1 add  A     arrayList1 add  A     arrayList1 add  B     arrayList1 add  B     arrayList1 add  B     arrayList1 add  C      for  int x 0  x lt  arrayList1 size    x        System out println  arrayList1    arrayList1 get x        Set s new TreeSet     s addAll arrayList1    Iterator it s iterator     while  it hasNext        System out println  Set     String it next

User · Answer

This should work for sorted and unsorted   public void testFindDuplicates          List lt Integer gt  list   new ArrayList lt Integer gt         list add 1       list add 1       list add 2       list add 3       list add 3       list add 3        Set lt Integer gt  result   new HashSet lt Integer gt         int currentIndex   0      for  Integer i   list            if   result contains i   amp  amp  list subList currentIndex   1  list size    contains i                 result add i                     currentIndex              assertEquals 2  result size         assertTrue result contains 1        assertTrue result contains 3

User · Answer

You can use something like this   List lt Integer gt  newList   new ArrayList lt Integer gt     for int i   yourOldList        yourOldList remove i       if yourOldList contains i   amp  amp   newList contains i   newList add i

User · Answer

just in case for those that also want to include both the duplicate and the non duplicates  basically the answer similiar to the correct answer but instead of returning from if not part you return the else part  use this code  change  to the type that you need   public Set lt String gt  findDup List lt String gt  Duplicates       Set lt String gt  returning   new HashSet lt  gt         Set lt String gt  nonreturning   new HashSet lt  gt         Set lt String gt  setup   new HashSet lt  gt         for String i Duplicates           if  setup add  i                 returning add  i             else              nonreturning add  i                        Toast makeText  context  hello set  returning nonreturning   size  nonreturning size   Toast LENGTH SHORT   show        return nonreturning

User · Answer

Similar to some answers here  but if you want to find duplicates based on some property     public static  lt T  R gt  Set lt R gt  findDuplicates Collection lt   extends T gt  collection  Function lt   super T    extends R gt  mapper        Set lt R gt  uniques   new HashSet lt  gt         return collection stream            map mapper           filter e - gt   uniques add e            collect toSet

User · Answer

Use a MultiMap to store each value as a key   value set  Then iterate through the keys and find the ones with multiple values

User · Answer

More generic method as variant of https   stackoverflow com a 52296246                 Returns a duplicated values found in given collection based on fieldClassifier                param collection given collection of elements         param fieldClassifier field classifier which specifies element to check for duplicates useful in complex objects           param  lt T gt  Type of element in collection         param  lt K gt  Element which will be returned from method in fieldClassifier          return returns list of values that are duplocated              public static  lt T  K gt  List lt K gt  lookForDuplicates List lt T gt  collection  Function lt   super T    extends K gt  fieldClassifier             return collection stream   collect Collectors groupingBy fieldClassifier                             entrySet                             stream                             filter e - gt  e getValue   size    gt  1                            map Map Entry  getKey                            collect Collectors toList

User · Answer

I needed a solution to this as well   I used leifg s solution and made it generic   private  lt T gt  Set lt T gt  findDuplicates Collection lt T gt  collection         Set lt T gt  duplicates   new LinkedHashSet lt  gt         Set lt T gt  uniques   new HashSet lt  gt          for T t   collection            if  uniques add t                 duplicates add t                        return duplicates

User · Answer

If you know the maximum value  for example  lt  10000  you could sacrifice space for speed   I Can   t remember exact name of this technique   pseudo code     does not handle case when mem allocation fails    probably can be extended to unknown values  larger values   maybe by sorting first public List lt int gt  GetDuplicates int max             allocate and clear memory to 0 false     bit   buckets new bit max      memcpy buckets 0 max         find duplicates     List lt int gt  result new List lt int gt         foreach int val in List                if  buckets val                         result add value                     else                       buckets val  1                      return  result

User · Answer

This would be a good method to find Duplicate values  without using Set   public static  lt T gt  List lt T gt  findDuplicates List lt T gt  list    List lt T gt  nonDistinctElements   new ArrayList lt  gt        for T s   list      if list indexOf s     list lastIndexOf s         if  nonDistinctElements contains s           nonDistinctElements add s      return nonDistinctElements      And say  that you want a method that returns you a distinct list  i e  if you pass a list where elements are occurring more than once  you ll get a list with distinct elements   public static  lt T gt  void distinctList List lt T gt  list    List lt T gt  nonDistinctElements   new ArrayList lt  gt     for T s   list    if list indexOf s     list lastIndexOf s       nonDistinctElements add s    for T nonDistinctElement   nonDistinctElements    if list indexOf nonDistinctElement     list lastIndexOf nonDistinctElement       list remove nonDistinctElement

User · Answer

Here is a solution using Streams with Java 8     lets assume the original list is filled with  1 1 2 3 6 3 8 7  List lt String gt  original   new ArrayList lt  gt     List lt String gt  result   new ArrayList lt  gt       You just look if the frequency of this object is more than once in your list  Then call  distinct   to only have unique elements in your result  result   original stream        filter e - gt  Collections frequency original  e   gt  1       distinct        collect Collectors toList        returns  1 3     returns only numbers which occur more than once  result   original stream        filter e - gt  Collections frequency original  e     1       collect Collectors toList        returns  2 6 8 7     returns numbers which occur only once  result   original stream        distinct        collect Collectors toList        returns  1 2 3 6 8 7     returns the list without duplicates

User · Answer

Put list in set  this effectively filter only unique items   remove all set items from original list  so it will contains only items  which have more then 1 occurence   and put list in new set  this will again filter out only unique items    List lt Item gt  list        list removeAll new HashSet lt Item gt  list    return new HashSet lt Item gt  list

User · Answer

Using Guava on Java 8  private Set lt Integer gt  findDuplicates List lt Integer gt  input           Linked  preserves insertion order so the returned Sets iteration order is somewhat like the original list     LinkedHashMultiset lt Integer gt  duplicates   LinkedHashMultiset create input           Remove all entries with a count of 1     duplicates entrySet   removeIf entry - gt  entry getCount      1        return duplicates elementSet

User · Answer

public class practicese          public static void main String   args                   List lt Integer gt  listOf   new ArrayList lt Integer gt                listOf add 3              listOf add 1              listOf add 2              listOf add 3              listOf add 3              listOf add 2              listOf add 1               List lt Integer gt  tempList   new ArrayList lt Integer gt                for Integer obj listOf                   if  tempList contains obj                        tempList add obj                                                System out println tempList

User · Answer

int   nums    new int    1  1  2  3  3  3   Arrays sort nums   for  int i   0  i  lt  nums length-1  i           if  nums i     nums i 1             System out println  duplicate item   nums i 1    at Location   i 1                Obviously you can do whatever you want with them  i e  put in a Set to get a unique list of duplicate values  instead of printing    This also has the benefit of recording the location of duplicate items too

User · Answer

I took John Strickler s solution and remade it to use the streams API introduced in JDK8   private  lt T gt  Set lt T gt  findDuplicates Collection lt T gt  collection        Set lt T gt  uniques   new HashSet lt  gt         return collection stream            filter e - gt   uniques add e            collect Collectors toSet

User · Answer

Compact generified version of the top answer  also added empty check and preallocated Set size   public static final  lt T gt  Set lt T gt  findDuplicates final List lt T gt  listWhichMayHaveDuplicates        final Set lt T gt  duplicates   new HashSet lt  gt         final int listSize   listWhichMayHaveDuplicates size        if  listSize  gt  0          final Set lt T gt  tempSet   new HashSet lt  gt  listSize         for  final T element   listWhichMayHaveDuplicates            if   tempSet add element               duplicates add element                               return duplicates

User · Answer

I took Sebastian s answer and added a keyExtractor to it -      private  lt U  T gt  Set lt T gt  findDuplicates Collection lt T gt  collection  Function lt   super T   extends U gt  keyExtractor            Map lt U  T gt  uniques   new HashMap lt  gt        maps unique keys to corresponding values         return collection stream                filter e - gt  uniques put keyExtractor apply e   e     null               collect Collectors toSet

User · Answer

public class DuplicatesWithOutCollection        public static void main String   args             int   arr   new int     2  3  4  6  6  8  10  10  10  11  12  12             boolean flag   false          int k   1          while  k    1                 arr   removeDuplicate arr               flag   checkDuplicate arr  flag               if  flag                    k   1                else                   k   0                                       private static boolean checkDuplicate int   arr  boolean flag            int i   0           while  i  lt  arr length - 1                 if  arr i     arr i   1                      flag   true                 else                   flag   false                            i                        return flag             private static int   removeDuplicate int   arr             int i   0  j   0          int   temp   new int arr length           while  i  lt  arr length - 1                 if  arr i     arr i   1                      temp j    arr i   1                   i   i   2                 else                    temp j    arr i                   i   i   1                   if  i    arr length - 1                        temp j   1    arr i   1                       break                                               j                       System out println            return temp

[java] Identify duplicates in a List

Examples related to java

Examples related to collections