Ignore duplicates when producing map using streams

Question

Map lt String  String gt  phoneBook   people stream                                          collect toMap Person  getName                                                       Person  getAddress      I get java lang IllegalStateException  Duplicate key when a duplicated element is found    Is it possible to ignore such exception on adding values to the map   When there is duplicate it simply should continue by ignoring that duplicate key

User · Answer

I have encountered such a problem when grouping object, i always resolved them by a simple way: perform a custom filter using a java.util.Set to remove duplicate object with whatever attribute of your choice as bellow

Set<String> uniqueNames = new HashSet<>();
Map<String, String> phoneBook = people
                  .stream()
                  .filter(person -> person != null && !uniqueNames.add(person.getName()))
                  .collect(toMap(Person::getName, Person::getAddress));

Hope this helps anyone having the same problem !

User · Answer

For grouping by Objects  Map lt Integer  Data gt  dataMap   dataList stream   collect Collectors toMap Data  getId  data- gt  data   data1  data2 - gt   LOG info  Duplicate Group For      data2 getId    return data1

User · Answer

Assuming you have people is List of object     Map lt String  String gt  phoneBook people stream                                            collect toMap Person  getName  Person  getAddress      Now you need two steps    1    people  removeDuplicate people     2    Map lt String  String gt  phoneBook people stream                                            collect toMap Person  getName  Person  getAddress      Here is method to remove duplicate   public static List removeDuplicate Collection lt Person gt   list            if list   null    list isEmpty                 return null                     Object removedDuplicateList                   list stream                         distinct                         collect Collectors toList          return  List  removedDuplicateList             Adding full example here   package com example khan vaquar   import java util Arrays  import java util Collection  import java util List  import java util Map  import java util stream Collectors   public class RemovedDuplicate        public static void main String   args            Person vaquar   new Person 1   Vaquar    Khan            Person zidan   new Person 2   Zidan    Khan            Person zerina   new Person 3   Zerina    Khan                Add some random persons         Collection lt Person gt  duplicateList   Arrays asList vaquar  zidan  zerina  vaquar  zidan  vaquar                       System out println  Before removed duplicate list    duplicateList                      Collection lt Person gt  nonDuplicateList   removeDuplicate duplicateList                      System out println              System out println  After removed duplicate list    nonDuplicateList                         1  solution Working code         Map lt Object  Object gt  k   nonDuplicateList stream   distinct                    collect Collectors toMap s1 - gt  s1 getId    s1 - gt  s1            System out println              System out println  Result 1 using method                                                           System out println  k    k           System out println                                                                                       2  solution using inline distinct           Map lt Object  Object gt  k1   duplicateList stream   distinct                    collect Collectors toMap s1 - gt  s1 getId    s1 - gt  s1            System out println              System out println  Result 2 using inline                                                           System out println  k1    k1           System out println                                                                                      breacking code         System out println              System out println  Throwing exception                                                            Map lt Object  Object gt  k2   duplicateList stream                    collect Collectors toMap s1 - gt  s1 getId    s1 - gt  s1            System out println              System out println  k2    k2           System out println                                                                                      public static List removeDuplicate Collection lt Person gt  list            if  list    null    list isEmpty                  return null                     Object removedDuplicateList   list stream   distinct   collect Collectors toList             return  List  removedDuplicateList                Model class class Person       public Person Integer id  String fname  String lname            super            this id   id          this fname   fname          this lname   lname             private Integer id      private String fname      private String lname          Getters and Setters      public Integer getId             return id             public void setId Integer id            this id   id             public String getFname             return fname             public void setFname String fname            this fname   fname             public String getLname             return lname             public void setLname String lname            this lname   lname              Override     public String toString             return  Person  id     id      fname     fname      lname     lname                   Results    Before removed duplicate list Person  id 1  fname Vaquar  lname Khan   Person  id 2  fname Zidan  lname Khan   Person  id 3  fname Zerina  lname Khan   Person  id 1  fname Vaquar  lname Khan   Person  id 2  fname Zidan  lname Khan   Person  id 1  fname Vaquar  lname Khan    After removed duplicate list Person  id 1  fname Vaquar  lname Khan   Person  id 2  fname Zidan  lname Khan   Person  id 3  fname Zerina  lname Khan    Result 1 using method                                                k 1 Person  id 1  fname Vaquar  lname Khan   2 Person  id 2  fname Zidan  lname Khan   3 Person  id 3  fname Zerina  lname Khan                                                                          Result 2 using inline                                                k1 1 Person  id 1  fname Vaquar  lname Khan   2 Person  id 2  fname Zidan  lname Khan   3 Person  id 3  fname Zerina  lname Khan                                                                          Throwing exception                                                 Exception in thread  main  java lang IllegalStateException  Duplicate key Person  id 1  fname Vaquar  lname Khan      at java util stream Collectors lambda throwingMerger 0 Collectors java 133      at java util HashMap merge HashMap java 1253      at java util stream Collectors lambda toMap 58 Collectors java 1320      at java util stream ReduceOps 3ReducingSink accept ReduceOps java 169      at java util Spliterators ArraySpliterator forEachRemaining Spliterators java 948      at java util stream AbstractPipeline copyInto AbstractPipeline java 481      at java util stream AbstractPipeline wrapAndCopyInto AbstractPipeline java 471      at java util stream ReduceOps ReduceOp evaluateSequential ReduceOps java 708      at java util stream AbstractPipeline evaluate AbstractPipeline java 234      at java util stream ReferencePipeline collect ReferencePipeline java 499      at com example khan vaquar RemovedDuplicate main RemovedDuplicate java 48

User · Answer

The  alaster answer help me a lot  but I would like to add a meaninful information if someone is trying to group the information   If you have  per example  two Orders with the same code but different quantity of products for each one  and your desire is sum the quantities  you can do   List lt Order gt  listQuantidade   new ArrayList lt  gt     listOrders add new Order  COD 1   1L    listOrders add new Order  COD 1   5L    listOrders add new Order  COD 1   3L    listOrders add new Order  COD 2   3L    listOrders add new Order  COD 3   4L     listOrders collect Collectors toMap Order  getCode                                       o - gt  o getQuantity                                          o1  o2  - gt  o1   o2      Result    COD 3 4  COD 2 3  COD 1 9

User · Answer

For anyone else getting this issue but without duplicate keys in the map being streamed  make sure your keyMapper function isn t returning null values   It s very annoying to track this down because the error will say  Duplicate key 1  when 1 is actually the value of the entry instead of the key   In my case  my keyMapper function tried to look up values in a different map  but due to a typo in the strings was returning null values   final Map lt String  String gt  doop   new HashMap lt  gt     doop put  a    1    doop put  b    2     final Map lt String  String gt  lookup   new HashMap lt  gt     doop put  c    e    doop put  d    f     doop entrySet   stream   collect Collectors toMap e - gt  lookup get e getKey     e - gt  e getValue

User · Answer

As said in JavaDocs      If the mapped keys contains duplicates  according to   Object equals Object    an IllegalStateException is thrown when the   collection operation is performed  If the mapped keys may have   duplicates  use toMap Function keyMapper  Function valueMapper  BinaryOperator mergeFunction  instead    So you should use toMap Function keyMapper  Function valueMapper  BinaryOperator mergeFunction  instead  Just provide a merge function  that will determine which one of duplicates is put in the map    For example  if you don t care which one  just call  Map lt String  String gt  phoneBook            people stream                  collect Collectors toMap Person  getName                                           Person  getAddress                                            a1  a2  - gt  a1

User · Answer

This is possible using the mergeFunction parameter of Collectors toMap keyMapper  valueMapper  mergeFunction    Map lt String  String gt  phoneBook        people stream              collect Collectors toMap               Person  getName               Person  getAddress                address1  address2  - gt                     System out println  duplicate key found                      return address1                                 mergeFunction is a function that operates on two values associated with the same key  adress1 corresponds to the first address that was encountered when collecting elements and adress2 corresponds to the second address encountered  this lambda just tells to keep the first address and ignores the second

[java] Ignore duplicates when producing map using streams

Examples related to java

Examples related to java-8

Examples related to java-stream