Java 8 Distinct by property

Question

In Java 8 how can I filter a collection using the Stream API by checking the distinctness of a property of each object   For example I have a list of Person object and I want to remove people with the same name   persons stream   distinct      Will use the default equality check for a Person object  so I need something like   persons stream   distinct p - gt  p getName       Unfortunately the distinct   method has no such overload  Without modifying the equality check inside the Person class is it possible to do this succinctly

User · Answer

I recommend using Vavr  if you can  With this library you can do the following   io vavr collection List ofAll persons                          distinctBy Person  getName                          toJavaSet      or any another Java 8 Collection

User · Answer

A variation of the top answer that handles null      public static  lt T  K gt  Predicate lt T gt  distinctBy final Function lt   super T  K gt  getKey            val seen   ConcurrentHashMap  lt Optional lt K gt  gt newKeySet            return obj - gt  seen add Optional ofNullable getKey apply obj            In my tests          assertEquals                  asList  quot a quot    quot bb quot                    Stream of  quot a quot    quot b quot    quot bb quot    quot aa quot   filter distinctBy String  length   collect toList               assertEquals                  asList 5  null  2  3                   Stream of 5  null  2  null  3  3  2  filter distinctBy x - gt  x   collect toList               val maps   asList                  hashMapWith 0  2                   hashMapWith 1  2                   hashMapWith 2  null                   hashMapWith 3  1                   hashMapWith 4  null                   hashMapWith 5  2             assertEquals                  asList 0  2  3                   maps stream                            filter distinctBy m - gt  m get  quot val quot                              map m - gt  m get  quot i quot                             collect toList

User · Answer

In my case I needed to control what was the previous element  I then created a stateful Predicate where I controled if the previous element was  different from the current element  in that case I kept it    public List lt Log gt  fetchLogById Long id        return this findLogById id  stream            filter new LogPredicate             collect Collectors toList        public class LogPredicate implements Predicate lt Log gt         private Log previous       public boolean test Log atual            boolean isDifferent   previouws    null    verifyIfDifferentLog current  previous            if  isDifferent                previous   current                    return isDifferent             private boolean verifyIfDifferentLog Log current  Log previous            return  current getId   equals previous getId

User · Answer

You can use groupingBy collector   persons collect Collectors groupingBy p - gt  p getName     values   forEach t - gt  System out println t get 0  getId        If you want to have another stream you can use this   persons collect Collectors groupingBy p - gt  p getName     values   stream   map l - gt   l get 0

User · Answer

Maybe will be useful for somebody  I had a little bit another requirement  Having list of objects A from 3rd party remove all which have same A b field for same A id  multiple A object with same A id in  list   Stream partition answer by Tagir Valeev inspired me to use custom Collector which returns Map lt A id  List lt A gt  gt   Simple flatMap will do the rest    public static  lt T  K  K2 gt  Collector lt T     Map lt K  List lt T gt  gt  gt  groupingDistinctBy Function lt T  K gt  keyFunction  Function lt T  K2 gt  distinctFunction        return groupingBy keyFunction  Collector of  Supplier lt Map lt K2  T gt  gt   HashMap  new               map  error  - gt  map putIfAbsent distinctFunction apply error   error                left  right  - gt                    left putAll right                   return left                 map - gt  new ArrayList lt  gt  map values                 Collector Characteristics UNORDERED

User · Answer

I made a generic version   private  lt T  R gt  Collector lt T     Stream lt T gt  gt  distinctByKey Function lt T  R gt  keyExtractor        return Collectors collectingAndThen              toMap                      keyExtractor                      t - gt  t                       t1  t2  - gt  t1                             Map lt R  T gt  map  - gt  map values   stream              An exemple   Stream of new Person  Jean               new Person  Jean              new Person  Paul          filter           collect distinctByKey Person  getName      return a stream of Person with 2 elements  jean and Paul      map           collect toList

User · Answer

Another library that supports this is jOO   and its Seq distinct Function lt T U gt   method   Seq seq persons  distinct Person  getName  toList      Under the hood  it does practically the same thing as the accepted answer  though

User · Answer

There s a simpler approach using a TreeSet with a custom comparator   persons stream        collect Collectors toCollection           - gt  new TreeSet lt Person gt   p1  p2  - gt  p1 getName   compareTo p2 getName

User · Answer

We can also use RxJava  very powerful reactive extension library   Observable from persons  distinct Person  getName    or  Observable from persons  distinct p - gt  p getName

User · Answer

Late to the party but I sometimes use this one-liner as an equivalent    Function lt Value  Key gt   Value  getKey  andThen new HashSet lt  gt     add   apply  The expression is a Predicate lt Value gt  but since the map is inline  it works as a filter  This is of course less readable but sometimes it can be helpful to avoid the method

User · Answer

The Most simple code you can write       persons stream   map x- gt  x getName    distinct   collect Collectors toList

User · Answer

Another solution  using Set  May not be the ideal solution  but it works  Set lt String gt  set   new HashSet lt  gt  persons size     persons stream   filter p - gt  set add p getName     collect Collectors toList       Or if you can modify the original list  you can use removeIf method  persons removeIf p - gt   set add p getName

User · Answer

You can wrap the person objects into another class  that only compares the names of the persons  Afterward  you unwrap the wrapped objects to get a person stream again  The stream operations might look as follows  persons stream        map Wrapper  new       distinct        map Wrapper  unwrap            The class Wrapper might look as follows  class Wrapper       private final Person person      public Wrapper Person person            this person   person            public Person unwrap             return person            public boolean equals Object other            if  other instanceof Wrapper                return   Wrapper  other  person getName   equals person getName               else               return false                      public int hashCode             return person getName   hashCode

User · Answer

While the highest upvoted answer is absolutely best answer wrt Java 8  it is at the same time absolutely worst in terms of performance  If you really want a bad low performant application  then go ahead and use it  Simple requirement of extracting a unique set of Person Names shall be achieved by mere  For-Each  and a  Set   Things get even worse if list is above size of 10   Consider you have a collection of 20 Objects  like this   public static final List lt SimpleEvent gt  testList   Arrays asList              new SimpleEvent  Tom    new SimpleEvent  Dick   new SimpleEvent  Harry   new SimpleEvent  Tom                new SimpleEvent  Dick   new SimpleEvent  Huckle   new SimpleEvent  Berry   new SimpleEvent  Tom                new SimpleEvent  Dick   new SimpleEvent  Moses   new SimpleEvent  Chiku   new SimpleEvent  Cherry                new SimpleEvent  Roses   new SimpleEvent  Moses   new SimpleEvent  Chiku   new SimpleEvent  gotya                new SimpleEvent  Gotye   new SimpleEvent  Nibble   new SimpleEvent  Berry   new SimpleEvent  Jibble       Where you object SimpleEvent looks like this   public class SimpleEvent    private String name  private String type   public SimpleEvent String name        this name   name      this type    type   name     public String getName         return name     public void setName String name        this name   name     public String getType         return type     public void setType String type        this type   type        And to test  you have JMH code like this  Please note  im using the same distinctByKey Predicate mentioned in accepted answer      Benchmark  OutputTimeUnit TimeUnit SECONDS  public void aStreamBasedUniqueSet Blackhole blackhole  throws Exception       Set lt String gt  uniqueNames   testList              stream                filter distinctByKey SimpleEvent  getName                map SimpleEvent  getName               collect Collectors toSet         blackhole consume uniqueNames       Benchmark  OutputTimeUnit TimeUnit SECONDS  public void aForEachBasedUniqueSet Blackhole blackhole  throws Exception      Set lt String gt  uniqueNames   new HashSet lt  gt          for  SimpleEvent event   testList            uniqueNames add event getName               blackhole consume uniqueNames      public static void main String   args  throws RunnerException       Options opt   new OptionsBuilder                include MyBenchmark class getSimpleName                 forks 1               mode Mode Throughput               warmupBatchSize 3               warmupIterations 3               measurementIterations 3               build         new Runner opt  run        Then you ll have Benchmark results like this   Benchmark                                  Mode  Samples        Score  Score error  Units c s MyBenchmark aForEachBasedUniqueSet    thrpt        3  2635199 952  1663320 718  ops s c s MyBenchmark aStreamBasedUniqueSet     thrpt        3   729134 695   895825 697  ops s   And as you can see  a simple For-Each is 3 times better in throughput and less in error score as compared to Java 8 Stream   Higher the throughput  better the performance

User · Answer

I had a situation  where I was suppose to get distinct elements from list based on 2 keys   If you want distinct based on two keys or may composite key  try this  class Person      int rollno      String name    List lt Person gt  personList    Function lt Person  List lt Object gt  gt  compositeKey   personList- gt          Arrays  lt Object gt asList personList getName    personList getRollno      Map lt Object  List lt Person gt  gt  map   personList stream   collect Collectors groupingBy compositeKey  Collectors toList       List lt Object gt  duplicateEntrys   map entrySet   stream   enter code here           filter settingMap - gt                  settingMap getValue   size    gt  1           collect Collectors toList

User · Answer

An alternative would be to place the persons in a map using the name as a key   persons collect Collectors toMap Person  getName  p - gt  p   p  q  - gt  p   values      Note that the Person that is kept  in case of a duplicate name  will be the first encontered

User · Answer

You can use StreamEx library   StreamEx of persons           distinct Person  getName           toList

User · Answer

Consider distinct to be a stateful filter  Here is a function that returns a predicate that maintains state about what it s seen previously  and that returns whether the given element was seen for the first time   public static  lt T gt  Predicate lt T gt  distinctByKey Function lt   super T    gt  keyExtractor        Set lt Object gt  seen   ConcurrentHashMap newKeySet        return t - gt  seen add keyExtractor apply t        Then you can write   persons stream   filter distinctByKey Person  getName     Note that if the stream is ordered and is run in parallel  this will preserve an arbitrary element from among the duplicates  instead of the first one  as distinct   does    This is essentially the same as my answer to this question  Java Lambda Stream Distinct   on arbitrary key

User · Answer

Set lt YourPropertyType gt  set   new HashSet lt  gt     list          stream            filter it - gt  set add it getYourProperty              forEach it - gt

User · Answer

Here is the example public class PayRoll        private int payRollId      private int id      private String name      private String dept      private int salary        public PayRoll int payRollId  int id  String name  String dept  int salary            super            this payRollId   payRollId          this id   id          this name   name          this dept   dept          this salary   salary            import java util ArrayList  import java util Comparator  import java util List  import java util Map  import java util Optional  import java util stream Collector  import java util stream Collectors   public class Prac       public static void main String   args             int salary 70000          PayRoll payRoll new PayRoll 1311  1   A    HR   salary           PayRoll payRoll2 new PayRoll 1411  2       B    Technical   salary           PayRoll payRoll3 new PayRoll 1511  1   C    HR   salary           PayRoll payRoll4 new PayRoll 1611  1   D    Technical   salary           PayRoll payRoll5 new PayRoll 711  3  E    Technical   salary           PayRoll payRoll6 new PayRoll 1811  3   F    Technical   salary           List lt PayRoll gt list new ArrayList lt PayRoll gt             list add payRoll           list add payRoll2           list add payRoll3           list add payRoll4           list add payRoll5           list add payRoll6             Map lt Object  Optional lt PayRoll gt  gt  k   list stream   collect Collectors groupingBy p- gt p getId       p getDept   Collectors maxBy Comparator comparingInt PayRoll  getPayRollId                k entrySet   forEach p- gt                        if p getValue   isPresent                                  System out println p getValue   get                                           Output   PayRoll  payRollId 1611  id 1  name D  dept Technical  salary 70000  PayRoll  payRollId 1811  id 3  name F  dept Technical  salary 70000  PayRoll  payRollId 1411  id 2  name B  dept Technical  salary 70000  PayRoll  payRollId 1511  id 1  name C  dept HR  salary 70000

User · Answer

If you want to List of Persons following would be the simple way      Set lt String gt  set   new HashSet lt  gt  persons size     persons stream   filter p - gt  set add p getName     collect Collectors toList       Additionally  if you want to find distinct or unique list of names  not Person   you can do using following two method as well   Method 1  using distinct                                                    persons stream   map x- gt x getName    distinct collect Collectors toList       Method 2  using HashSet  Set lt E gt  set   new HashSet lt  gt     set addAll person stream   map x- gt x getName    collect Collectors toList

User · Answer

Distinct objects list can be found using    List distinctPersons   persons stream                        collect Collectors collectingAndThen                              Collectors toCollection    - gt  new TreeSet lt  gt  Comparator comparing Person   getName                                 ArrayList  new

User · Answer

The easiest way to implement this is to jump on the sort feature as it already provides an optional Comparator which can be created using an element   s property  Then you have to filter duplicates out which can be done using a statefull Predicate which uses the fact that for a sorted stream all equal elements are adjacent   Comparator lt Person gt  c Comparator comparing Person  getName   stream sorted c  filter new Predicate lt Person gt          Person previous      public boolean test Person p          if previous  null  amp  amp  c compare previous  p   0          return false        previous p        return true              more stream operations here       Of course  a statefull Predicate is not thread-safe  however if that   s your need you can move this logic into a Collector and let the stream take care of the thread-safety when using your Collector  This depends on what you want to do with the stream of distinct elements which you didn   t tell us in your question

User · Answer

Building on  josketres s answer  I created a generic utility method   You could make this more Java 8-friendly by creating a Collector   public static  lt T gt  Set lt T gt  removeDuplicates Collection lt T gt  input  Comparator lt T gt  comparer        return input stream                collect toCollection    - gt  new TreeSet lt  gt  comparer          Test public void removeDuplicatesWithDuplicates         ArrayList lt C gt  input   new ArrayList lt  gt         Collections addAll input  new C 7   new C 42   new C 42        Collection lt C gt  result   removeDuplicates input   c1  c2  - gt  Integer compare c1 value  c2 value        assertEquals 2  result size         assertTrue result stream   anyMatch c - gt  c value    7        assertTrue result stream   anyMatch c - gt  c value    42        Test public void removeDuplicatesWithoutDuplicates         ArrayList lt C gt  input   new ArrayList lt  gt         Collections addAll input  new C 1   new C 2   new C 3        Collection lt C gt  result   removeDuplicates input   t1  t2  - gt  Integer compare t1 value  t2 value        assertEquals 3  result size         assertTrue result stream   anyMatch c - gt  c value    1        assertTrue result stream   anyMatch c - gt  c value    2        assertTrue result stream   anyMatch c - gt  c value    3       private class C       public final int value       private C int value            this value   value

User · Answer

Similar approach which Saeed Zarinfam used but more Java 8 style    persons collect Collectors groupingBy p - gt  p getName     values   stream     map plans - gt  plans stream   findFirst   get      collect toList

User · Answer

You can use the distinct HashingStrategy  method in Eclipse Collections   List lt Person gt  persons        MutableList lt Person gt  distinct       ListIterate distinct persons  HashingStrategies fromFunction Person  getName      If you can refactor persons to implement an Eclipse Collections interface  you can call the method directly on the list   MutableList lt Person gt  persons        MutableList lt Person gt  distinct       persons distinct HashingStrategies fromFunction Person  getName      HashingStrategy is simply a strategy interface that allows you to define custom implementations of equals and hashcode   public interface HashingStrategy lt E gt        int computeHashCode E object       boolean equals E object1  E object2       Note  I am a committer for Eclipse Collections

User · Answer

Extending Stuart Marks s answer  this can be done in a shorter way and without a concurrent map  if you don t need parallel streams    public static  lt T gt  Predicate lt T gt  distinctByKey Function lt   super T    gt  keyExtractor        final Set lt Object gt  seen   new HashSet lt  gt         return t - gt  seen add keyExtractor apply t        Then call   persons stream   filter distinctByKey p - gt  p getName

User · Answer

My approach to this is to group all the objects with same property together  then cut short the groups to size of 1 and then finally collect them as a List     List lt YourPersonClass gt  listWithDistinctPersons     persons stream                 operators to remove duplicates based on person name              collect Collectors groupingBy p - gt  p getName                  values                stream                 cut short the groups to size of 1              flatMap group - gt  group stream   limit 1                 collect distinct users as list              collect Collectors toList

User · Answer

My solution in this listing   List lt HolderEntry gt  result       List lt HolderEntry gt  dto3s   new ArrayList lt  gt  result stream   collect toMap              HolderEntry  getId              holder - gt  holder     or Function identity   if you want              holder1  holder2  - gt  holder1         values       In my situation i want to find distinct values and put their in List

[java] Java 8 Distinct by property

Examples related to java

Examples related to collections

Examples related to java-8

Examples related to java-stream

Examples related to distinct-values