Remove duplicates from a List T in C

Question

Anyone have a quick method for de-duplicating a generic List in C

User · Answer

If you re using  Net 3   you can use Linq   List lt T gt  withDupes   LoadSomeData    List lt T gt  noDupes   withDupes Distinct   ToList

User · Answer

This takes distinct  the elements without duplicating elements  and convert it into a list again   List lt type gt  myNoneDuplicateValue   listValueWithDuplicate Distinct   ToList

User · Answer

All answers copy lists  or create a new list  or use slow functions  or are just painfully slow   To my understanding  this is the fastest and cheapest method I know  also  backed by a very experienced programmer specialized on real-time physics optimization       Duplicates will be noticed after a sort O nLogn  list Sort        Store the current and last items  Current item declaration is not really needed  and probably optimized by the compiler  but in case it s not    int lastItem   -1  int currItem   -1   int size   list Count      Store the index pointing to the last item we want to keep in the list int last   size - 1      Travel the items from last to first O n  for  int i   last  i  gt   0  --i        currItem   list i           If this item was the same as the previous one  we don t want it     if  currItem    lastItem                   Overwrite last in current place  It is a swap but we don t need the last        list i    list last               Reduce the last index  we don t want that one anymore         last--                A new item  we store it and continue     else         lastItem   currItem        We now have an unsorted list with the duplicates at the end      Remove the last items just once list RemoveRange last   1  size - last - 1       Sort again O n logn  list Sort      Final cost is   nlogn   n   nlogn   n   2nlogn   O nlogn  which is pretty nice   Note about RemoveRange  Since we cannot set the count of the list and avoid using the Remove funcions  I don t know exactly the speed of this operation but I guess it is the fastest way

User · Answer

How about   var noDupes   list Distinct   ToList      In  net 3 5

User · Answer

It worked for me  simply use  List lt Type gt  liIDs   liIDs Distinct   ToList lt Type gt       Replace  Type  with your desired type e g  int

User · Answer

If you have tow classes Product and Customer and we want to remove duplicate items from their list  public class Product       public int Id   get  set        public string ProductName   get  set       public class Customer       public int Id   get  set        public string CustomerName   get  set         You must define a generic class in the form below  public class ItemEqualityComparer lt T gt    IEqualityComparer lt T gt  where T   class       private readonly PropertyInfo  propertyInfo       public ItemEqualityComparer string keyItem                 propertyInfo   typeof T  GetProperty keyItem  BindingFlags GetProperty   BindingFlags Instance   BindingFlags Public              public bool Equals T x  T y                var xValue    propertyInfo  GetValue x  null           var yValue    propertyInfo  GetValue y  null           return xValue    null  amp  amp  yValue    null  amp  amp  xValue Equals yValue              public int GetHashCode T obj                var propertyValue    propertyInfo GetValue obj  null           return propertyValue    null   0   propertyValue GetHashCode              then  You can remove duplicate items in your list   var products   new List lt Product gt                                new Product ProductName    product 1   Id   1                    new Product ProductName    product 2   Id   2                    new Product ProductName    product 2   Id   4                    new Product ProductName    product 2   Id   4                   var productList   products Distinct new ItemEqualityComparer lt Product gt  nameof Product Id    ToList     var customers   new List lt Customer gt                                new Customer CustomerName    Customer 1   Id   5                    new Customer CustomerName    Customer 2   Id   5                    new Customer CustomerName    Customer 2   Id   5                    new Customer CustomerName    Customer 2   Id   5                   var customerList   customers Distinct new ItemEqualityComparer lt Customer gt  nameof Customer Id    ToList      this code remove duplicate items by Id if you want remove duplicate items by other property  you can change nameof YourClass DuplicateProperty   same nameof Customer CustomerName  then remove duplicate items by CustomerName Property

User · Answer

There are many ways to solve - the duplicates issue in the List  below is one of them   List lt Container gt  containerList   LoadContainer     Assume it has duplicates List lt Container gt  filteredList   new  List lt Container gt     foreach  var container in containerList       Container duplicateContainer   containerList Find delegate Container checkContainer      return  checkContainer UniqueId    container UniqueId            Assume  UniqueId  is the property of the Container class on which u r making a search      if  containerList Contains duplicateContainer    Add object when not found in the new class object                 filteredList Add container                  Cheers Ravi Ganesan

User · Answer

You can use Union  obj2   obj1 Union obj1  ToList

User · Answer

If you don t care about the order you can just shove the items into a HashSet  if you do want to maintain the order you can do something like this   var unique   new List lt T gt     var hs   new HashSet lt T gt     foreach  T t in list      if  hs Add t           unique Add t     Or the Linq way   var hs   new HashSet lt T gt     list All  x   gt   hs Add x       Edit  The HashSet method is O N  time and O N  space while sorting and then making unique  as suggested by  lassevk and others  is O N lgN  time and O 1  space so it s not so clear to me  as it was at first glance  that the sorting way is inferior  my apologies for the temporary down vote

User · Answer

As kronoz said in  Net 3 5 you can use Distinct     In  Net 2 you could mimic it   public IEnumerable lt T gt  DedupCollection lt T gt   IEnumerable lt T gt  input         var passedValues   new HashSet lt T gt             Relatively simple dupe check alg used as example     foreach T item in input          if passedValues Add item      True if item is new             yield return item      This could be used to dedupe any collection and will return the values in the original order   It s normally much quicker to filter a collection  as both Distinct   and this sample does  than it would be to remove items from it

User · Answer

David J  s answer is a good method  no need for extra objects  sorting  etc  It can be improved on however   for  int innerIndex   items Count - 1  innerIndex  gt  outerIndex   innerIndex--   So the outer loop goes top bottom for the entire list  but the inner loop goes bottom  until the outer loop position is reached    The outer loop makes sure the entire list is processed  the inner loop finds the actual duplicates  those can only happen in the part that the outer loop hasn t processed yet   Or if you don t want to do bottom up for the inner loop you could have the inner loop start at outerIndex   1

User · Answer

Might be easier to simply make sure that duplicates are not added to the list   if items IndexOf new item   lt  0       items add new item

User · Answer

Installing the MoreLINQ package via Nuget  you can easily distinct object list by a property   IEnumerable lt Catalogue gt  distinctCatalogues   catalogues DistinctBy c   gt  c CatalogueCode

User · Answer

I like to use this command   List lt Store gt  myStoreList   Service GetStoreListbyProvince provinceId                                                    GroupBy s   gt  s City                                                    Select grp   gt  grp FirstOrDefault                                                      OrderBy s   gt  s City                                                    ToList      I have these fields in my list  Id  StoreName  City  PostalCode  I wanted to show list of cities in a dropdown which has duplicate values  solution  Group by city then pick the first one for the list   I hope it helps

User · Answer

As a helper method  without Linq    public static List lt T gt  Distinct lt T gt  this List lt T gt  list        return  new HashSet lt T gt  list   ToList

User · Answer

public static void RemoveDuplicates lt T gt  IList lt T gt  list            if  list    null                 return              int i   1       while i lt list Count                 int j   0          bool remove   false          while  j  lt  i  amp  amp   remove                       if  list i  Equals list j                               remove   true                          j                      if  remove                       list RemoveAt i                     else                      i

User · Answer

Another way in  Net 2 0      static void Main string   args                List lt string gt  alpha   new List lt string gt              for char a    a   a  lt    d   a                          alpha Add a ToString                 alpha Add a ToString                        Console WriteLine  Data              alpha ForEach delegate string t    Console WriteLine t                alpha ForEach delegate  string v                                                            if  alpha FindAll delegate string t    return t    v     Count  gt  1                                    alpha Remove v                                          Console WriteLine  Unique Result              alpha ForEach delegate string t    Console WriteLine t              Console ReadKey

User · Answer

Here s a simple solution that doesn t require any hard-to-read LINQ or any prior sorting of the list      private static void CheckForDuplicateItems List lt string gt  items                if  items    null                items Count    0              return           for  int outerIndex   0  outerIndex  lt  items Count  outerIndex                          for  int innerIndex   0  innerIndex  lt  items Count  innerIndex                                  if  innerIndex    outerIndex  continue                  if  items outerIndex  Equals items innerIndex                                             Duplicate Found

User · Answer

In Java  I assume C  is more or less identical    list   new ArrayList lt T gt  new HashSet lt T gt  list     If you really wanted to mutate the original list   List lt T gt  noDupes   new ArrayList lt T gt  new HashSet lt T gt  list    list clear    list addAll noDupes     To preserve order  simply replace HashSet with LinkedHashSet

User · Answer

An extension method might be a decent way to go    something like this   public static List lt T gt  Deduplicate lt T gt  this List lt T gt  listToDeduplicate        return listToDeduplicate Distinct   ToList        And then call like this  for example   List lt int gt  myFilteredList   unfilteredList Deduplicate

User · Answer

Perhaps you should consider using a HashSet   From the MSDN link   using System  using System Collections Generic   class Program       static void Main                 HashSet lt int gt  evenNumbers   new HashSet lt int gt             HashSet lt int gt  oddNumbers   new HashSet lt int gt              for  int i   0  i  lt  5  i                             Populate numbers with just even numbers              evenNumbers Add i   2                   Populate oddNumbers with just odd numbers              oddNumbers Add  i   2    1                      Console Write  evenNumbers contains  0  elements     evenNumbers Count           DisplaySet evenNumbers            Console Write  oddNumbers contains  0  elements     oddNumbers Count           DisplaySet oddNumbers               Create a new HashSet populated with even numbers          HashSet lt int gt  numbers   new HashSet lt int gt  evenNumbers           Console WriteLine  numbers UnionWith oddNumbers               numbers UnionWith oddNumbers            Console Write  numbers contains  0  elements     numbers Count           DisplaySet numbers              private static void DisplaySet HashSet lt int gt  set                Console Write               foreach  int i in set                        Console Write    0    i                     Console WriteLine                    This example produces output similar to the following     evenNumbers contains 5 elements    0 2 4 6 8      oddNumbers contains 5 elements    1 3 5 7 9      numbers UnionWith oddNumbers       numbers contains 10 elements    0 2 4 6 8 1 3 5 7 9

User · Answer

Use Linq s Union method    Note  This solution requires no knowledge of Linq  aside from that it exists   Code  Begin by adding the following to the top of your class file   using System Linq    Now  you can use the following to remove duplicates from an object called  obj1   obj1   obj1 Union obj1  ToList      Note  Rename obj1 to the name of your object   How it works   The Union command lists one of each entry of two source objects  Since obj1 is both source objects  this reduces obj1 to one of each entry  The ToList   returns a new List  This is necessary  because Linq commands like Union returns the result as an IEnumerable result instead of modifying the original List or returning a new List

User · Answer

A simple intuitive implementation   public static List lt PointF gt  RemoveDuplicates List lt PointF gt  listPoints        List lt PointF gt  result   new List lt PointF gt          for  int i   0  i  lt  listPoints Count  i                  if   result Contains listPoints i                result Add listPoints i                       return result

User · Answer

Here s an extension method for removing adjacent duplicates in-situ  Call Sort   first and pass in the same IComparer  This should be more efficient than Lasse V  Karlsen s version which calls RemoveAt repeatedly  resulting in multiple block memory moves    public static void RemoveAdjacentDuplicates lt T gt  this List lt T gt  List  IComparer lt T gt  Comparer        int NumUnique   0      for  int i   0  i  lt  List Count  i            if   i    0      Comparer Compare List NumUnique - 1   List i      0               List NumUnique      List i       List RemoveRange NumUnique  List Count - NumUnique

User · Answer

Sort it  then check two and two next to each others  as the duplicates will clump together   Something like this   list Sort    Int32 index   list Count - 1  while  index  gt  0        if  list index     list index - 1                 if  index  lt  list Count - 1               list index   list list Count - 1      list list Count - 1   list index            list RemoveAt list Count - 1           index--            else         index--      Notes    Comparison is done from back to front  to avoid having to resort list after each removal This example now uses C  Value Tuples to do the swapping  substitute with appropriate code if you can t use that The end-result is no longer sorted

User · Answer

I think the simplest way is  Create a new list and add unique item  Example          class MyList      int id      string date      string email                 List lt MyList gt  ml   new Mylist     ml Add new MyList    id   1  date    quot 2020 09 06 quot   email    quot zarezadeh gmailcom quot       ml Add new MyList    id   2  date    quot 2020 09 01 quot   email    quot zarezadeh gmailcom quot        List lt MyList gt  New ml   new Mylist     foreach  var item in ml                                        if  New ml Where w   gt  w email    item email  SingleOrDefault      null                                                New ml Add new MyList                                                       id   item id       date   item date                 email   item email

User · Answer

Simply initialize a HashSet with a List of the same type   var noDupes   new HashSet lt T gt  withDupes     Or  if you want a List returned   var noDupsList   new HashSet lt T gt  withDupes  ToList

[c#] Remove duplicates from a List<T> in C#

Examples related to c#

Examples related to list

Examples related to generics

Examples related to duplicates