Removing duplicates in the lists

Question

Pretty much I need to write a program that checks if a list has any duplicates and if it does it removes them and returns a new list with the items that weren t duplicated removed  This is what I have tried but honestly I don t know what to do  def remove duplicates        t     a    b    c    d       t2     a    c    d       for t in t2          t append t remove        return t

User · Answer

You can use set to remove duplicates   mylist   list set mylist     But note the results will be unordered  If that s an issue   mylist sort

User · Answer

gt  gt  gt  t    1  2  3  1  2  5  6  7  8   gt  gt  gt  t  1  2  3  1  2  5  6  7  8   gt  gt  gt  s       gt  gt  gt  for i in t         if i not in s            s append i   gt  gt  gt  s  1  2  3  5  6  7  8

User · Answer

Very simple way in Python 3    gt  gt  gt  n    1  2  3  4  1  1   gt  gt  gt  n  1  2  3  4  1  1   gt  gt  gt  m   sorted list set n     gt  gt  gt  m  1  2  3  4

User · Answer

Python has built-in many functions You can use set   to remove the duplicate inside the list  As per your example there are below two lists t and t2  t     a    b    c    d   t2     a    c    d   result   list set t  - set t2   result   Answer    b

User · Answer

There are many other answers suggesting different ways to do this  but they re all batch operations  and some of them throw away the original order  That might be okay depending on what you need  but if you want to iterate over the values in the order of the first instance of each value  and you want to remove the duplicates on-the-fly versus all at once  you could use this generator   def uniqify iterable       seen   set       for item in iterable          if item not in seen              seen add item              yield item   This returns a generator iterator  so you can use it anywhere that you can use an iterator   for unique item in uniqify  1  2  3  4  3  2  4  5  6  7  6  8  8        print unique item  end       print     Output   1 2 3 4 5 6 7 8     If you do want a list  you can do this   unique list   list uniqify  1  2  3  4  3  2  4  5  6  7  6  8  8     print unique list    Output    1  2  3  4  5  6  7  8

User · Answer

Here is an example  returning list without repetiotions preserving order  Does not need any external imports   def GetListWithoutRepetitions loInput         return list  consisting of elements of list tuple loInput  without repetitions        Example  GetListWithoutRepetitions  None None 1 1 2 2 3 3 3         Returns   None  1  2  3       if loInput              return         loOutput           if loInput 0  is None          oGroupElement 1     else    loInput 0  lt  gt None         oGroupElement None      for oElement in loInput          if oElement lt  gt oGroupElement              loOutput append oElement              oGroupElement   oElement     return loOutput

User · Answer

There are also solutions using Pandas and Numpy  They both return numpy array so you have to use the function  tolist   if you want a list   t   a   a   b   b   b   c   c   c   t2    c   c   b   b   b   a   a   a     Pandas solution  Using Pandas function unique     import pandas as pd pd unique t  tolist    gt  gt  gt   a   b   c   pd unique t2  tolist    gt  gt  gt   c   b   a     Numpy solution  Using numpy function unique     import numpy as np np unique t  tolist    gt  gt  gt   a   b   c   np unique t2  tolist    gt  gt  gt   a   b   c     Note that numpy unique   also sort the values  So the list t2 is returned sorted  If you want to have the order preserved use as in this answer      idx   np unique t2  return index True  t2 np sort idx   tolist    gt  gt  gt   c   b   a     The solution is not so elegant compared to the others  however  compared to pandas unique    numpy unique   allows you also to check if nested arrays are unique along one selected axis

User · Answer

If you don t care about order and want something different than the pythonic ways suggested above  that is  it can be used in interviews  then     def remove dup arr       size   len arr      j   0      To store index of next unique element     for i in range 0  size-1             If current element is not equal           to next element then store that           current element         if arr i     arr i 1                arr j    arr i              j  1      arr j    arr size-1    Store the last element as whether it is unique or repeated  it hasn t stored previously      return arr 0 j 1   if   name         main         arr    10  10  1  1  1  3  3  4  5  6  7  8  8  9      print remove dup sorted arr      Time Complexity   O n   Auxiliary Space   O n   Reference  http   www geeksforgeeks org remove-duplicates-sorted-array

User · Answer

I think converting to set is the easiest way to remove duplicate   list1    1 2 1  list1   list set list1   print list1

User · Answer

One more better approach could be   import pandas as pd  myList    1  2  3  1  2  5  6  7  8  cleanList   pd Series myList  drop duplicates   tolist   print cleanList     gt   1  2  3  5  6  7  8    and the order remains preserved

User · Answer

Another way of doing    gt  gt  gt  seq    1 2 3  a    a   1 2   gt  gt  dict fromkeys seq  keys     a   1  2  3

User · Answer

Reduce variant with ordering preserve   Assume that we have list   l    5  6  6  1  1  2  2  3  4    Reduce variant  unefficient     gt  gt  gt  reduce lambda r  v  v in r and r or r    v   l       5  6  1  2  3  4    5 x faster but more sophisticated   gt  gt  gt  reduce lambda r  v  v in r 1  and r or  r 0  append v  or r 1  add v   or r  l       set     0   5  6  1  2  3  4    Explanation   default    list    set      user list to keep order   use set to make lookup faster  def reducer result  item       if item not in result 1           result 0  append item          result 1  add item      return result  reduce reducer  l  default  0

User · Answer

The Magic of Python Built-in type  In python  it is very easy to process the complicated cases like this and only by python s built-in type    Let me show you how to do     Method 1  General Case   The way  1 line code  to remove duplicated element in list and still keep sorting order  line    1  2  3  1  2  5  6  7  8  new line   sorted set line   key line index    remove duplicated element print new line    You will get the result   1  2  3  5  6  7  8    Method 2  Special Case   TypeError  unhashable type   list    The special case to process unhashable  3 line codes   line    16 4966155686595    -27 59776154691    52 3786295521147      16 4966155686595    -27 59776154691    52 3786295521147      17 6508629295574    -27 143305738671    47 534955022564      17 6508629295574    -27 143305738671    47 534955022564      18 8051102904552    -26 688849930432    42 6912804930134      18 8051102904552    -26 688849930432    42 6912804930134      19 5504702331098    -26 205884452727    37 7709192714727      19 5504702331098    -26 205884452727    37 7709192714727      20 2929416861422    -25 722717575124    32 8500163147157      20 2929416861422    -25 722717575124    32 8500163147157     tuple line    tuple pt  for pt in line    convert list of list into list of tuple tuple new line   sorted set tuple line  key tuple line index    remove duplicated element new line    list t  for t in tuple new line    convert list of tuple into list of list  print  new line    You will get the result           16 4966155686595    -27 59776154691    52 3786295521147         17 6508629295574    -27 143305738671    47 534955022564         18 8051102904552    -26 688849930432    42 6912804930134         19 5504702331098    -26 205884452727    37 7709192714727         20 2929416861422    -25 722717575124    32 8500163147157       Because tuple is hashable and you can convert data between list and tuple easily

User · Answer

I did this with pure python function  This works when your items value is JSON    i for n  i in enumerate items  if i not in items n   1

User · Answer

this is just a readable funtion  easily understandable  and i have used the dict data structure i have used some builtin funtions and a better complexity of O n   def undup dup list       b        for i in dup list          b update  i 1       return b keys   a   a   b   a   print undup a    disclamer  u may get an indentation error if copy and paste   use the above code with proper indentation before pasting

User · Answer

In this answer  will be two sections  Two unique solutions  and a graph of speed for specific solutions  Removing Duplicate Items Most of these answers only remove duplicate items which are hashable  but this question doesn t imply it doesn t just need hashable items  meaning I ll offer some solutions which don t require hashable items  collections Counter is a powerful tool in the standard library which could be perfect for this  There s only one other solution which even has Counter in it  However  that solution is also limited to hashable keys  To allow unhashable keys in Counter  I made a Container class  which will try to get the object s default hash function  but if it fails  it will try its identity function  It also defines an eq and a hash method  This should be enough to allow unhashable items in our solution  Unhashable objects will be treated as if they are hashable  However  this hash function uses identity for unhashable objects  meaning two equal objects that are both unhashable won t work  I suggest you override this  and changing it to use the hash of an equivalent mutable type  like using hash tuple my list   if my list is a list   I also made two solutions  Another solution which keeps the order of the items  using a subclass of both OrderedDict and Counter which is named  OrderedCounter   Now  here are the functions  from collections import OrderedDict  Counter  class Container      def   init   self  obj           self obj   obj     def   eq   self  obj           return self obj    obj     def   hash   self           try              return hash self obj          except              return id self obj   class OrderedCounter Counter  OrderedDict         Counter that remembers the order elements are first encountered        def   repr   self            return   s  r      self   class     name    OrderedDict self         def   reduce   self            return self   class     OrderedDict self     def remd sequence       cnt   Counter       for x in sequence          cnt Container x      1     return  item obj for item in cnt   def oremd sequence       cnt   OrderedCounter       for x in sequence          cnt Container x      1     return  item obj for item in cnt   remd is non-ordered sorting  oremd is ordered sorting  You can clearly tell which one is faster  but I ll explain anyways  The non-ordered sorting is slightly faster  It keeps less data  since it doesn t need order  Now  I also wanted to show the speed comparisons of each answer  So  I ll do that now  Which Function is the Fastest  For removing duplicates  I gathered 10 functions from a few answers  I calculated the speed of each function and put it into a graph using matplotlib pyplot  I divided this into three rounds of graphing  A hashable is any object which can be hashed  an unhashable is any object which cannot be hashed  An ordered sequence is a sequence which preserves order  an unordered sequence does not preserve order  Now  here are a few more terms  Unordered Hashable was for any method which removed duplicates  which didn t necessarily have to keep the order  It didn t have to work for unhashables  but it could  Ordered Hashable was for any method which kept the order of the items in the list  but it didn t have to work for unhashables  but it could  Ordered Unhashable was any method which kept the order of the items in the list  and worked for unhashables  On the y-axis is the amount of seconds it took  On the x-axis is the number the function was applied to  We generated sequences for unordered hashables and ordered hashables with the following comprehension   list range x     list range x   for x in range 0  1000  10   For ordered unhashables    list range y     list range y   for y in range x   for x in range 0  1000  10   Note there is a  step  in the range because without it  this would ve taken 10x as long  Also because in my personal opinion  I thought it might ve looked a little easier to read  Also note the keys on the legend are what I tried to guess as the most vital parts of the function  As for what function does the worst or best  The graph speaks for itself  With that settled  here are the graphs  Unordered Hashables   Zoomed in   Ordered Hashables   Zoomed in   Ordered Unhashables   Zoomed in

User · Answer

list with unique items   list set list with duplicates

User · Answer

To make a new list  retaining the order of first elements of duplicates in L  newlist  ii for n ii in enumerate L  if ii not in L  n    for example if L  1  2  2  3  4  2  4  3  5  then newlist will be  1 2 3 4 5   This checks each new element has not appeared previously in the list before adding it   Also it does not need imports

User · Answer

It requires installing a 3rd-party module but the package iteration utilities contains a unique everseen1 function that can remove all duplicates while preserving the order    gt  gt  gt  from iteration utilities import unique everseen   gt  gt  gt  list unique everseen   a    b    c    d       a    c    d       a    b    c    d     In case you want to avoid the overhead of the list addition operation you can use itertools chain instead    gt  gt  gt  from itertools import chain  gt  gt  gt  list unique everseen chain   a    b    c    d      a    c    d        a    b    c    d     The unique everseen also works if you have unhashable items  for example lists  in the lists    gt  gt  gt  from iteration utilities import unique everseen  gt  gt  gt  list unique everseen    a      b     c    d       a    c    d        a      b     c    d    a     However that will be  much  slower than if the items are hashable     1 Disclosure  I m the author of the iteration utilities-library

User · Answer

Here s the fastest pythonic solution comaring to others listed in replies   Using implementation details of short-circuit evaluation allows to use list comprehension  which is fast enough  visited add item  always returns None as a result  which is evaluated as False  so the right-side of or would always be the result of such an expression   Time it yourself  def deduplicate sequence       visited   set       adder   visited add    get rid of qualification overhead     out    adder item  or item for item in sequence if item not in visited      return out

User · Answer

Best approach of removing duplicates from a list is using set   function  available in python  again converting that set into list  In  2   some list     a   a   v   v   v   c   c   d   In  3   list set some list   Out 3     a    c    d    v

User · Answer

There are a lot of answers here that use a set      which is fast given the elements are hashable   or a list  which has the downside that it results in an O n2  algorithm   The function I propose is a hybrid one  we use a set     for items that are hashable  and a list     for the ones that are not  Furthermore it is implemented as a generator such that we can for instance limit the number of items  or do some additional filtering   Finally we also can use a key argument to specify in what way the elements should be unique  For instance we can use this if we want to filter a list of strings such that every string in the output has a different length   def uniq iterable  key lambda x  x       seens   set       seenl          for item in iterable          k   key item          try              seen   k in seens         except TypeError              seen   k in seenl         if not seen              yield item             try                  seens add k              except TypeError                  seenl append k   We can now for instance use this like    gt  gt  gt  list uniq   apple    pear    banana    lemon    len     apple    pear    banana    gt  gt  gt  list uniq   apple    pear    lemon    banana    len     apple    pear    banana    gt  gt  gt  list uniq   apple    pear        lemon        banana    len     apple    pear        banana    gt  gt  gt  list uniq   apple    pear        lemon        banana       apple    pear        lemon        banana    gt  gt  gt  list uniq   apple    pear        lemon        banana       apple    pear        lemon    banana     It is thus a uniqeness filter that can work on any iterable and filter out uniques  regardless whether these are hashable or not   It makes one assumption  that if one object is hashable  and another one is not  the two objects are never equal  This can strictly speaking happen  although it would be very uncommon

User · Answer

Try using sets   import sets t   sets Set   a    b    c    d    t1   sets Set   a    b    c     print t   t1 print t - t1

User · Answer

This one cares about the order without too much hassle  OrderdDict  amp  others   Probably not the most Pythonic way  nor shortest way  but does the trick   def remove duplicates list           Removes duplicate items from a list         singles list          for element in list          if element not in singles list              singles list append element      return singles list

User · Answer

I had a dict in my list  so I could not use the above approach  I got the error   TypeError  unhashable type    So if you care about order and or some items are unhashable  Then you might find this useful   def make unique original list       unique list           unique list append obj  for obj in original list if obj not in unique list      return unique list   Some may consider list comprehension with a side effect to not be a good solution  Here s an alternative   def make unique original list       unique list          map lambda x  unique list append x  if  x not in unique list  else False  original list      return unique list

User · Answer

In Python 2 7  the new way of removing duplicates from an iterable while keeping it in the original order is    gt  gt  gt  from collections import OrderedDict  gt  gt  gt  list OrderedDict fromkeys  abracadabra      a    b    r    c    d     In Python 3 5  the OrderedDict has a C implementation  My timings show that this is now both the fastest and shortest of the various approaches for Python 3 5   In Python 3 6  the regular dict became both ordered and compact    This feature is holds for CPython and PyPy but may not present in other implementations    That gives us a new fastest way of deduping while retaining order    gt  gt  gt  list dict fromkeys  abracadabra      a    b    r    c    d     In Python 3 7  the regular dict is guaranteed to both ordered across all implementations   So  the shortest and fastest solution is    gt  gt  gt  list dict fromkeys  abracadabra      a    b    r    c    d

User · Answer

It s a one-liner  list set source list   will do the trick   A set is something that can t possibly have duplicates   Update  an order-preserving approach is two lines   from collections import OrderedDict OrderedDict  x  True  for x in source list  keys     Here we use the fact that OrderedDict remembers the insertion order of keys  and does not change it when a value at a particular key is updated  We insert True as values  but we could insert anything  values are just not used   set works a lot like a dict with ignored values  too

User · Answer

Simple and easy   myList    1  2  3  1  2  5  6  7  8  cleanlist       cleanlist append x  for x in myList if x not in cleanlist    Output    gt  gt  gt  cleanlist   1  2  3  5  6  7  8

User · Answer

For completeness  and since this is a very popular question  the toolz library offers a unique function    gt  gt  gt  tuple unique  1  2  3     1  2  3   gt  gt  gt  tuple unique  1  2  1  3     1  2  3

User · Answer

Unfortunately  Most answers here either do not preserve the order or are too long  Here is a simple  order preserving answer   s    1 2 3 4 5 2 5 6 7 1 3 9 3 5  x      x append i  for i in s if i not in x  print x    This will give you x with duplicates removed but preserving the order

User · Answer

If you want to preserve the order  and not use any external modules here is an easy way to do this    gt  gt  gt  t    1  9  2  3  4  5  3  6  7  5  8  9   gt  gt  gt  list dict fromkeys t    1  9  2  3  4  5  6  7  8    Note  This method preserves the order of appearance  so  as seen above  nine will come after one because it was the first time it appeared  This however  is the same result as you would get with doing  from collections import OrderedDict ulist list OrderedDict fromkeys l     but it is much shorter  and runs faster   This works because each time the fromkeys function tries to create a new key  if the value already exists it will simply overwrite it  This wont affect the dictionary at all however  as fromkeys creates a dictionary where all keys have the value None  so effectively it eliminates all duplicates this way

User · Answer

You can do this simply by using sets   Step1  Get Different elements of lists  Step2 Get Common elements of lists  Step3 Combine them  In  1   a     apples    bananas    cucumbers    In  2   b     pears    apples    watermelons    In  3   set a  symmetric difference b  union set a  intersection b   Out 3     apples    bananas    cucumbers    pears    watermelons

User · Answer

The common approach to get a unique collection of items is to use a set  Sets are unordered collections of distinct objects  To create a set from any iterable  you can simply pass it to the built-in set   function  If you later need a real list again  you can similarly pass the set to the list   function   The following example should cover whatever you are trying to do    gt  gt  gt  t    1  2  3  1  2  5  6  7  8   gt  gt  gt  t  1  2  3  1  2  5  6  7  8   gt  gt  gt  list set t    1  2  3  5  6  7  8   gt  gt  gt  s    1  2  3   gt  gt  gt  list set t  - set s    8  5  6  7    As you can see from the example result  the original order is not maintained  As mentioned above  sets themselves are unordered collections  so the order is lost  When converting a set back to a list  an arbitrary order is created   Maintaining order  If order is important to you  then you will have to use a different mechanism  A very common solution for this is to rely on OrderedDict to keep the order of keys during insertion    gt  gt  gt  from collections import OrderedDict  gt  gt  gt  list OrderedDict fromkeys t    1  2  3  5  6  7  8    Starting with Python 3 7  the built-in dictionary is guaranteed to maintain the insertion order as well  so you can also use that directly if you are on Python 3 7 or later  or CPython 3 6     gt  gt  gt  list dict fromkeys t    1  2  3  5  6  7  8    Note that this may have some overhead of creating a dictionary first  and then creating a list from it  If you don   t actually need to preserve the order  you   re often better off using a set  especially because it gives you a lot more operations to work with  Check out this question for more details and alternative ways to preserve the order when removing duplicates     Finally note that both the set as well as the OrderedDict dict solutions require your items to be hashable  This usually means that they have to be immutable  If you have to deal with items that are not hashable  e g  list objects   then you will have to use a slow approach in which you will basically have to compare every item with every other item in a nested loop

User · Answer

below code is simple for removing duplicate in list  def remove duplicates x       a          for i in x          if i not in a              a append i      return a  print remove duplicates  1 2 2 3 3 4     it returns  1 2 3 4

User · Answer

def remove duplicates A       A pop count  for count elem in enumerate A  if A count elem   1     return A   A list comprehesion to remove duplicates

User · Answer

If you don t care about the order  just do this   def remove duplicates l       return list set l     A set is guaranteed to not have duplicates

User · Answer

Another solution might be the following  Create a dictionary out of the list with item as key and index as value  and then print the dictionary keys    gt  gt  gt  lst    1  3  4  2  1  21  1  32  21  1  6  5  7  8  2   gt  gt  gt   gt  gt  gt  dict enum    item index for index  item in enumerate lst    gt  gt  gt  print dict enum keys    32  1  2  3  4  5  6  7  8  21

User · Answer

Test    1 8 2 7 3 4 5 1 2 3 6  Test sort   i 1 while i lt  len Test     if Test i     Test i-1       Test remove Test i     i  i 1 print Test

User · Answer

A colleague have sent the accepted answer as part of his code to me for a codereview today  While I certainly admire the elegance of the answer in question  I am not happy with the performance  I have tried this solution  I use set to reduce lookup time   def ordered set in list       out list          added   set       for val in in list          if not val in added              out list append val              added add val      return out list   To compare efficiency  I used a random sample of 100 integers - 62 were unique  from random import randint x    randint 0 100  for   in xrange 100    In  131   len set x   Out 131   62   Here are the results of the measurements  In  129    timeit list OrderedDict fromkeys x   10000 loops  best of 3  86 4 us per loop  In  130    timeit ordered set x  100000 loops  best of 3  15 1 us per loop   Well  what happens if set is removed from the solution   def ordered set inlist       out list          for val in inlist          if not val in out list              out list append val      return out list   The result is not as bad as with the OrderedDict  but still more than 3 times of the original solution  In  136    timeit ordered set x  10000 loops  best of 3  52 6 us per loop

User · Answer

Very late answer  If you don t care about the list order  you can use  arg expansion with set uniqueness to remove dupes  i e   l       l    Demo

User · Answer

Using set    a    0 1 2 3 4 3 3 4  a   list set a   print a   Using unique    import numpy as np a    0 1 2 3 4 3 3 4  a   np unique a  tolist   print a

User · Answer

Without using set   data  1  2  3  1  2  5  6  7  8  uni data    for dat in data      if dat not in uni data          uni data append dat   print uni data

User · Answer

Sometimes you need to remove the duplicate items in-place  without creating new list  For example  the list is big  or keep it as a shadow copy  from collections import Counter cntDict   Counter t  for item cnt in cntDict items        for   in range cnt-1           t remove item

User · Answer

To remove the duplicates  make it a SET and then again make it a LIST and print use it  A set is guaranteed to have unique elements  For example     a    1 2 3 4 5 9 11 15  b    4 5 6 7 8  c a b print c print list set c    one line for getting unique elements of c   The output will be as follows  checked in python 2 7    1  2  3  4  5  9  11  15  4  5  6  7  8    simple list addition with duplicates  1  2  3  4  5  6  7  8  9  11  15   duplicates removed

User · Answer

Check for the string  a  and  b  clean list          for ele in raw list          if  b  in ele or  a  in ele              pass         else              clean list append ele

User · Answer

I didn t see answers for non-hashable values  one liner  n log n  standard-library only  so here s my answer  list map operator itemgetter 0   itertools groupby sorted items      Or as a generator function  def unique items  Iterable T   - gt  Iterable T        quot  quot  quot For unhashable items  can t use set to unique  with a partial order quot  quot  quot      yield from map operator itemgetter 0   itertools groupby sorted items

User · Answer

def remove duplicates input list     if input list            return       sort list from smallest to largest   input list sorted input list     initialize ouput list with first element of the       sorted input list   output list    input list 0     for item in input list      if item  gt output list -1         output list append item    return output list

User · Answer

You can use the following function    def rem dupes dup list        yooneeks           for elem in dup list           if elem not in yooneeks               yooneeks append elem       return yooneeks   Example    my list     this   is   a   list   with   dupicates   in    the    list     Usage   rem dupes my list      this    is    a    list    with    dupicates    in    the

User · Answer

All the order-preserving approaches I ve seen here so far either use naive  comparison  with O n 2  time-complexity at best  or heavy-weight OrderedDicts set list combinations that are limited to hashable inputs  Here is a hash-independent O nlogn  solution   Update added the key argument  documentation and Python 3 compatibility     from functools import reduce  lt -- add this import on Python 3  def uniq iterable  key lambda x  x               Remove duplicates from an iterable  Preserves order        type iterable  Iterable Ord   gt  A       param iterable  an iterable of objects of any orderable type      type key  Callable A  - gt   Ord   gt  B       param key  optional argument  by default an item  A  is discarded      if another item  B   such that A    B  has already been encountered and taken       If you provide a key  this condition changes to key A     key B   the callable      must return orderable objects                Enumerate the list to restore order lately  reduce the sorted list  restore order     def append unique acc  item           return acc if key acc -1  1      key item 1   else acc append item  or acc      srt enum   sorted enumerate iterable   key lambda item  key item 1        return  item 1  for item in sorted reduce append unique  srt enum   srt enum 0

User · Answer

Check this if you want to remove duplicates  in-place edit rather than returning new list  without using inbuilt set  dict keys  uniqify  counter   gt  gt  gt  t    1  2  3  1  2  5  6  7  8   gt  gt  gt  for i in t          if i in t t index i  1                t remove i        gt  gt  gt  t  3  1  2  5  6  7  8

User · Answer

You could also do this    gt  gt  gt  t    1  2  3  3  2  4  5  6   gt  gt  gt  s    x for i  x in enumerate t  if i    t index x    gt  gt  gt  s  1  2  3  4  5  6    The reason that above works is that index method returns only the first index of an element  Duplicate elements have higher indices  Refer to here      list index x   start   end      Return zero-based index in the list of   the first item whose value is x     Raises a ValueError if there is no   such item

User · Answer

If your list is ordered  you can use the following approach to iterate over it skipping the repeated values  This is especially useful to handle big lists with low memory consumption evading the cost of building a dict or a set   def uniq iterator       prev   None     for item in iterator          if item    prev              prev   item             yield item   Then   for item in uniq  1  1  3  5  5  6        print item  end        The output is going to be  1 3 5 6  To return a list object  you could do    gt  gt  gt  print list uniq  1  1  3  5  5  6      1  3  5  6

[python] Removing duplicates in the lists

Examples related to python

Examples related to algorithm

Examples related to list

Examples related to duplicates

Examples related to intersection