Get unique values from a list in python

Question

I want to get the unique values from the following list     nowplaying    PBS    PBS    nowplaying    job    debate    thenandnow     The output which I require is     nowplaying    PBS    job    debate    thenandnow     This code works   output      for x in trends      if x not in output          output append x  print output    is there a better solution I should use

User · Answer

use set to de-duplicate a list  return as list  def get unique list lst           if isinstance lst list               return list set lst

User · Answer

A Python list    gt  gt  gt  a     a    b    c    d    b     To get unique items  just transform it into a set  which you can transform back again into a list if required     gt  gt  gt  b   set a   gt  gt  gt  print b    b    c    d    a

User · Answer

set can help you filter out the elements from the list that are duplicates  It will work well for str  int or tuple elements  but if your list contains dict or other list elements  then you will end up with TypeError exceptions   Here is a general order-preserving solution to handle some  not all  non-hashable types   def unique elements iterable       seen   set       result          for element in iterable          hashed   element         if isinstance element  dict               hashed   tuple sorted element iteritems             elif isinstance element  list               hashed   tuple element          if hashed not in seen              result append element              seen add hashed      return result

User · Answer

First declare your list properly  separated by commas  You can get the unique values by converting the list to a set   mylist     nowplaying    PBS    PBS    nowplaying    job    debate    thenandnow   myset   set mylist  print myset    If you use it further as a list  you should convert it back to a list by doing   mynewlist   list myset    Another possibility  probably faster would be to use a set from the beginning  instead of a list  Then your code should be   output   set   for x in trends      output add x  print output    As it has been pointed out  sets do not maintain the original order  If you need that  you should look for an ordered set implementation  see this question for more

User · Answer

If you want to get unique elements from a list and keep their original order  then you may employ OrderedDict data structure from Python s standard library   from collections import OrderedDict  def keep unique elements       return list OrderedDict fromkeys elements  keys     elements    2  1  4  2  1  1  5  3  1  1  required output    2  1  4  5  3   assert keep unique elements     required output   In fact  if you are using Python   3 6  you can use plain dict for that   def keep unique elements       return list dict fromkeys elements  keys      It s become possible after the introduction of  compact  representation of dicts  Check it out here  Though this  considered an implementation detail and should not be relied upon

User · Answer

I know this is an old question  but here s my unique solution  class inheritance    class UniqueList list       def appendunique self item           if item not in self              self append item              return True         return False   Then  if you want to uniquely append items to a list you just call appendunique on a UniqueList  Because it inherits from a list  it basically acts like a list  so you can use functions like index   etc  And because it returns true or false  you can find out if appending succeeded  unique item  or failed  already in the list    To get a unique list of items from a list  use a for loop appending items to a UniqueList  then copy over to the list    Example usage code   unique   UniqueList    for each in  1 2 2 3 3 4       if unique appendunique each           print  Uniquely appended     str each      else          print  Already contains     str each    Prints   Uniquely appended 1 Uniquely appended 2 Already contains 2 Uniquely appended 3 Already contains 3 Uniquely appended 4   Copying to list   unique   UniqueList    for each in  1 2 2 3 3 4       unique appendunique each   newlist   unique    print newlist   Prints    1  2  3  4

User · Answer

You can use sets  Just to be clear  I am explaining what is the difference between a list and a set   sets are unordered collection of unique elements Lists are ordered collection of elements  So       unicode list  u nowplaying   u PBS   u PBS   u nowplaying   u job  u debate   u thenandnow       list unique list set unicode list       print list unique  u nowplaying   u job   u debate   u PBS   u thenandnow     But  Do not use list set in naming the variables  It will cause error   EX  Instead of use list instead of unicode list in the above one    list  u nowplaying   u PBS   u PBS   u nowplaying   u job  u debate   u thenandnow           list unique list set list           print list unique     list unique list set list   TypeError   list  object is not callable

User · Answer

def setlist lst         return list set lst

User · Answer

If you are using numpy in your code  which might be a good choice for larger amounts of data   check out numpy unique    gt  gt  gt  import numpy as np  gt  gt  gt  wordsList    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow    gt  gt  gt  np unique wordsList  array  u PBS   u debate   u job   u nowplaying   u thenandnow           dtype   lt U10      http   docs scipy org doc numpy reference generated numpy unique html   As you can see  numpy supports not only numeric data  string arrays are also possible  Of course  the result is a numpy array  but it doesn t matter a lot  because it still behaves like a sequence    gt  gt  gt  for word in np unique wordsList           print word      PBS debate job nowplaying thenandnow   If you really want to have a vanilla python list back  you can always call list     However  the result is automatically sorted  as you can see from the above code fragments  Check out numpy unique without sort if retaining list order is required

User · Answer

Set is a collection of un-ordered and unique elements  So  you can use set as below to get a unique list   unique list   list set  u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow

User · Answer

As a bonus  Counter is a simple way to get both the unique values and the count for each value   from collections import Counter l    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow   c   Counter l

User · Answer

What type is your output variable   Python sets are what you need  Declare output like this   output   set      initialize an empty set   and you re ready to go adding elements with output add elem  and be sure they re unique   Warning  sets DO NOT preserve the original order of the list

User · Answer

Options to remove duplicates may include the following generic data structures    set  unordered  unique elements ordered set  ordered  unique elements   Here is a summary on quickly getting either one in Python   Given  from collections import OrderedDict   seq    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow     Code  Option 1 - A set  unordered    list set seq       thenandnow    PBS    debate    job    nowplaying     Option 2 - Python doesn t have ordered sets  but here are some ways to mimic one  insertion ordered    list OrderedDict fromkeys seq       nowplaying    PBS    job    debate    thenandnow       list dict fromkeys seq                                   py36     nowplaying    PBS    job    debate    thenandnow     The last option is recommended if using Python 3 6    See more details in this post   Note  listed elements must be hashable   See details on the latter example in this blog post   Furthermore  see R  Hettinger s post on the same technique  the order preserving dict is extended from one of his early implementations   See also more on total ordering

User · Answer

My solution to check contents for uniqueness but preserve the original order    def getUnique self       notunique   self readLines       unique          for line in notunique    Loop over content         append   True   Will be set to false if line matches existing line         for existing in unique              if line    existing    Line exists   do not append and go to the next line                 append   False                 break   Already know file is unique  break loop         if append  unique append line    Line not found  add to list     return unique   Edit  Probably can be more efficient by using dictionary keys to check for existence instead of doing a whole file loop for each line  I wouldn t use my solution for large sets

User · Answer

Try this function  it s similar to your code but it s a dynamic range   def unique a        k 0     while k  lt  len a           if a k  in a k 1                a pop k          else              k k 1        return a

User · Answer

For long arrays  s   np empty len var    s      np nan  for  x in  set var        x positions   np where var  x       s x positions 0  0   x   sorted var s  np isnan s

User · Answer

At the begin of your code just declare your output list as empty  output    Instead of your code you may use this code trends list set trends

User · Answer

def get distinct original list       distinct list          for each in original list          if each not in distinct list              distinct list append each      return distinct list

User · Answer

If we need to keep the elements order  how about this  used   set   mylist    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow   unique    x for x in mylist if x not in used and  used add x  or True    And one more solution using reduce and without the temporary used var  mylist    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow   unique   reduce lambda l  x  l append x  or l if x not in l else l  mylist       UPDATE - Dec  2020 - Maybe the best approach  Starting from python 3 7  the standard dict preserves insertion order   Changed in version 3 7  Dictionary order is guaranteed to be insertion order  This behavior was an implementation detail of CPython from 3 6   So this gives us the ability to use dict from keys for de-duplication  NOTE  Credits goes to  rlat for giving us this approach in the comments  mylist    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow   unique   list dict fromkeys mylist    In terms of speed - for me its fast enough and readable enough to become my new favorite approach  UPDATE - March  2019 And a 3rd solution  which is a neat one  but kind of slow since  index is O n   mylist    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow   unique    x for i  x in enumerate mylist  if i    mylist index x    UPDATE - Oct  2016 Another solution with reduce  but this time without  append which makes it more human readable and easier to understand  mylist    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow   unique   reduce lambda l  x  l  x  if x not in l else l  mylist       which can also be writed as  unique   reduce lambda l  x  l if x in l else l  x   mylist       NOTE  Have in mind that more human-readable we get  more unperformant the script is  Except only for the dict from keys approach which is python 3 7  specific  import timeit  setup    quot mylist    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow   quot    10x to Michael for pointing out that we can get faster with set   timeit timeit   x for x in mylist if x not in used and  used add x  or True     setup  used   set     setup  0 2029558869980974  timeit timeit   x for x in mylist if x not in used and  used append x  or True     setup  used        setup  0 28999493700030143    10x to rlat for suggesting this approach     timeit timeit  list dict fromkeys mylist     setup setup  0 31227896199925453  timeit timeit  reduce lambda l  x  l append x  or l if x not in l else l  mylist        setup  from functools import reduce   setup  0 7149233570016804  timeit timeit  reduce lambda l  x  l  x  if x not in l else l  mylist        setup  from functools import reduce   setup  0 7379565160008497  timeit timeit  reduce lambda l  x  l if x in l else l  x   mylist        setup  from functools import reduce   setup  0 7400134069976048  timeit timeit   x for i  x in enumerate mylist  if i    mylist index x     setup setup  0 9154880290006986  ANSWERING COMMENTS Because  monica asked a good question about  quot how is this working  quot   For everyone having problems figuring it out  I will try to give a more deep explanation about how this works and what sorcery is happening here    So she first asked   I try to understand why unique    used append x  for x in mylist if x not in used  is not working   Well it s actually working  gt  gt  gt  used       gt  gt  gt  mylist    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow    gt  gt  gt  unique    used append x  for x in mylist if x not in used   gt  gt  gt  print used  u nowplaying   u PBS   u job   u debate   u thenandnow    gt  gt  gt  print unique  None  None  None  None  None   The problem is that we are just not getting the desired results inside the unique variable  but only inside the used variable  This is because during the list comprehension  append modifies the used variable and returns None  So in order to get the results into the unique variable  and still use the same logic with  append x  if x not in used  we need to move this  append call on the right side of the list comprehension and just return x on the left side  But if we are too naive and just go with   gt  gt  gt  unique    x for x in mylist if x not in used and used append x    gt  gt  gt  print unique     We will get nothing in return  Again  this is because the  append method returns None  and it this gives on our logical expression the following look  x not in used and None  This will basically always   evaluates to False when x is in used  evaluates to None when x is not in used   And in both cases  False None   this will be treated as falsy value and we will get an empty list as a result  But why this evaluates to None when x is not in used  Someone may ask  Well it s because this is how Python s short-circuit operators works   The expression x and y first evaluates x  if x is false  its value is returned  otherwise  y is evaluated and the resulting value is returned   So when x is not in used  i e  when its True  the next part or the expression will be evaluated  used append x   and its value  None  will be returned  But that s what we want in order to get the unique elements from a list with duplicates  we want to  append them into a new list only when we they came across for a fist time  So we really want to evaluate used append x  only when x is not in used  maybe if there is a way to turn this None value into a truthy one we will be fine  right  Well  yes and here is where the 2nd type of short-circuit operators come to play   The expression x or y first evaluates x  if x is true  its value is returned  otherwise  y is evaluated and the resulting value is returned   We know that  append x  will always be falsy  so if we just add one or next to him  we will always get the next part  That s why we write  x not in used and  used append x  or True   so we can evaluate used append x  and get True as a result  only when the first part of the expression  x not in used  is True  Similar fashion can be seen in the 2nd approach with the reduce method   l append x  or l  if x not in l else l  similar as the above  but maybe more readable  we return l unchanged when x is in l  we append x to l and return l when x is not in l l if x in l else  l append x  or l   where we   Append x to l and return that l when x is not in l  Thanks to the or statement  append is evaluated and l is returned after that  Return l untouched when x is in l

User · Answer

In addition to the previous answers  which say you can convert your list to set  you can do that in this way too  mylist    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenadnow   mylist    i for i in set mylist     output will be   u nowplaying   u job   u debate   u PBS   u thenadnow     though order will not be preserved    Another simpler answer could be  without using sets    gt  gt  gt  t    v for i v in enumerate mylist  if mylist index v     i   u nowplaying   u PBS   u job   u debate   u thenadnow

User · Answer

By using basic property of Python Dictionary   inp  u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow   d  i for i in inp  print d   Output will be   set  u nowplaying   u job   u debate   u PBS   u thenandnow

User · Answer

I am surprised that nobody so far has given a direct order-preserving answer   def unique sequence          Generate unique items from sequence in the order of first occurrence         seen   set       for value in sequence          if value in seen              continue          seen add value           yield value   It will generate the values so it works with more than just lists  e g  unique range 10    To get a list  just call list unique sequence    like this    gt  gt  gt  list unique  u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow      u nowplaying   u PBS   u job   u debate   u thenandnow     It has the requirement that each item is hashable and not just comparable  but most stuff in Python is and it is O n  and not O n 2   so will work just fine with a long list

User · Answer

To get unique values from your list use code below   trends    u nowplaying   u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow   output   set trends  output   list output    IMPORTANT  Approach above won t work if any of items in a list is not hashable which is case for mutable types  for instance list or dict   trends      super  u nowplaying    u PBS   u PBS   u nowplaying   u job   u debate   u thenandnow   output   set trends  Traceback  most recent call last     File   lt stdin gt    line 1  in  lt module gt    TypeError  unhashable type   dict    That means that you have to be sure that trends list would always contains only hashable items otherwise you have to use more sophisticated code   from copy import deepcopy  try      trends      super  u nowplaying     u PBS      u PBS     u nowplaying   u job   u debate   u thenandnow     super  u nowplaying        output   set trends      output   list output  except TypeError      trends copy   deepcopy trends      while trends copy          trend   trends copy pop           if trends copy count trend     0              output append trend  print output

User · Answer

Maintaining order     oneliners   slow - gt    --- 14 417 seconds ---  x for i  x in enumerate array  if x not in array 0 i      fast - gt    --- 0 0378 seconds ---  x for i  x in enumerate array  if array index x     i     multiple lines   fastest - gt  --- 0 012 seconds --- uniq       uniq append x  for x in array if x not in uniq  uniq   Order doesn t matter        fastest-est - gt  --- 0 0035 seconds --- list set array

User · Answer

set - unordered collection of unique elements  List of elements can be passed to set s constructor  So  pass list with duplicate elements  we get set with unique elements and transform it back to list then get list with unique elements  I can say nothing about performance and memory overhead  but I hope  it s not so important with small lists   list set my not unique list     Simply and short

User · Answer

Use the following function   def uniquefy list input list       This function  takes a list as input and return a list containing only unique elements from the input list      output list    for elm123 in input list      in both lists 0     for elm234 in output list          if elm123    elm234              in both lists 1             break     if in both lists    0          output list append elm123   return output list

User · Answer

To be consistent with the type I would use   mylist   list set mylist

User · Answer

Getting unique elements from List  mylist    1 2 3 4 5 6 6 7 7 8 8 9 9 10       Using Simple Logic from Sets - Sets are unique list of items   mylist list set mylist    In  0   mylist Out 0    1  2  3  4  5  6  7  8  9  10       Using Simple Logic   newList    for i in mylist      if i not in newList          newList append i   In  0   mylist Out 0    1  2  3  4  5  6  7  8  9  10       Using pop method   - pop removes the last or indexed item and displays that to user  video   k 0 while k  lt  len mylist       if mylist k  in mylist k 1            mylist pop mylist k       else          k k 1  In  0   mylist Out 0    1  2  3  4  5  6  7  8  9  10       Using Numpy   import numpy as np np unique mylist   In  0   mylist Out 0    1  2  3  4  5  6  7  8  9  10    Reference

User · Answer

First thing  the example you gave is not a valid list   example list    u nowplaying  u PBS   u PBS   u nowplaying   u job   u debate  u thenandnow     Suppose if above is the example list  Then you can use the following recipe as give the itertools example doc  that can return the unique values and preserving the order as you seem to require  The iterable here is the example list   from itertools import ifilterfalse  def unique everseen iterable  key None        List unique elements  preserving order  Remember all elements ever seen         unique everseen  AAAABBBCCDAABBB   -- gt  A B C D       unique everseen  ABBCcAD   str lower  -- gt  A B C D     seen   set       seen add   seen add     if key is None          for element in ifilterfalse seen   contains    iterable               seen add element              yield element     else          for element in iterable              k   key element              if k not in seen                  seen add k                  yield element

User · Answer

Same order unique list using only a list compression    gt  my list    1  2  1  3  2  4  3  5  4  3  2  3  1   gt  unique list      gt     e  gt     for i  e in enumerate my list   gt     if my list index e     i  gt     gt  unique list  1  2  3  4  5    enumerates gives the index i and element e as a tuple   my list index returns the first index of e  If the first index isn t i then the current iteration s e is not the first e in the list   Edit  I should note that this isn t a good way to do it  performance-wise  This is just a way that achieves it using only a list compression

[python] Get unique values from a list in python

Examples related to python

Examples related to list