How to remove items from a list while iterating

Question

I m iterating over a list of tuples in Python  and am attempting to remove them if they meet certain criteria    for tup in somelist      if determine tup            code to remove tup   What should I use in place of code to remove tup  I can t figure out how to remove the item in this fashion

User · Answer

I can think of three approaches to solve your problem. As an example, I will create a random list of tuples somelist = [(1,2,3), (4,5,6), (3,6,6), (7,8,9), (15,0,0), (10,11,12)]. The condition that I choose is sum of elements of a tuple = 15. In the final list we will only have those tuples whose sum is not equal to 15.

What I have chosen is a randomly chosen example. Feel free to change the list of tuples and the condition that I have chosen.

Method 1.> Use the framework that you had suggested (where one fills in a code inside a for loop). I use a small code with del to delete a tuple that meets the said condition. However, this method will miss a tuple (which satisfies the said condition) if two consecutively placed tuples meet the given condition.

for tup in somelist:
    if ( sum(tup)==15 ): 
        del somelist[somelist.index(tup)]

print somelist
>>> [(1, 2, 3), (3, 6, 6), (7, 8, 9), (10, 11, 12)]

Method 2.> Construct a new list which contains elements (tuples) where the given condition is not met (this is the same thing as removing elements of list where the given condition is met). Following is the code for that:

newlist1 = [somelist[tup] for tup in range(len(somelist)) if(sum(somelist[tup])!=15)]

print newlist1
>>>[(1, 2, 3), (7, 8, 9), (10, 11, 12)]

Method 3.> Find indices where the given condition is met, and then use remove elements (tuples) corresponding to those indices. Following is the code for that.

indices = [i for i in range(len(somelist)) if(sum(somelist[i])==15)]
newlist2 = [tup for j, tup in enumerate(somelist) if j not in indices]

print newlist2
>>>[(1, 2, 3), (7, 8, 9), (10, 11, 12)]

Method 1 and method 2 are faster than method 3. Method2 and method3 are more efficient than method1. I prefer method2. For the aforementioned example, time(method1) : time(method2) : time(method3) = 1 : 1 : 1.7

User · Answer

One possible solution  useful if you want not only remove some things  but also do something with all elements in a single loop   alist     good    bad    good    bad    good   i   0 for x in alist         if x     bad           alist pop i          i -  1       do something cool with x or just print x     print x      i    1

User · Answer

If you want to delete elements from a list while iterating  use a while-loop so you can alter the current index and end index after each deletion   Example   i   0 length   len list1   while i  lt  length      if condition          list1 remove list1 i           i -  1         length -  1      i    1

User · Answer

It might be smart to also just create a new list if the current list item meets the desired criteria    so   for item in originalList     if  item    badValue           newList append item    and to avoid having to re-code the entire project with the new lists name   originalList      newList   note  from Python documentation       copy copy x     Return a shallow copy of x       copy deepcopy x     Return a deep copy of x

User · Answer

The other answers are correct that it is usually a bad idea to delete from a list that you re iterating  Reverse iterating avoids the pitfalls  but it is much more difficult to follow code that does that  so usually you re better off using a list comprehension or filter   There is  however  one case where it is safe to remove elements from a sequence that you are iterating  if you re only removing one item while you re iterating  This can be ensured using a return or a break  For example   for i  item in enumerate lst       if item   4    0          foo item          del lst i          break   This is often easier to understand than a list comprehension when you re doing some operations with side effects on the first item in a list that meets some condition and then removing that item from the list immediately after

User · Answer

uppose a list of number and you want to remove all no which are divisible by 3   list number   i for i in range 100     using list comprehension this will careate a new list and create new memory space  new list   i for i in list number if i 3  0    using lambda filter function  this will create resultant new list and consume memeory space  new list   list filter lambda x x 3  0  list number     without consuming memory space for new list and modify existing list  for index  value in enumerate list number       if list number index  3  0          list number remove value

User · Answer

Most of the answers here want you to create a copy of the list  I had a use case where the list was quite long  110K items  and it was smarter to keep reducing the list instead   First of all you ll need to replace foreach loop with while loop   i   0 while i  lt  len somelist       if determine somelist i             del somelist i      else          i    1   The value of i is not changed in the if block because you ll want to get value of the new item FROM THE SAME INDEX  once the old item is deleted

User · Answer

The most effective method is list comprehension  many people show their case  of course  it is also a good way to get an iterator through filter      Filter receives a function and a sequence  Filter applies the passed function to each element in turn  and then decides whether to retain or discard the element depending on whether the function return value is True or False    There is an example   get the odds in the tuple    list filter lambda x x 2  1   1  2  4  5  6  9  10  15        result   1  5  9  15    Caution  You can also not handle iterators  Iterators are sometimes better than sequences

User · Answer

In some situations  where you re doing more than simply filtering a list one item at time  you want your iteration to change while iterating   Here is an example where copying the list beforehand is incorrect  reverse iteration is impossible and a list comprehension is also not an option       Sieve of Eratosthenes      def generate primes n           Generates all primes less than n          primes   list range 2 n       idx   0     while idx  lt  len primes           p   primes idx          for multiple in range p p  n  p               try                  primes remove multiple              except ValueError                  pass  EAFP         idx    1         yield p

User · Answer

The answers suggesting list comprehensions are ALMOST correct -- except that they build a completely new list and then give it the same name the old list as  they do NOT modify the old list in place  That s different from what you d be doing by selective removal  as in  Lennart s suggestion -- it s faster  but if your list is accessed via multiple references the fact that you re just reseating one of the references and NOT altering the list object itself can lead to subtle  disastrous bugs   Fortunately  it s extremely easy to get both the speed of list comprehensions AND the required semantics of in-place alteration -- just code   somelist       tup for tup in somelist if determine tup     Note the subtle difference with other answers  this one is NOT assigning to a barename - it s assigning to a list slice that just happens to be the entire list  thereby replacing the list contents within the same Python list object  rather than just reseating one reference  from previous list object to new list object  like the other answers

User · Answer

You can use a list comprehension to create a new list containing only the elements you don t want to remove   somelist    x for x in somelist if not determine x     Or  by assigning to the slice somelist     you can mutate the existing list to contain only the items you want   somelist       x for x in somelist if not determine x     This approach could be useful if there are other references to somelist that need to reflect the changes   Instead of a comprehension  you could also use itertools  In Python 2   from itertools import ifilterfalse somelist      ifilterfalse determine  somelist    Or in Python 3   from itertools import filterfalse somelist      filterfalse determine  somelist    For the sake of clarity and for those who find the use of the     notation hackish or fuzzy  here s a more explicit alternative  Theoretically  it should perform the same with regards to space and time than the one-liners above   temp      while somelist      x   somelist pop       if not determine x           temp append x  while temp      somelist append templist pop      It also works in other languages that may not have the replace items ability of Python lists  with minimal modifications  For instance  not all languages cast empty lists to a False as Python does  You can substitute while somelist  for something more explicit like while len somelist   gt  0

User · Answer

This answer was originally written in response to a question which has since been marked as duplicate  Removing coordinates from list on python  There are two problems in your code   1  When using remove    you attempt to remove integers whereas you need to remove a tuple   2  The for loop will skip items in your list   Let s run through what happens when we execute your code    gt  gt  gt  L1     1 2    5 6    -1 -2    1 -2    gt  gt  gt  for  a b  in L1        if a  lt  0 or b  lt  0          L1 remove a b       Traceback  most recent call last     File   lt stdin gt    line 3  in  lt module gt  TypeError  remove   takes exactly one argument  2 given    The first problem is that you are passing both  a  and  b  to remove    but remove   only accepts a single argument  So how can we get remove   to work properly with your list  We need to figure out what each element of your list is  In this case  each one is a tuple  To see this  let s access one element of the list  indexing starts at 0     gt  gt  gt  L1 1   5  6   gt  gt  gt  type L1 1    lt type  tuple  gt    Aha  Each element of L1 is actually a tuple  So that s what we need to be passing to remove    Tuples in python are very easy  they re simply made by enclosing values in parentheses   a  b  is not a tuple  but   a  b   is a tuple  So we modify your code and run it again     The remove line now includes an extra      to make a tuple out of  a b  L1 remove  a b     This code runs without any error  but let s look at the list it outputs    L1 is now    1  2    5  6    1  -2     Why is  1 -2  still in your list  It turns out modifying the list while using a loop to iterate over it is a very bad idea without special care  The reason that  1  -2  remains in the list is that the locations of each item within the list changed between iterations of the for loop  Let s look at what happens if we feed the above code a longer list   L1     1 2   5 6   -1 -2   1 -2   3 4   5 7   -4 4   2 1   -3 -3   5 -1   0 6       Outputs  L1 is now    1  2    5  6    1  -2    3  4    5  7    2  1    5  -1    0  6     As you can infer from that result  every time that the conditional statement evaluates to true and a list item is removed  the next iteration of the loop will skip evaluation of the next item in the list because its values are now located at different indices   The most intuitive solution is to copy the list  then iterate over the original list and only modify the copy  You can try doing so like this   L2   L1 for  a b  in L1      if a  lt  0 or b  lt  0           L2 remove  a b     Now  remove the original copy of L1 and replace with L2 print L2 is L1 del L1 L1   L2  del L2 print   L1 is now     L1    However  the output will be identical to before    L1 is now       1  2    5  6    1  -2    3  4    5  7    2  1    5  -1    0  6     This is because when we created L2  python did not actually create a new object  Instead  it merely referenced L2 to the same object as L1  We can verify this with  is  which is different from merely  equals          gt  gt  gt  L2 L1  gt  gt  gt  L1 is L2 True   We can make a true copy using copy copy    Then everything works as expected   import copy L1     1 2    5 6   -1 -2    1 -2   3 4   5 7   -4 4   2 1   -3 -3   5 -1   0 6   L2   copy copy L1  for  a b  in L1      if a  lt  0 or b  lt  0           L2 remove  a b     Now  remove the original copy of L1 and replace with L2 del L1 L1   L2  del L2  gt  gt  gt  L1 is now    1  2    5  6    3  4    5  7    2  1    0  6     Finally  there is one cleaner solution than having to make an entirely new copy of L1  The reversed   function   L1     1 2    5 6   -1 -2    1 -2   3 4   5 7   -4 4   2 1   -3 -3   5 -1   0 6   for  a b  in reversed L1       if a  lt  0 or b  lt  0           L1 remove  a b   print   L1 is now     L1   gt  gt  gt  L1 is now    1  2    5  6    3  4    5  7    2  1    0  6     Unfortunately  I cannot adequately describe how reversed   works  It returns a  listreverseiterator  object when a list is passed to it  For practical purposes  you can think of it as creating a reversed copy of its argument  This is the solution I recommend

User · Answer

for i in range len somelist  - 1  -1  -1       if some condition somelist  i           del somelist i    You need to go backwards otherwise it s a bit like sawing off the tree-branch that you are sitting on  -   Python 2 users  replace range by xrange to avoid creating a hardcoded list

User · Answer

You need to take a copy of the list and iterate over it first  or the iteration will fail with what may be unexpected results   For example  depends on what type of list    for tup in somelist         etc       An example    gt  gt  gt  somelist   range 10   gt  gt  gt  for x in somelist          somelist remove x   gt  gt  gt  somelist  1  3  5  7  9    gt  gt  gt  somelist   range 10   gt  gt  gt  for x in somelist             somelist remove x   gt  gt  gt  somelist

User · Answer

You might want to use filter   available as the built-in   For more details check here

User · Answer

For anything that has the potential to be really big  I use the following    import numpy as np  orig list   np array  1  2  3  4  5  100  8  13    remove me    100  1   cleaned   np delete orig list  remove me  print cleaned    That should be significantly faster than anything else

User · Answer

I needed to do this with a huge list  and duplicating the list seemed expensive  especially since in my case the number of deletions would be few compared to the items that remain  I took this low-level approach   array    lots of stuff  arraySize   len array  i   0 while i  lt  arraySize      if someTest array i            del array i          arraySize -  1     else          i    1   What I don t know is how efficient a couple of deletes are compared to copying a large list  Please comment if you have any insight

User · Answer

for loop will be iterate through index    consider you have a list    5  7  13  29  65  91    you have using list variable called lis  and you using same to remove    your variable   lis    5  7  13  29  35  65  91         0  1   2   3   4   5   6   during 5th iteration   your number 35 was not a prime so you removed it from a list   lis remove y    and then next value  65  move on to previous index   lis    5  7  13  29  65  91         0  1   2   3   4   5   so 4th iteration done pointer moved onto 5th     thats why your loop doesnt cover 65 since its moved into previous index   so you shouldn t reference list into another variable which still reference original instead of copy   ite   lis  dont do it will reference instead copy   so do copy of list using list      now you it will give    5  7  13  29    Problem is you removed a value from a list during iteration then your list index will collapse   so you can try comprehension instead   which supports all the iterable like  list  tuple  dict  string etc

User · Answer

Your best approach for such an example would be a list comprehension  somelist    tup for tup in somelist if determine tup     In cases where you re doing something more complex than calling a determine function  I prefer constructing a new list and simply appending to it as I go   For example  newlist      for tup in somelist        lots of code here  possibly setting things up for calling determine     if determine tup           newlist append tup  somelist   newlist   Copying the list using remove might make your code look a little cleaner  as described in one of the answers below   You should definitely not do this for extremely large lists  since this involves first copying the entire list  and also performing an O n  remove operation for each element being removed  making this an O n 2  algorithm   for tup in somelist           lots of code here  possibly setting things up for calling determine     if determine tup           newlist append tup

User · Answer

You can try for-looping in reverse so for some list you ll do something like   list len   len some list  for i in range list len       reverse i   list len - 1 - i     cur   some list reverse i         some logic with cur element      if some condition          some list pop reverse i    This way the index is aligned and doesn t suffer from the list updates  regardless whether you pop cur element or not

User · Answer

If you want to do anything else during the iteration  it may be nice to get both the index  which guarantees you being able to reference it  for example if you have a list of dicts  and the actual list item contents   inlist      field1  10   field2  20     field1  30   field2  15       for idx  i in enumerate inlist       do some stuff with i  field1       if somecondition          xlist append idx  for i in reversed xlist   del inlist i    enumerate gives you access to the item and the index at once  reversed is so that the indices that you re going to later delete don t change on you

User · Answer

TLDR   I wrote a library that allows you to do this   from fluidIter import FluidIterable fSomeList   FluidIterable someList    for tup in fSomeList      if determine tup             remove  tup  without  breaking  the iteration         fSomeList remove tup            tup has also been removed from  someList            as well as  fSomeList    It s best to use another method if possible that doesn t require modifying your iterable while iterating over it  but for some algorithms it might not be that straight forward  And so if you are sure that you really do want the code pattern described in the original question  it is possible   Should work on all mutable sequences not just lists     Full answer   Edit  The last code example in this answer gives a use case for why you might sometimes want to modify a list in place rather than use a list comprehension  The first part of the answers serves as tutorial of how an array can be modified in place   The solution follows on from this answer  for a related question  from senderle  Which explains how the the array index is updated while iterating through a list that has been modified  The solution below is designed to correctly track the array index even if the list is modified   Download fluidIter py from here https   github com alanbacon FluidIterator  it is just a single file so no need to install git  There is no installer so you will need to make sure that the file is in the python path your self  The code has been written for python 3 and is untested on python 2   from fluidIter import FluidIterable l    0 1 2 3 4 5 6 7 8    fluidL   FluidIterable l                         for i in fluidL      print  initial state of list on this iteration      str fluidL        print  current iteration value      str i       print  popped value      str fluidL pop 2        print       print  Final List Value      str l     This will produce the following output   initial state of list on this iteration   0  1  2  3  4  5  6  7  8  current iteration value  0 popped value  2  initial state of list on this iteration   0  1  3  4  5  6  7  8  current iteration value  1 popped value  3  initial state of list on this iteration   0  1  4  5  6  7  8  current iteration value  4 popped value  4  initial state of list on this iteration   0  1  5  6  7  8  current iteration value  5 popped value  5  initial state of list on this iteration   0  1  6  7  8  current iteration value  6 popped value  6  initial state of list on this iteration   0  1  7  8  current iteration value  7 popped value  7  initial state of list on this iteration   0  1  8  current iteration value  8 popped value  8  Final List Value   0  1    Above we have used the pop method on the fluid list object  Other common iterable methods are also implemented such as del fluidL i    remove   insert   append   extend  The list can also be modified using slices  sort and reverse methods are not implemented    The only condition is that you must only modify the list in place  if at any point fluidL or l were reassigned to a different list object the code would not work  The original fluidL object would still be used by the for loop but would become out of scope for us to modify   i e   fluidL 2     a      is OK fluidL    0  1   a   3  4  5  6  7  8     is not OK   If we want to access the current index value of the list we cannot use enumerate  as this only counts how many times the for loop has run  Instead we will use the iterator object directly   fluidArr   FluidIterable  0 1 2 3     get iterator first so can query the current index fluidArrIter   fluidArr   iter     for i  v in enumerate fluidArrIter       print  enum     i      print  current val     v      print  current ind     fluidArrIter currentIndex      print fluidArr      fluidArr insert 0  a       print       print  Final List Value      str fluidArr     This will output the following   enum   0 current val   0 current ind   0  0  1  2  3   enum   1 current val   1 current ind   2   a   0  1  2  3   enum   2 current val   2 current ind   4   a    a   0  1  2  3   enum   3 current val   3 current ind   6   a    a    a   0  1  2  3   Final List Value    a    a    a    a   0  1  2  3    The FluidIterable class just provides a wrapper for the original list object  The original object can be accessed as a property of the fluid object like so   originalList   fluidArr fixedIterable   More examples   tests can be found in the if   name   is    main     section at the bottom of fluidIter py  These are worth looking at because they explain what happens in various situations  Such as  Replacing a large sections of the list using a slice  Or using  and modifying  the same iterable in nested for loops   As I stated to start with  this is a complicated solution that will hurt the readability of your code and make it more difficult to debug  Therefore other solutions such as the list comprehensions mentioned in David Raznick s answer should be considered first  That being said  I have found times where this class has been useful to me and has been easier to use than keeping track of the indices of elements that need deleting     Edit  As mentioned in the comments  this answer does not really present a problem for which this approach provides a solution  I will try to address that here   List comprehensions provide a way to generate a new list but these approaches tend to look at each element in isolation rather than the current state of the list as a whole   i e   newList    i for i in oldList if testFunc i     But what if the result of the testFunc depends on the elements that have been added to newList already  Or the elements still in oldList that might be added next  There might still be a way to use a list comprehension but it will begin to lose it s elegance  and for me it feels easier to modify a list in place   The code below is one example of an algorithm that suffers from the above problem  The algorithm will reduce a list so that no element is a multiple of any other element   randInts    70  20  61  80  54  18  7  18  55  9  fRandInts   FluidIterable randInts  fRandIntsIter   fRandInts   iter       for each value in the list  outer loop    test against every other value in the list  inner loop  for i in fRandIntsIter      print          print  outer val     i      innerIntsIter   fRandInts   iter         for j in innerIntsIter          innerIndex   innerIntsIter currentIndex           skip the element that the outloop is currently on           because we don t want to test a value against itself         if not innerIndex    fRandIntsIter currentIndex                if the test element  j  is a multiple                of the reference element  i  then remove  j              if j i    0                  print  remove val     j                    remove element in place  without breaking the                   iteration of either loop                 del fRandInts innerIndex                end if multiple  then remove           end if not the same value as outer loop       end inner loop   end outerloop  print     print  final list     randInts    The output and the final reduced list are shown below  outer val   70  outer val   20 remove val   80  outer val   61  outer val   54  outer val   18 remove val   54 remove val   18  outer val   7 remove val   70  outer val   55  outer val   9 remove val   18  final list    20  61  7  55  9

User · Answer

I needed to do something similar and in my case the problem was memory - I needed to merge multiple dataset objects within a list  after doing some stuff with them  as a new object  and needed to get rid of each entry I was merging to avoid duplicating all of them and blowing up memory  In my case having the objects in a dictionary instead of a list worked fine        k   range 5  v     a   b   c   d   e   d    key val for key val in zip k  v    print d for i in range 5       print d i      d pop i  print d

User · Answer

For those that like functional programming   somelist      filter lambda tup  not determine tup   somelist    or  from itertools import ifilterfalse somelist      list ifilterfalse determine  somelist

User · Answer

If you will use the new list later  you can simply set the elem to None  and then judge it in the later loop  like this  for i in li      i   None  for elem in li      if elem is None          continue   In this way  you dont t need copy the list and it s easier to understand

User · Answer

Overview of workarounds Either   use a linked list implementation roll your own  A linked list is the proper data structure to support efficient item removal  and does not force you to make space time tradeoffs  A CPython list is implemented with dynamic arrays as mentioned here  which is not a good data type to support removals  There doesn t seem to be a linked list in the standard library however   Is there a linked list predefined library in Python  https   github com ajakubek python-llist   start a new list   from scratch  and  append   back at the end as mentioned at  https   stackoverflow com a 1207460 895245 This time efficient  but less space efficient because it keeps an extra copy of the array around during iteration   use del with an index as mentioned at  https   stackoverflow com a 1207485 895245 This is more space efficient since it dispenses the array copy  but it is less time efficient  because removal from dynamic arrays requires shifting all following items back by one  which is O N     Generally  if you are doing it quick and dirty and don t want to add a custom LinkedList class  you just want to go for the faster  append   option by default unless memory is a big concern  Official Python 2 tutorial 4 2   quot for Statements quot  https   docs python org 2 tutorial controlflow html for-statements This part of the docs makes it clear that   you need to make a copy of the iterated list to modify it one way to do it is with the slice notation       If you need to modify the sequence you are iterating over while inside the loop  for example to duplicate selected items   it is recommended that you first make a copy  Iterating over a sequence does not implicitly make a copy  The slice notation makes this especially convenient   gt  gt  gt  words     cat    window    defenestrate    gt  gt  gt  for w in words        Loop over a slice copy of the entire list          if len w   gt  6              words insert 0  w       gt  gt  gt  words   defenestrate    cat    window    defenestrate     Python 2 documentation 7 3   quot The for statement quot  https   docs python org 2 reference compound stmts html for This part of the docs says once again that you have to make a copy  and gives an actual removal example   Note  There is a subtlety when the sequence is being modified by the loop  this can only occur for mutable sequences  i e  lists   An internal counter is used to keep track of which item is used next  and this is incremented on each iteration  When this counter has reached the length of the sequence the loop terminates  This means that if the suite deletes the current  or a previous  item from the sequence  the next item will be skipped  since it gets the index of the current item which has already been treated   Likewise  if the suite inserts an item in the sequence before the current item  the current item will be treated again the next time through the loop  This can lead to nasty bugs that can be avoided by making a temporary copy using a slice of the whole sequence  e g   for x in a           if x  lt  0  a remove x   However  I disagree with this implementation  since  remove   has to iterate the entire list to find the value  Could Python do this better  It seems like this particular Python API could be improved  Compare it  for instance  with   Java ListIterator  remove which documents  quot This call can only be made once per call to next or previous quot  C   std  vector  erase which returns a valid interator to the element after the one removed  both of which make it crystal clear that you cannot modify a list being iterated except with the iterator itself  and gives you efficient ways to do so without copying the list  Perhaps the underlying rationale is that Python lists are assumed to be dynamic array backed  and therefore any type of removal will be time inefficient anyways  while Java has a nicer interface hierarchy with both ArrayList and LinkedList implementations of ListIterator  There doesn t seem to be an explicit linked list type in the Python stdlib either  Python Linked List

[python] How to remove items from a list while iterating?

Examples related to python

Examples related to iteration