By lfaraone


2009-07-30 15:36:42 8 Comments

I'm iterating over a list of tuples in Python, and am attempting to remove them if they meet certain criteria.

for tup in somelist:
    if determine(tup):
         code_to_remove_tup

What should I use in place of code_to_remove_tup? I can't figure out how to remove the item in this fashion.

30 comments

@Algebra 2019-12-04 06:03:23

The problem is that list is of mutable object

#+begin_src ipython :session alinbx :results output
mylist = list(range(8))
idx = 0
for i in mylist:
    print(f"i={i}, idx={idx}, mylist{[idx]}={mylist[idx]}, {mylist}")
    mylist.remove(i)
    idx += 1

#+end_src

#+RESULTS:
: i=0, idx=0, mylist[0]=0, [0, 1, 2, 3, 4, 5, 6, 7]
: i=2, idx=1, mylist[1]=2, [1, 2, 3, 4, 5, 6, 7]
: i=4, idx=2, mylist[2]=4, [1, 3, 4, 5, 6, 7]
: i=6, idx=3, mylist[3]=6, [1, 3, 5, 6, 7]

As the idx step forwards, mylist shrinks as well.

#+begin_src ipython :session alinbx :results output
mylist = list(range(8))
idx = 0
for i, e in enumerate(mylist):
    print(i, e)
    mylist.remove(e)

#+end_src

#+RESULTS:
: 0 0
: 1 2
: 2 4
: 3 6

@Lennart Regebro 2009-07-30 15:38:32

You need to take a copy of the list and iterate over it first, or the iteration will fail with what may be unexpected results.

For example (depends on what type of list):

for tup in somelist[:]:
    etc....

An example:

>>> somelist = range(10)
>>> for x in somelist:
...     somelist.remove(x)
>>> somelist
[1, 3, 5, 7, 9]

>>> somelist = range(10)
>>> for x in somelist[:]:
...     somelist.remove(x)
>>> somelist
[]

@jdero 2013-08-09 18:55:30

Zen #3, Simple is better than complex. Gets my vote!

@Lennart Regebro 2014-06-18 13:47:31

@Zen Because the second one iterates over a copy of the list. So when you modify the original list, you do not modify the copy that you iterate over.

@Zen 2014-06-18 13:51:05

@LennartRegebro, oh my god, that's a genius idea, that list copy will automatically disappeared after for loop executed?

@Lennart Regebro 2014-06-19 08:14:17

@Zen it will get garbage collected, yes.

@fantabolous 2014-08-18 09:11:22

Correct me if I'm wrong but as this is not using the index but rather whatever's actually stored in the list to do the remove, it won't work for more complex items such as dicts, etc.

@Lennart Regebro 2014-08-19 16:16:14

@fantabolous: Dicts doesn't have an index, so it wouldn't work anyway.

@fantabolous 2014-08-20 01:47:44

@LennartRegebro, I meant lists of dicts, but actually after testing it looks like I was wrong anyway - sorry.

@Mariusz Jamro 2015-02-04 10:01:22

What's better in doing somelist[:] compared to list(somelist) ?

@Lennart Regebro 2015-02-05 12:21:22

list(somelist) will convert an iterable into a list. somelist[:] makes a copy of an object that supports slicing. So they don't necessarily do the same thing. In this case I want to make a copy of the somelistobject, so I use [:]

@vitiral 2015-02-11 23:22:14

Note to anyone reading this, this is VERY slow for lists. remove() has to go over the WHOLE list for every iteration, so it will take forever.

@Ciro Santilli 新疆改造中心法轮功六四事件 2015-12-12 10:18:52

This is the method mentioned in the official tutorial: stackoverflow.com/a/34238688/895245

@vitiral 2016-02-09 16:23:00

@navin and twice the memory!

@User 2016-05-03 16:40:57

it's nice to know the alter syntax, given it's given in documentation also. But please also mention in your answer it's more time and memory consuming.

@Steve 2016-08-06 17:21:20

Big O time doesn't matter when dealing with lists of only a dozen items. Often clear and simple for future programmers to understand is far more valuable than performance.

@miorey 2019-11-29 15:04:21

I think that using while and un-increment when deleting an element is the simplest way.

import random

def f(l, n):
    # n shall not pass
    idx = 0
    while idx < len(l):
        if l[idx] % 2 == n:
            del l[idx]
            continue
        idx += 1


res = [random.randrange(1, 50) for i in range(10000)]
f(res, 0)
f(res, 1)

At the end res should be [].

@chrisaramar 2019-11-20 10:02:33

Hi I had following problem:

I wanted to iterate over 2 lists and remove the obj if it is in both lists. Also in the end there should be still 2 lists which cointain the difference of both lists.

Then I had the same index problem as in this question.

I think I found a nice solution for solving the index problem and wanted to share it with you.

def get_diff(list1,list2):

    index = 0
    for obj in list1[index:]:
        if obj in list2:
            list1.remove(obj)
            list2.remove(obj)
        else:
            index += 1


    return list_rep_vols,list_aix_vols

I am defining the index as a var and tell list1 to start the iterating from the index to the end of list1. So technically im telling him that my index 1 is now index 0 if I remove index 0.

Here an example:

test = ["a", "z", "b", "f", "d"]
test1 = ["a", "b", "c", "e"]

test_re, test1_re = get_del_and_add_list_aix(test,test1)

print(test_re)
print(test1_re)

Output:

test ['z', 'f', 'd']

test1 ['c', 'e']

Regards

chrisaramar

@NoName 2019-10-10 02:24:59

If you want to delete elements from a list while iterating, use a while-loop so you can alter the current index and end index after each deletion.

Example:

i = 0
length = len(list1)

while i < length:
    if condition:
        list1.remove(list1[i])
        i -= 1
        length -= 1

    i += 1

@Ciro Santilli 新疆改造中心法轮功六四事件 2015-12-12 10:18:22

Official Python 2 tutorial 4.2. "for Statements"

https://docs.python.org/2/tutorial/controlflow.html#for-statements

This part of the docs makes it clear that:

  • you need to make a copy of the iterated list to modify it
  • one way to do it is with the slice notation [:]

If you need to modify the sequence you are iterating over while inside the loop (for example to duplicate selected items), it is recommended that you first make a copy. Iterating over a sequence does not implicitly make a copy. The slice notation makes this especially convenient:

>>> words = ['cat', 'window', 'defenestrate']
>>> for w in words[:]:  # Loop over a slice copy of the entire list.
...     if len(w) > 6:
...         words.insert(0, w)
...
>>> words
['defenestrate', 'cat', 'window', 'defenestrate']

Python 2 documentation 7.3. "The for statement"

https://docs.python.org/2/reference/compound_stmts.html#for

This part of the docs says once again that you have to make a copy, and gives an actual removal example:

Note: There is a subtlety when the sequence is being modified by the loop (this can only occur for mutable sequences, i.e. lists). An internal counter is used to keep track of which item is used next, and this is incremented on each iteration. When this counter has reached the length of the sequence the loop terminates. This means that if the suite deletes the current (or a previous) item from the sequence, the next item will be skipped (since it gets the index of the current item which has already been treated). Likewise, if the suite inserts an item in the sequence before the current item, the current item will be treated again the next time through the loop. This can lead to nasty bugs that can be avoided by making a temporary copy using a slice of the whole sequence, e.g.,

for x in a[:]:
    if x < 0: a.remove(x)

However, I disagree with this implementation, since .remove() has to iterate the entire list to find the value.

Best workarounds

Either:

Generally you just want to go for the faster .append() option by default unless memory is a big concern.

Could Python do this better?

It seems like this particular Python API could be improved. Compare it, for instance, with its Java counterpart ListIterator, which makes it crystal clear that you cannot modify a list being iterated except with the iterator itself, and gives you efficient ways to do so without copying the list.

Perhaps the underlying rationale is that Python lists are assumed to be dynamic array backed, and therefore any type of removal will be time inefficient anyways, while Java has a nicer interface hierarchy with both ArrayList and LinkedList implementations of ListIterator.

There doesn't seem to be an explicit linked list type in the Python stdlib either: Python Linked List

@temmo 2019-05-29 05:02:40

uppose a list of number and you want to remove all no which are divisible by 3,

list_number =[i for i in range(100)]

using list comprehension,this will careate a new list and create new memory space

new_list =[i for i in list_number if i%3!=0]

using lambda filter function, this will create resultant new list and consume memeory space

new_list = list(filter(lambda x:x%3!=0, list_number))

without consuming memory space for new list and modify existing list

for index, value in enumerate(list_number):
    if list_number[index]%3==0:
        list_number.remove(value)

@Alex Martelli 2009-07-30 19:28:27

The answers suggesting list comprehensions are ALMOST correct -- except that they build a completely new list and then give it the same name the old list as, they do NOT modify the old list in place. That's different from what you'd be doing by selective removal, as in @Lennart's suggestion -- it's faster, but if your list is accessed via multiple references the fact that you're just reseating one of the references and NOT altering the list object itself can lead to subtle, disastrous bugs.

Fortunately, it's extremely easy to get both the speed of list comprehensions AND the required semantics of in-place alteration -- just code:

somelist[:] = [tup for tup in somelist if determine(tup)]

Note the subtle difference with other answers: this one is NOT assigning to a barename - it's assigning to a list slice that just happens to be the entire list, thereby replacing the list contents within the same Python list object, rather than just reseating one reference (from previous list object to new list object) like the other answers.

@PaulMcG 2011-03-25 19:29:37

How do I do the same sliced assignment with a dict? In Python 2.6?

@Sven Marnach 2011-04-01 23:51:47

@Paul: Since dicts are unordered, slices are meaningless for dicts. If your want to replace the contents of dict a by the contents of dict b, use a.clear(); a.update(b).

@Derek Dahmer 2011-08-07 22:59:53

Why can 'reseating' one of the references by replacing what the variable refers to cause bugs? It seems like that would only be a potential problem in multi-threaded applications, not single-threaded.

@Steven T. Snyder 2011-11-15 19:38:42

@Derek x = ['foo','bar','baz']; y = x; x = [item for item in x if determine(item)]; This reassigns x to the result of the list comprehension, but y still refers to the original list ['foo','bar','baz']. If you expected x and y to refer to the same list, you may have introduced bugs. You prevent this by assigning to a slice of the entire list, as Alex shows, and I show here: x = ["foo","bar","baz"]; y = x; x[:] = [item for item in x if determine(item)];. The list is modified in place. ensuring that all references to the list (both x and y here) refer to the new list.

@John Strood 2016-07-13 08:48:01

in fact, using filter function too creates a new list, does not modify elements in place... only olist[:] = [i for i in olist if not dislike(i)]

@Dor 2016-11-29 11:01:36

It doesn't work , see (it skips the number 4): ` l = range(1, 6) for x in l: print x if x == 3: l = [i for i in l if i != x] `

@ShadowRanger 2018-02-14 01:58:28

@Dor: That doesn't work because you're iterating l and mutating it inside the loop, and on top of that, as written, you're rebinding l instead of slice assigning (but the loop is over the iterator over the old l, so the outer loop wouldn't change). You're supposed to replace the explicit loop with a listcomp (that isn't inside a loop at all), e.g. for your case, you'd do a print loop, then when it's done, strip the "bad" value. You might have for x in l: print x, then outside and below the for loop's block, you'd do l[:] = [i for i in l if i != 3] to remove the 3s.

@Mohideen bin Mohammed 2019-04-30 06:25:49

for loop will be iterate through index..

consider you have a list,

[5, 7, 13, 29, 65, 91]

you have using list variable called lis. and you using same to remove..

your variable

lis = [5, 7, 13, 29, 35, 65, 91]
       0  1   2   3   4   5   6

during 5th iteration,

your number 35 was not a prime so you removed it from a list.

lis.remove(y)

and then next value (65) move on to previous index.

lis = [5, 7, 13, 29, 65, 91]
       0  1   2   3   4   5

so 4th iteration done pointer moved onto 5th..

thats why your loop doesnt cover 65 since its moved into previous index.

so you shouldn't reference list into another variable which still reference original instead of copy.

ite = lis #dont do it will reference instead copy

so do copy of list using list[::]

now you it will give,

[5, 7, 13, 29]

Problem is you removed a value from a list during iteration then your list index will collapse.

so you can try comprehension instead.

which supports all the iterable like, list, tuple, dict, string etc

@john zhang 2019-04-16 09:08:13

If you will use the new list later, you can simply set the elem to None, and then judge it in the later loop, like this

for i in li:
    i = None

for elem in li:
    if elem is None:
        continue

In this way, you dont't need copy the list and it's easier to understand.

@Siddharth Satpathy 2018-12-03 19:41:17

I can think of three approaches to solve your problem. As an example, I will create a random list of tuples somelist = [(1,2,3), (4,5,6), (3,6,6), (7,8,9), (15,0,0), (10,11,12)]. The condition that I choose is sum of elements of a tuple = 15. In the final list we will only have those tuples whose sum is not equal to 15.

What I have chosen is a randomly chosen example. Feel free to change the list of tuples and the condition that I have chosen.

Method 1.> Use the framework that you had suggested (where one fills in a code inside a for loop). I use a small code with del to delete a tuple that meets the said condition. However, this method will miss a tuple (which satisfies the said condition) if two consecutively placed tuples meet the given condition.

for tup in somelist:
    if ( sum(tup)==15 ): 
        del somelist[somelist.index(tup)]

print somelist
>>> [(1, 2, 3), (3, 6, 6), (7, 8, 9), (10, 11, 12)]

Method 2.> Construct a new list which contains elements (tuples) where the given condition is not met (this is the same thing as removing elements of list where the given condition is met). Following is the code for that:

newlist1 = [somelist[tup] for tup in range(len(somelist)) if(sum(somelist[tup])!=15)]

print newlist1
>>>[(1, 2, 3), (7, 8, 9), (10, 11, 12)]

Method 3.> Find indices where the given condition is met, and then use remove elements (tuples) corresponding to those indices. Following is the code for that.

indices = [i for i in range(len(somelist)) if(sum(somelist[i])==15)]
newlist2 = [tup for j, tup in enumerate(somelist) if j not in indices]

print newlist2
>>>[(1, 2, 3), (7, 8, 9), (10, 11, 12)]

Method 1 and method 2 are faster than method 3. Method2 and method3 are more efficient than method1. I prefer method2. For the aforementioned example, time(method1) : time(method2) : time(method3) = 1 : 1 : 1.7

@chseng 2018-11-10 07:05:16

The most effective method is list comprehension, many people show their case, of course, it is also a good way to get an iterator through filter.

Filter receives a function and a sequence. Filter applies the passed function to each element in turn, and then decides whether to retain or discard the element depending on whether the function return value is True or False.

There is an example (get the odds in the tuple):

list(filter(lambda x:x%2==1, (1, 2, 4, 5, 6, 9, 10, 15)))  
# result: [1, 5, 9, 15]

Caution: You can also not handle iterators. Iterators are sometimes better than sequences.

@John Machin 2009-07-30 15:44:59

for i in range(len(somelist) - 1, -1, -1):
    if some_condition(somelist, i):
        del somelist[i]

You need to go backwards otherwise it's a bit like sawing off the tree-branch that you are sitting on :-)

Python 2 users: replace range by xrange to avoid creating a hardcoded list

@ncoghlan 2011-03-23 07:08:43

In recent versions of Python, you can do this even more cleanly by using the reversed() builtin

@drevicko 2014-11-02 04:11:21

Just a note that in python 2 you can do for i,v in enumerate(reversed(somelist)):, but to get at the indices you need to store the original lenght. The index of element v is originalLength - i - 1.

@vitiral 2015-02-11 23:24:50

@ncoghlan don't use reversed, that creates an entirely new list for no reason. If you want to do it that way, use itertools.islice(somelist, None, None, -1) (not tested, but should work)

@ncoghlan 2015-02-12 06:44:12

reversed() does not create a new list, it creates a reverse iterator over the supplied sequence. Like enumerate(), you have to wrap it in list() to actually get a list out of it. You may be thinking of sorted(), which does create a new list every time (it has to, so it can sort it).

@Lynn 2015-07-31 11:58:23

@drevicko Why not reversed(enumerate(somelist)) and get the right indices immediately?

@drevicko 2015-08-02 23:27:15

@Mauris because enumerate returns an iterator and reversed expects a sequence. I guess you could do reversed(list(enumerate(somelist))) if you don't mind creating an extra list in memory.

@Sam Watkins 2015-09-15 15:04:59

This is O(N*M) for arrays, it is very slow if you remove many items from a large list. So not recommended.

@Navin 2016-02-07 21:36:43

@SamWatkins Yeah, this answer is for when you're removing a couple of elements from a very large array. Less memory usage, but it can be m times slower.

@Barmaley 2017-05-05 02:14:03

Love this answer and tree-branch cutting analogy! List comprehension would work as well as long as you don't have to do anything complex. The only comment, I would use reversed(xrange(len(somelist))) in python2 and reversed(range(len(somelist))) in python3

@Anh Tuan 2018-03-08 07:39:31

I like this method, it is just like remove element in Java ArrayList. Other answers keep talking about iteration over a shallow copy of list, which is quite illogical solution for a simple thing like this

@cardamom 2018-11-28 14:54:49

First I have run up against this, strange but going through it backwards does fix it. Also works with pop if you need to retain the deleted element. From the comments below, reversed(list(enumerate(some_list))) can make it a bit neater.

@Mujeeb 2018-10-23 11:13:00

Most of the answers here want you to create a copy of the list. I had a use case where the list was quite long (110K items) and it was smarter to keep reducing the list instead.

First of all you'll need to replace foreach loop with while loop,

i = 0
while i < len(somelist):
    if determine(somelist[i]):
         del somelist[i]
    else:
        i += 1

The value of i is not changed in the if block because you'll want to get value of the new item FROM THE SAME INDEX, once the old item is deleted.

@CodeKid 2018-09-21 16:14:52

In some situations, where you're doing more than simply filtering a list one item at time, you want your iteration to change while iterating.

Here is an example where copying the list beforehand is incorrect, reverse iteration is impossible and a list comprehension is also not an option.

""" Sieve of Eratosthenes """

def generate_primes(n):
    """ Generates all primes less than n. """
    primes = list(range(2,n))
    idx = 0
    while idx < len(primes):
        p = primes[idx]
        for multiple in range(p+p, n, p):
            try:
                primes.remove(multiple)
            except ValueError:
                pass #EAFP
        idx += 1
        yield p

@CENTURION 2018-08-22 23:36:31

For anything that has the potential to be really big, I use the following.

import numpy as np

orig_list = np.array([1, 2, 3, 4, 5, 100, 8, 13])

remove_me = [100, 1]

cleaned = np.delete(orig_list, remove_me)
print(cleaned)

That should be significantly faster than anything else.

@Georgy 2019-05-14 13:06:00

From what I measured, NumPy starts to be faster for lists of more than 20 elements, and reaches >12x faster filtering for big lists of 1000 elements and more.

@Alexey 2016-09-02 13:16:37

One possible solution, useful if you want not only remove some things, but also do something with all elements in a single loop:

alist = ['good', 'bad', 'good', 'bad', 'good']
i = 0
for x in alist[:]:
    if x == 'bad':
        alist.pop(i)
        i -= 1
    # do something cool with x or just print x
    print(x)
    i += 1

@Beefster 2018-03-15 23:46:07

You should really just use comprehensions. They're much easier to understand.

@Alexey 2018-03-16 07:38:18

What if I want to remove bad things, do something with it and also do something with good things in one loop?

@Beefster 2018-03-29 19:14:08

Actually, I realized there's some cleverness here in that you make a copy of the list with an open slice (alist[:]) And since you might be doing something fancy, it actually has a use case. Good revision is good. Take my upvote.

@Beefster 2018-03-15 23:42:50

The other answers are correct that it is usually a bad idea to delete from a list that you're iterating. Reverse iterating avoids the pitfalls, but it is much more difficult to follow code that does that, so usually you're better off using a list comprehension or filter.

There is, however, one case where it is safe to remove elements from a sequence that you are iterating: if you're only removing one item while you're iterating. This can be ensured using a return or a break. For example:

for i, item in enumerate(lst):
    if item % 4 == 0:
        foo(item)
        del lst[i]
        break

This is often easier to understand than a list comprehension when you're doing some operations with side effects on the first item in a list that meets some condition and then removing that item from the list immediately after.

@Xolve 2011-01-09 12:20:34

You might want to use filter() available as the built-in.

For more details check here

@spacexengineer 2017-07-01 01:21:45

Right away you want to create a copy of the list so you can have that as a reference when you are iterating through and deleting tuples in that list that meet a certain criteria.

Then it depends on what type of list you want for the output whether that be a list of the removed tuples or a list of the tuples that are not removed.

As David pointed out, I recommend list comprehension to keep the elements you don't want to remove.

somelist = [x for x in somelist if not determine(x)]

@Michael 2017-03-13 20:54:41

I needed to do this with a huge list, and duplicating the list seemed expensive, especially since in my case the number of deletions would be few compared to the items that remain. I took this low-level approach.

array = [lots of stuff]
arraySize = len(array)
i = 0
while i < arraySize:
    if someTest(array[i]):
        del array[i]
        arraySize -= 1
    else:
        i += 1

What I don't know is how efficient a couple of deletes are compared to copying a large list. Please comment if you have any insight.

@gustavovelascoh 2017-05-05 02:13:11

In my case I need to move those 'unwanted' elements into another list. Do you have any new comment about this solution? I also think that it is better to use some deletions instead of duplicate the list.

@max 2017-05-05 11:29:23

This is the right answer if performance is an issue (although same as @Alexey). That said, the choice of list as a data structure in the first place should be carefully considered since removal from the middle of a list takes linear time in the length of the list. If you don't really need random access to k-th sequential item, maybe consider OrderedDict?

@max 2017-05-05 11:31:31

@GVelascoh why not create newlist = [], and then newlist.append(array[i]) just before del array[i]?

@Ciro Santilli 新疆改造中心法轮功六四事件 2017-06-05 16:08:37

Note that this is likely time inefficient: if list() is a linked list, the random access is expensive, if list() is an array, the deletes are expensive because they require to move all following elements forward. A decent iterator could make things good for the linked list implementation. This could however be space efficient.

@ntk4 2016-03-19 01:41:18

It might be smart to also just create a new list if the current list item meets the desired criteria.

so:

for item in originalList:
   if (item != badValue):
        newList.append(item)

and to avoid having to re-code the entire project with the new lists name:

originalList[:] = newList

note, from Python documentation:

copy.copy(x) Return a shallow copy of x.

copy.deepcopy(x) Return a deep copy of x.

@Mark Amery 2016-06-21 22:36:22

This adds no new information that wasn't in the accepted answer years earlier.

@ntk4 2016-06-23 03:08:11

It's simple and just another way to look at a problem @MarkAmery. It's less condensed for those people that don't like compressed coding syntax.

@Cinghiale 2016-10-21 13:10:31

This answer was originally written in response to a question which has since been marked as duplicate: Removing coordinates from list on python

There are two problems in your code:

1) When using remove(), you attempt to remove integers whereas you need to remove a tuple.

2) The for loop will skip items in your list.

Let's run through what happens when we execute your code:

>>> L1 = [(1,2), (5,6), (-1,-2), (1,-2)]
>>> for (a,b) in L1:
...   if a < 0 or b < 0:
...     L1.remove(a,b)
... 
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
TypeError: remove() takes exactly one argument (2 given)

The first problem is that you are passing both 'a' and 'b' to remove(), but remove() only accepts a single argument. So how can we get remove() to work properly with your list? We need to figure out what each element of your list is. In this case, each one is a tuple. To see this, let's access one element of the list (indexing starts at 0):

>>> L1[1]
(5, 6)
>>> type(L1[1])
<type 'tuple'>

Aha! Each element of L1 is actually a tuple. So that's what we need to be passing to remove(). Tuples in python are very easy, they're simply made by enclosing values in parentheses. "a, b" is not a tuple, but "(a, b)" is a tuple. So we modify your code and run it again:

# The remove line now includes an extra "()" to make a tuple out of "a,b"
L1.remove((a,b))

This code runs without any error, but let's look at the list it outputs:

L1 is now: [(1, 2), (5, 6), (1, -2)]

Why is (1,-2) still in your list? It turns out modifying the list while using a loop to iterate over it is a very bad idea without special care. The reason that (1, -2) remains in the list is that the locations of each item within the list changed between iterations of the for loop. Let's look at what happens if we feed the above code a longer list:

L1 = [(1,2),(5,6),(-1,-2),(1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)]
### Outputs:
L1 is now: [(1, 2), (5, 6), (1, -2), (3, 4), (5, 7), (2, 1), (5, -1), (0, 6)]

As you can infer from that result, every time that the conditional statement evaluates to true and a list item is removed, the next iteration of the loop will skip evaluation of the next item in the list because its values are now located at different indices.

The most intuitive solution is to copy the list, then iterate over the original list and only modify the copy. You can try doing so like this:

L2 = L1
for (a,b) in L1:
    if a < 0 or b < 0 :
        L2.remove((a,b))
# Now, remove the original copy of L1 and replace with L2
print L2 is L1
del L1
L1 = L2; del L2
print ("L1 is now: ", L1)

However, the output will be identical to before:

'L1 is now: ', [(1, 2), (5, 6), (1, -2), (3, 4), (5, 7), (2, 1), (5, -1), (0, 6)]

This is because when we created L2, python did not actually create a new object. Instead, it merely referenced L2 to the same object as L1. We can verify this with 'is' which is different from merely "equals" (==).

>>> L2=L1
>>> L1 is L2
True

We can make a true copy using copy.copy(). Then everything works as expected:

import copy
L1 = [(1,2), (5,6),(-1,-2), (1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)]
L2 = copy.copy(L1)
for (a,b) in L1:
    if a < 0 or b < 0 :
        L2.remove((a,b))
# Now, remove the original copy of L1 and replace with L2
del L1
L1 = L2; del L2
>>> L1 is now: [(1, 2), (5, 6), (3, 4), (5, 7), (2, 1), (0, 6)]

Finally, there is one cleaner solution than having to make an entirely new copy of L1. The reversed() function:

L1 = [(1,2), (5,6),(-1,-2), (1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)]
for (a,b) in reversed(L1):
    if a < 0 or b < 0 :
        L1.remove((a,b))
print ("L1 is now: ", L1)
>>> L1 is now: [(1, 2), (5, 6), (3, 4), (5, 7), (2, 1), (0, 6)]

Unfortunately, I cannot adequately describe how reversed() works. It returns a 'listreverseiterator' object when a list is passed to it. For practical purposes, you can think of it as creating a reversed copy of its argument. This is the solution I recommend.

@Resonance 2016-05-17 13:15:10

TLDR:

I wrote a library that allows you to do this:

from fluidIter import FluidIterable
fSomeList = FluidIterable(someList)  
for tup in fSomeList:
    if determine(tup):
        # remove 'tup' without "breaking" the iteration
        fSomeList.remove(tup)
        # tup has also been removed from 'someList'
        # as well as 'fSomeList'

It's best to use another method if possible that doesn't require modifying your iterable while iterating over it, but for some algorithms it might not be that straight forward. And so if you are sure that you really do want the code pattern described in the original question, it is possible.

Should work on all mutable sequences not just lists.


Full answer:

Edit: The last code example in this answer gives a use case for why you might sometimes want to modify a list in place rather than use a list comprehension. The first part of the answers serves as tutorial of how an array can be modified in place.

The solution follows on from this answer (for a related question) from senderle. Which explains how the the array index is updated while iterating through a list that has been modified. The solution below is designed to correctly track the array index even if the list is modified.

Download fluidIter.py from here https://github.com/alanbacon/FluidIterator, it is just a single file so no need to install git. There is no installer so you will need to make sure that the file is in the python path your self. The code has been written for python 3 and is untested on python 2.

from fluidIter import FluidIterable
l = [0,1,2,3,4,5,6,7,8]  
fluidL = FluidIterable(l)                       
for i in fluidL:
    print('initial state of list on this iteration: ' + str(fluidL)) 
    print('current iteration value: ' + str(i))
    print('popped value: ' + str(fluidL.pop(2)))
    print(' ')

print('Final List Value: ' + str(l))

This will produce the following output:

initial state of list on this iteration: [0, 1, 2, 3, 4, 5, 6, 7, 8]
current iteration value: 0
popped value: 2

initial state of list on this iteration: [0, 1, 3, 4, 5, 6, 7, 8]
current iteration value: 1
popped value: 3

initial state of list on this iteration: [0, 1, 4, 5, 6, 7, 8]
current iteration value: 4
popped value: 4

initial state of list on this iteration: [0, 1, 5, 6, 7, 8]
current iteration value: 5
popped value: 5

initial state of list on this iteration: [0, 1, 6, 7, 8]
current iteration value: 6
popped value: 6

initial state of list on this iteration: [0, 1, 7, 8]
current iteration value: 7
popped value: 7

initial state of list on this iteration: [0, 1, 8]
current iteration value: 8
popped value: 8

Final List Value: [0, 1]

Above we have used the pop method on the fluid list object. Other common iterable methods are also implemented such as del fluidL[i], .remove, .insert, .append, .extend. The list can also be modified using slices (sort and reverse methods are not implemented).

The only condition is that you must only modify the list in place, if at any point fluidL or l were reassigned to a different list object the code would not work. The original fluidL object would still be used by the for loop but would become out of scope for us to modify.

i.e.

fluidL[2] = 'a'   # is OK
fluidL = [0, 1, 'a', 3, 4, 5, 6, 7, 8]  # is not OK

If we want to access the current index value of the list we cannot use enumerate, as this only counts how many times the for loop has run. Instead we will use the iterator object directly.

fluidArr = FluidIterable([0,1,2,3])
# get iterator first so can query the current index
fluidArrIter = fluidArr.__iter__()
for i, v in enumerate(fluidArrIter):
    print('enum: ', i)
    print('current val: ', v)
    print('current ind: ', fluidArrIter.currentIndex)
    print(fluidArr)
    fluidArr.insert(0,'a')
    print(' ')

print('Final List Value: ' + str(fluidArr))

This will output the following:

enum:  0
current val:  0
current ind:  0
[0, 1, 2, 3]

enum:  1
current val:  1
current ind:  2
['a', 0, 1, 2, 3]

enum:  2
current val:  2
current ind:  4
['a', 'a', 0, 1, 2, 3]

enum:  3
current val:  3
current ind:  6
['a', 'a', 'a', 0, 1, 2, 3]

Final List Value: ['a', 'a', 'a', 'a', 0, 1, 2, 3]

The FluidIterable class just provides a wrapper for the original list object. The original object can be accessed as a property of the fluid object like so:

originalList = fluidArr.fixedIterable

More examples / tests can be found in the if __name__ is "__main__": section at the bottom of fluidIter.py. These are worth looking at because they explain what happens in various situations. Such as: Replacing a large sections of the list using a slice. Or using (and modifying) the same iterable in nested for loops.

As I stated to start with: this is a complicated solution that will hurt the readability of your code and make it more difficult to debug. Therefore other solutions such as the list comprehensions mentioned in David Raznick's answer should be considered first. That being said, I have found times where this class has been useful to me and has been easier to use than keeping track of the indices of elements that need deleting.


Edit: As mentioned in the comments, this answer does not really present a problem for which this approach provides a solution. I will try to address that here:

List comprehensions provide a way to generate a new list but these approaches tend to look at each element in isolation rather than the current state of the list as a whole.

i.e.

newList = [i for i in oldList if testFunc(i)]

But what if the result of the testFunc depends on the elements that have been added to newList already? Or the elements still in oldList that might be added next? There might still be a way to use a list comprehension but it will begin to lose it's elegance, and for me it feels easier to modify a list in place.

The code below is one example of an algorithm that suffers from the above problem. The algorithm will reduce a list so that no element is a multiple of any other element.

randInts = [70, 20, 61, 80, 54, 18, 7, 18, 55, 9]
fRandInts = FluidIterable(randInts)
fRandIntsIter = fRandInts.__iter__()
# for each value in the list (outer loop)
# test against every other value in the list (inner loop)
for i in fRandIntsIter:
    print(' ')
    print('outer val: ', i)
    innerIntsIter = fRandInts.__iter__()
    for j in innerIntsIter:
        innerIndex = innerIntsIter.currentIndex
        # skip the element that the outloop is currently on
        # because we don't want to test a value against itself
        if not innerIndex == fRandIntsIter.currentIndex:
            # if the test element, j, is a multiple 
            # of the reference element, i, then remove 'j'
            if j%i == 0:
                print('remove val: ', j)
                # remove element in place, without breaking the
                # iteration of either loop
                del fRandInts[innerIndex]
            # end if multiple, then remove
        # end if not the same value as outer loop
    # end inner loop
# end outerloop

print('')
print('final list: ', randInts)

The output and the final reduced list are shown below

outer val:  70

outer val:  20
remove val:  80

outer val:  61

outer val:  54

outer val:  18
remove val:  54
remove val:  18

outer val:  7
remove val:  70

outer val:  55

outer val:  9
remove val:  18

final list:  [20, 61, 7, 55, 9]

@Mark Amery 2016-06-21 22:47:31

It's hard to tell whether this is over-engineered because it's unclear what problem it's trying to solve; what does removing elements using this approach achieve that some_list[:] = [x for x in some_list if not some_condition(x)] doesn't achieve? Without an answer to that, why should anyone believe that downloading and using your 600-line library complete with typos and commented-out code is a better solution to their problem than the one-liner? -1.

@Resonance 2016-06-28 13:27:10

@MarkAmery. The main use case for when this is when trying to determine if an item should be removed (or added or moved) based not on just the item itself, but on the state of another item in the list or the state of the list as a whole. For example, it is not possible with list comprehensions to write something like some_list[:] = [x for x in some_list if not some_condition(y)] where y is a different list element from x. Nor would it be possible to write some_list[:] = [x for x in some_list if not some_condition(intermediateStateOf_some_list)].

@David Raznick 2009-07-30 15:41:33

You can use a list comprehension to create a new list containing only the elements you don't want to remove:

somelist = [x for x in somelist if not determine(x)]

Or, by assigning to the slice somelist[:], you can mutate the existing list to contain only the items you want:

somelist[:] = [x for x in somelist if not determine(x)]

This approach could be useful if there are other references to somelist that need to reflect the changes.

Instead of a comprehension, you could also use itertools. In Python 2:

from itertools import ifilterfalse
somelist[:] = ifilterfalse(determine, somelist)

Or in Python 3:

from itertools import filterfalse
somelist[:] = filterfalse(determine, somelist)

@highBandWidth 2011-04-20 19:25:57

Can you make it faster if you know only a few will be deleted, i.e., only delete those and leave the others in-place rather than re-writing them?

@jpcgt 2014-11-15 23:43:18

What if my list is huge and can't afford making a copy?

@Rostislav Kondratenko 2015-04-29 14:54:28

@jpcgt You should use somelist[:] = (x for x in somelist if determine(x)) this will create generator that may not create any unnecessary copies.

@jfs 2015-05-07 20:48:28

@RostislavKondratenko: list_ass_slice() function that implements somelist[:]= calls PySequence_Fast() internally. This function always returns a list i.e., @Alex Martelli's solution that already uses a list instead of a generator is most probably more efficient

@Shuklaswag 2018-02-07 19:38:25

@TerraAshley When you say "these" are you referring to the slicing notation, [:]? I agree it's ugly, and looks like a bit of a hack.

@Tom Russell 2018-02-28 08:58:12

I think it would be better stated as simply [x for x in somelist if include(x)].

@Dims 2018-03-29 16:25:10

How this can guarantee conserving list position?

@Bowen Liu 2018-09-24 19:06:14

Would you care to explain what the differences are between assigning the list comprehension to the list and list clone please? Wouldn't the original list somelist be mutated in both methods?

@naktinis 2019-12-02 15:15:04

@Bowen Liu, not if there are other references to the list earlier on. Compare the following two examples: a = [1, 2, 3]; b = a; a = [i for i in a if i > 1]; print(a); print(b) and a = [1, 2, 3]; b = a; a[:] = [i for i in a if i > 1]; print(a); print(b).

@Cide 2009-07-30 15:46:46

For those that like functional programming:

somelist[:] = filter(lambda tup: not determine(tup), somelist)

or

from itertools import ifilterfalse
somelist[:] = list(ifilterfalse(determine, somelist))

@ShadowRanger 2016-10-22 00:22:22

1. List comprehension and generator expressions are borrowed from Haskell, a pure functional language; they're exactly as functional as filter, and more Pythonic. 2. If you need a lambda to use map or filter, the list comp or genexpr is always the better option; map and filter can be slightly faster when the transform/predicate function is a Python built-in implemented in C and the iterable is not trivially small, but they're always slower when you need a lambda that the listcomp/genexpr could avoid.

@rafa 2015-12-16 10:56:37

I needed to do something similar and in my case the problem was memory - I needed to merge multiple dataset objects within a list, after doing some stuff with them, as a new object, and needed to get rid of each entry I was merging to avoid duplicating all of them and blowing up memory. In my case having the objects in a dictionary instead of a list worked fine:

```

k = range(5)
v = ['a','b','c','d','e']
d = {key:val for key,val in zip(k, v)}

print d
for i in range(5):
    print d[i]
    d.pop(i)
print d

```

@Queequeg 2015-07-10 20:58:49

You can try for-looping in reverse so for some_list you'll do something like:

list_len = len(some_list)
for i in range(list_len):
    reverse_i = list_len - 1 - i
    cur = some_list[reverse_i]

    # some logic with cur element

    if some_condition:
        some_list.pop(reverse_i)

This way the index is aligned and doesn't suffer from the list updates (regardless whether you pop cur element or not).

@Mark Amery 2016-06-21 22:49:43

Looping over reversed(list(enumerate(some_list))) would be simpler than computing indexes yourself.

@Queequeg 2016-06-28 18:29:06

@MarkAmery don't think you can alter the list this way.

@fantabolous 2014-08-18 12:30:16

If you want to do anything else during the iteration, it may be nice to get both the index (which guarantees you being able to reference it, for example if you have a list of dicts) and the actual list item contents.

inlist = [{'field1':10, 'field2':20}, {'field1':30, 'field2':15}]    
for idx, i in enumerate(inlist):
    do some stuff with i['field1']
    if somecondition:
        xlist.append(idx)
for i in reversed(xlist): del inlist[i]

enumerate gives you access to the item and the index at once. reversed is so that the indices that you're going to later delete don't change on you.

@Mark Amery 2016-06-21 22:33:52

Why is getting the index any more relevant in the case where you have a list of dicts than in the case of any other kind of list? This doesn't make sense as far as I can tell.

@Eli Courtwright 2009-07-30 15:41:30

Your best approach for such an example would be a list comprehension

somelist = [tup for tup in somelist if determine(tup)]

In cases where you're doing something more complex than calling a determine function, I prefer constructing a new list and simply appending to it as I go. For example

newlist = []
for tup in somelist:
    # lots of code here, possibly setting things up for calling determine
    if determine(tup):
        newlist.append(tup)
somelist = newlist

Copying the list using remove might make your code look a little cleaner, as described in one of the answers below. You should definitely not do this for extremely large lists, since this involves first copying the entire list, and also performing an O(n) remove operation for each element being removed, making this an O(n^2) algorithm.

for tup in somelist[:]:
    # lots of code here, possibly setting things up for calling determine
    if determine(tup):
        newlist.append(tup)

Related Questions

Sponsored Content

45 Answered Questions

[SOLVED] How to make a flat list out of list of lists?

28 Answered Questions

[SOLVED] How do I check if a list is empty?

  • 2008-09-10 06:20:11
  • Ray Vega
  • 2548517 View
  • 3235 Score
  • 28 Answer
  • Tags:   python list

13 Answered Questions

[SOLVED] How to randomly select an item from a list?

  • 2008-11-20 18:42:21
  • Ray Vega
  • 1318139 View
  • 1658 Score
  • 13 Answer
  • Tags:   python list random

12 Answered Questions

[SOLVED] How do I use itertools.groupby()?

  • 2008-08-03 18:27:09
  • James Sulak
  • 265463 View
  • 466 Score
  • 12 Answer
  • Tags:   python iteration

11 Answered Questions

[SOLVED] What are "named tuples" in Python?

29 Answered Questions

[SOLVED] Finding the index of an item given a list containing it in Python

  • 2008-10-07 01:39:38
  • Eugene M
  • 3531591 View
  • 2894 Score
  • 29 Answer
  • Tags:   python list indexing

41 Answered Questions

[SOLVED] How do I efficiently iterate over each entry in a Java Map?

21 Answered Questions

[SOLVED] How do I list all files of a directory?

  • 2010-07-08 19:31:22
  • duhhunjonn
  • 3761708 View
  • 3474 Score
  • 21 Answer
  • Tags:   python directory

10 Answered Questions

[SOLVED] How to delete items from a dictionary while iterating over it?

Sponsored Content