By akdom


2008-08-21 00:36:11 8 Comments

How would one create an iterative function (or iterator object) in python?

9 comments

@ars 2008-08-23 16:57:28

Iterator objects in python conform to the iterator protocol, which basically means they provide two methods: __iter__() and __next__().

  • The __iter__ returns the iterator object and is implicitly called at the start of loops.

  • The __next__() method returns the next value and is implicitly called at each loop increment. This method raises a StopIteration exception when there are no more value to return, which is implicitly captured by looping constructs to stop iterating.

Here's a simple example of a counter:

class Counter:
    def __init__(self, low, high):
        self.current = low - 1
        self.high = high

    def __iter__(self):
        return self

    def __next__(self): # Python 2: def next(self)
        self.current += 1
        if self.current < self.high:
            return self.current
        raise StopIteration


for c in Counter(3, 9):
    print(c)

This will print:

3
4
5
6
7
8

This is easier to write using a generator, as covered in a previous answer:

def counter(low, high):
    current = low
    while current < high:
        yield current
        current += 1

for c in counter(3, 9):
    print(c)

The printed output will be the same. Under the hood, the generator object supports the iterator protocol and does something roughly similar to the class Counter.

David Mertz's article, Iterators and Simple Generators, is a pretty good introduction.

@Casey Rodarmor 2014-02-06 23:33:31

This is mostly a good answer, but the fact that it returns self is a little sub-optimal. For example, if you used the same counter object in a doubly nested for loop you would probably not get the behavior that you meant.

@leewz 2014-02-21 08:42:44

No, iterators SHOULD return themselves. Iterables return iterators, but iterables shouldn't implement __next__. counter is an iterator, but it is not a sequence. It doesn't store its values. You shouldn't be using the counter in a doubly-nested for-loop, for example.

@Curt 2016-04-05 23:00:00

In the Counter example, self.current should be assigned in __iter__ (in addition to in __init__). Otherwise, the object can be iterated only once. E.g., if you say ctr = Counters(3, 8), then you cannot use for c in ctr more than once.

@kdubs 2017-03-13 02:01:54

shouldn't the __iter__ code be setting the value of self.current?

@ShadowRanger 2018-02-24 01:16:43

@Curt: Absolutely not. Counter is an iterator, and iterators are only supposed to be iterated once. If you reset self.current in __iter__, then a nested loop over the Counter would be completely broken, and all sorts of assumed behaviors of iterators (that calling iter on them is idempotent) are violated. If you want to be able to iterate ctr more than once, it needs to be a non-iterator iterable, where it returns a brand new iterator each time __iter__ is invoked. Trying to mix and match (an iterator that is implicitly reset when __iter__ is invoked) violates the protocols.

@ShadowRanger 2018-02-24 01:19:21

For example, if Counter was to be a non-iterator iterable, you'd remove the definition of __next__/next entirely, and probably redefine __iter__ as a generator function of the same form as the generator described at the end of this answer (except instead of the bounds coming from arguments to __iter__, they'd be arguments to __init__ saved on self and accessed from self in __iter__).

@ShadowRanger 2018-02-24 01:43:52

BTW, a useful thing to do if you want to write portable iterator classes is to define either next or __next__, then assign one name to the other (next = __next__ or __next__ = next depending on the name you gave the method). Having both names defined means it works on both Py2 and Py3 without source code changes.

@Minh Tran 2018-05-19 21:24:10

Thanks for the answer. To clarify an ambiguity: __iter__() gets called once before entering the looping construct. "... at the start of loops" suggests __iter__ gets called at the beginning of each loop in the same looping construct, which is false. A double nested for loop using Counter will show that __iter__ gets called once each time and before the nested for-loop executes.

@Ethan Furman 2011-09-24 22:13:44

There are four ways to build an iterative function:

Examples:

# generator
def uc_gen(text):
    for char in text:
        yield char.upper()

# generator expression
def uc_genexp(text):
    return (char.upper() for char in text)

# iterator protocol
class uc_iter():
    def __init__(self, text):
        self.text = text
        self.index = 0
    def __iter__(self):
        return self
    def __next__(self):
        try:
            result = self.text[self.index].upper()
        except IndexError:
            raise StopIteration
        self.index += 1
        return result

# getitem method
class uc_getitem():
    def __init__(self, text):
        self.text = text
    def __getitem__(self, index):
        result = self.text[index].upper()
        return result

To see all four methods in action:

for iterator in uc_gen, uc_genexp, uc_iter, uc_getitem:
    for ch in iterator('abcde'):
        print ch,
    print

Which results in:

A B C D E
A B C D E
A B C D E
A B C D E

Note:

The two generator types (uc_gen and uc_genexp) cannot be reversed(); the plain iterator (uc_iter) would need the __reversed__ magic method (which must return a new iterator that goes backwards); and the getitem iteratable (uc_getitem) must have the __len__ magic method:

    # for uc_iter
    def __reversed__(self):
        return reversed(self.text)

    # for uc_getitem
    def __len__(self)
        return len(self.text)

To answer Colonel Panic's secondary question about an infinite lazily evaluated iterator, here are those examples, using each of the four methods above:

# generator
def even_gen():
    result = 0
    while True:
        yield result
        result += 2


# generator expression
def even_genexp():
    return (num for num in even_gen())  # or even_iter or even_getitem
                                        # not much value under these circumstances

# iterator protocol
class even_iter():
    def __init__(self):
        self.value = 0
    def __iter__(self):
        return self
    def __next__(self):
        next_value = self.value
        self.value += 2
        return next_value

# getitem method
class even_getitem():
    def __getitem__(self, index):
        return index * 2

import random
for iterator in even_gen, even_genexp, even_iter, even_getitem:
    limit = random.randint(15, 30)
    count = 0
    for even in iterator():
        print even,
        count += 1
        if count >= limit:
            break
    print

Which results in (at least for my sample run):

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

How to choose which one to use? This is mostly a matter of taste. The two methods I see most often are generators and the iterator protocol, as well as a hybrid (__iter__ returning a generator).

Generator expressions are useful for replacing list comprehensions (they are lazy and so can save on resources).

If one needs compatibility with earlier Python 2.x versions use __getitem__.

@Ian 2013-07-05 01:04:22

I like this summary because it is complete. Those three ways (yield, generator expression and iterator) are essentially the same, although some are more convenient than others. The yield operator captures the "continuation" which contains the state (for example the index that we are up to). The information is saved in the "closure" of the continuation. The iterator way saves the same information inside the fields of the iterator, which is essentially the same thing as a closure. The getitem method is a little different because it indexes into the contents and is not iterative in nature.

@Terrence Brannon 2013-11-05 15:25:55

You aren't incrementing the index in your last approach, uc_getitem() . Actually on reflection, it shouldnt increment the index, because it is not maintaining it. But it also is not a way to abstract iteration.

@Ethan Furman 2013-11-05 16:37:21

@metaperl: Actually, it is. In all four of the above cases you can use the same code to iterate.

@Asterisk 2018-04-19 09:30:16

@EthanFurman I am not an expert, but should there not be a reset of the index in the uc_iter class? I.e. inside iter method set self.index to 0 so that the next invocation of the iterator works

@Ethan Furman 2018-04-19 16:13:36

@Asterisk: No, an instance of uc_iter should expire when it's done (otherwise it would by infinite); if you want to do it again you have to get a new iterator by calling uc_iter() again.

@Josiah Yoder 2018-07-27 14:35:42

@TerrenceBrannon uc_getitem() works, but perhaps can be considered an option for backwards compatibility. It stops when an IndexError is raised.

@John Strood 2018-08-14 08:26:20

You can set self.index = 0 in __iter__ so that you can iterate many times over. Otherwise you can't.

@aaaaaa 2019-01-21 04:15:06

If you could spare the time I would appreciate an explanation for why you would choose any of the methods over the others.

@BlackJack 2019-06-27 14:02:05

@JohnStrood Absolutely not, because it would violate the iterator protocol. Iterators are expected to just return themselves in __iter__() so iter(iterator_instance) doesn't change the state of the given iterator instance.

@simpleuser 2019-08-27 20:40:35

@JohnStrood to iterate over a complex MyClass object more than once, create a MyClassIterator class with just __init__ and __next__ and return an instance of that from MyClass.__iter__ (e.g. return MyClassIterator(self)) instead of just self, so that MyClassIterator can store a reference to the MyClass instance as well as the current data index, making it safe to be called multiple times at once while still satisfying the other iterator protocol issues.

@Ethan Furman 2019-08-28 15:57:29

@aaaaaa: "How to choose" section added (at the bottom).

@John Strood 2018-08-14 08:25:18

All answers on this page are really great for a complex object. But for those containing builtin iterable types as attributes, like str, list, set or dict, or any implementation of collections.Iterable, you can omit certain things in your class.

class Test(object):
    def __init__(self, string):
        self.string = string

    def __iter__(self):
        # since your string is already iterable
        return (ch for ch in self.string)
        # or simply
        return self.string.__iter__()
        # also
        return iter(self.string)

It can be used like:

for x in Test("abcde"):
    print(x)

# prints
# a
# b
# c
# d
# e

@BlackJack 2019-06-27 14:07:39

As you said, the string is already iterable so why the extra generator expression in between instead of just asking the string for the iterator (which the generator expression does internally): return iter(self.string).

@John Strood 2019-06-27 16:40:54

@BlackJack You're indeed right. I do not know what persuaded me to write that way. Perhaps I was trying to avoid any confusion in an answer trying to explain the working of iterator syntax in terms of more iterator syntax.

@akdom 2008-08-21 00:36:33

First of all the itertools module is incredibly useful for all sorts of cases in which an iterator would be useful, but here is all you need to create an iterator in python:

yield

Isn't that cool? Yield can be used to replace a normal return in a function. It returns the object just the same, but instead of destroying state and exiting, it saves state for when you want to execute the next iteration. Here is an example of it in action pulled directly from the itertools function list:

def count(n=0):
    while True:
        yield n
        n += 1

As stated in the functions description (it's the count() function from the itertools module...) , it produces an iterator that returns consecutive integers starting with n.

Generator expressions are a whole other can of worms (awesome worms!). They may be used in place of a List Comprehension to save memory (list comprehensions create a list in memory that is destroyed after use if not assigned to a variable, but generator expressions can create a Generator Object... which is a fancy way of saying Iterator). Here is an example of a generator expression definition:

gen = (n for n in xrange(0,11))

This is very similar to our iterator definition above except the full range is predetermined to be between 0 and 10.

I just found xrange() (suprised I hadn't seen it before...) and added it to the above example. xrange() is an iterable version of range() which has the advantage of not prebuilding the list. It would be very useful if you had a giant corpus of data to iterate over and only had so much memory to do it in.

@user3850 2008-12-18 17:30:08

as of python 3.0 there is no longer an xrange() and the new range() behaves like the old xrange()

@Phob 2011-07-22 18:03:40

You should still use xrange in 2._, because 2to3 translates it automatically.

@Ace.Di 2018-07-13 17:34:13

Inspired by Matt Gregory's answer here is a bit more complicated iterator that will return a,b,...,z,aa,ab,...,zz,aaa,aab,...,zzy,zzz

    class AlphaCounter:
    def __init__(self, low, high):
        self.current = low
        self.high = high

    def __iter__(self):
        return self

    def __next__(self): # Python 3: def __next__(self)
        alpha = ' abcdefghijklmnopqrstuvwxyz'
        n_current = sum([(alpha.find(self.current[x])* 26**(len(self.current)-x-1)) for x in range(len(self.current))])
        n_high = sum([(alpha.find(self.high[x])* 26**(len(self.high)-x-1)) for x in range(len(self.high))])
        if n_current > n_high:
            raise StopIteration
        else:
            increment = True
            ret = ''
            for x in self.current[::-1]:
                if 'z' == x:
                    if increment:
                        ret += 'a'
                    else:
                        ret += 'z'
                else:
                    if increment:
                        ret += alpha[alpha.find(x)+1]
                        increment = False
                    else:
                        ret += x
            if increment:
                ret += 'a'
            tmp = self.current
            self.current = ret[::-1]
            return tmp

for c in AlphaCounter('a', 'zzz'):
    print(c)

@Daniil Mashkin 2018-04-26 08:38:39

If you looking for something short and simple, maybe it will be enough for you:

class A(object):
    def __init__(self, l):
        self.data = l

    def __iter__(self):
        return iter(self.data)

example of usage:

In [3]: a = A([2,3,4])

In [4]: [i for i in a]
Out[4]: [2, 3, 4]

@aq2 2016-03-21 17:39:14

This question is about iterable objects, not about iterators. In Python, sequences are iterable too so one way to make an iterable class is to make it behave like a sequence, i.e. give it __getitem__ and __len__ methods. I have tested this on Python 2 and 3.

class CustomRange:

    def __init__(self, low, high):
        self.low = low
        self.high = high

    def __getitem__(self, item):
        if item >= len(self):
            raise IndexError("CustomRange index out of range")
        return self.low + item

    def __len__(self):
        return self.high - self.low


cr = CustomRange(0, 10)
for i in cr:
    print(i)

@BlackJack 2019-06-27 14:05:37

It doesn't have to have a __len__() method. __getitem__ alone with the expected behaviour is sufficient.

@Nizam Mohamed 2016-03-03 17:55:26

This is an iterable function without yield. It make use of the iter function and a closure which keeps it's state in a mutable (list) in the enclosing scope for python 2.

def count(low, high):
    counter = [0]
    def tmp():
        val = low + counter[0]
        if val < high:
            counter[0] += 1
            return val
        return None
    return iter(tmp, None)

For Python 3, closure state is kept in an immutable in the enclosing scope and nonlocal is used in local scope to update the state variable.

def count(low, high):
    counter = 0
    def tmp():
        nonlocal counter
        val = low + counter
        if val < high:
            counter += 1
            return val
        return None
    return iter(tmp, None)  

Test;

for i in count(1,10):
    print(i)
1
2
3
4
5
6
7
8
9

@ShadowRanger 2018-02-24 01:30:05

I always appreciate a clever use of two-arg iter, but just to be clear: This is more complex and less efficient than just using a yield based generator function; Python has a ton of interpreter support for yield based generator functions that you can't take advantage of here, making this code significantly slower. Up-voted nonetheless.

@Manux 2012-07-27 15:05:12

I see some of you doing return self in __iter__. I just wanted to note that __iter__ itself can be a generator (thus removing the need for __next__ and raising StopIteration exceptions)

class range:
  def __init__(self,a,b):
    self.a = a
    self.b = b
  def __iter__(self):
    i = self.a
    while i < self.b:
      yield i
      i+=1

Of course here one might as well directly make a generator, but for more complex classes it can be useful.

@Ray 2013-02-05 19:32:35

Great! It so boring writing just return self in __iter__. When I was going to try using yield in it I found your code doing exactly what I want to try.

@Lenna 2013-04-05 19:52:12

But in this case, how would one implement next()? return iter(self).next()?

@Manux 2013-04-07 17:31:31

@Lenna, it is already "implemented" because iter(self) returns an iterator, not a range instance.

@Lenna 2013-04-24 19:06:36

@Manux iter(range(5,10)).next() is a bit cumbersome. Admittedly a bad example for next behavior. I'm still interested in how to give the range instance a next attribute.

@astrofrog 2014-03-31 13:35:53

This the easiest way of doing it, and doesn't involve having to keep track of e.g. self.current or any other counter. This should be the top-voted answer!

@Ethan Furman 2014-11-09 19:42:19

The difference: __iter__ being a generator is a different object than the range() instance. Sometimes this matters, sometimes it doesn't.

@Mad Physicist 2016-03-14 20:41:59

You shouldn't be using iter(range(5,10)).next() anyway. The correct way is next(iter(range(5,10))). The next builtin is there exactly so you don't have to care whether or not self is returned in this situation.

@swinman 2016-09-17 16:09:23

up-voted -- this method also works more like expected (relative to the accepted answer) for something like r = range(5); list_of_lists = list([ri, list(r)] for ri in r)

@Acumenus 2017-04-18 14:42:52

It's interesting that __iter__ doesn't have to raise StopIteration. A problem with defining only __iter__ is that next(myiterator) doesn't work if __next__ does not return individual items. Needing to use next(iter(myiterator)) is not a wise substitute.

@ShadowRanger 2018-02-24 01:25:07

To be clear, this approach makes your class iterable, but not an iterator. You get fresh iterators every time you call iter on instances of the class, but they're not themselves instances of the class.

@ShadowRanger 2018-02-24 01:27:54

@MadPhysicist: On Python 2, iter(range(5,10)).next() and next(iter(range(5,10))) are already exactly equivalent. The advantage to next as a function has nothing to do with whether self is returned by __iter__ (the behavior is identical for both code snippets). The advantages of the next built-in function are: 1. It works the same on Py2 and Py3, even though the method changes names between them and 2. When applicable, it can be given a second argument to return in the event that the iterator is already exhausted, rather than raising StopIteration.

Related Questions

Sponsored Content

29 Answered Questions

[SOLVED] Finding the index of an item given a list containing it in Python

  • 2008-10-07 01:39:38
  • Eugene M
  • 3597111 View
  • 2955 Score
  • 29 Answer
  • Tags:   python list indexing

18 Answered Questions

[SOLVED] What are metaclasses in Python?

18 Answered Questions

[SOLVED] Using global variables in a function

42 Answered Questions

[SOLVED] How do I merge two dictionaries in a single expression?

21 Answered Questions

[SOLVED] How do I list all files of a directory?

  • 2010-07-08 19:31:22
  • duhhunjonn
  • 3880708 View
  • 3474 Score
  • 21 Answer
  • Tags:   python directory

10 Answered Questions

[SOLVED] Iterating over dictionaries using 'for' loops

68 Answered Questions

[SOLVED] What is the most efficient way to deep clone an object in JavaScript?

63 Answered Questions

[SOLVED] Calling an external command from Python

28 Answered Questions

[SOLVED] Iterate through object properties

Sponsored Content