By thechrishaddad


2012-03-27 06:03:37 8 Comments

What is the most basic definition of "iterable", "iterator" and "iteration" in Python?

I have read multiple definitions but I am unable to identify the exact meaning as it still won't sink in.

Can someone please help me with the 3 definitions in layman terms?

12 comments

@Raymond Hettinger 2012-03-27 06:39:31

Here's the explanation I use in teaching Python classes:

An ITERABLE is:

  • anything that can be looped over (i.e. you can loop over a string or file) or
  • anything that can appear on the right-side of a for-loop: for x in iterable: ... or
  • anything you can call with iter() that will return an ITERATOR: iter(obj) or
  • an object that defines __iter__ that returns a fresh ITERATOR, or it may have a __getitem__ method suitable for indexed lookup.

An ITERATOR is an object:

  • with state that remembers where it is during iteration,
  • with a __next__ method that:
    • returns the next value in the iteration
    • updates the state to point at the next value
    • signals when it is done by raising StopIteration
  • and that is self-iterable (meaning that it has an __iter__ method that returns self).

Notes:

  • The __next__ method in Python 3 is spelt next in Python 2, and
  • The builtin function next() calls that method on the object passed to it.

For example:

>>> s = 'cat'      # s is an ITERABLE
                   # s is a str object that is immutable
                   # s has no state
                   # s has a __getitem__() method 

>>> t = iter(s)    # t is an ITERATOR
                   # t has state (it starts by pointing at the "c"
                   # t has a next() method and an __iter__() method

>>> next(t)        # the next() function returns the next value and advances the state
'c'
>>> next(t)        # the next() function returns the next value and advances
'a'
>>> next(t)        # the next() function returns the next value and advances
't'
>>> next(t)        # next() raises StopIteration to signal that iteration is complete
Traceback (most recent call last):
...
StopIteration

>>> iter(t) is t   # the iterator is self-iterable

@lmiguelvargasf 2017-02-06 15:26:41

what do you mean by fresh iterator?

@Raymond Hettinger 2017-02-07 03:15:35

@lmiguelvargasf "Fresh" as in "new and unconsumed" as opposed to "exhausted or partially consumed". The idea is that a new iterator starts at the beginning, while a partially used iterator picks up where it left off.

@fountainhead 2019-02-19 04:58:13

Your 2nd, 3rd, and 4th bullets clearly indicate what you mean, in terms of specific python constructs or built-ins or method calls. But the 1st bullet ("anything that can be looped over") doesn't have that clarity. Also, the 1st bullet seems to have an overlap with the 2nd bullet, since the 2nd bullet is about for loops, and the 1st bullet is about "looping over". Could you pls address these?

@fountainhead 2019-02-19 05:07:20

Pls consider re-phrasing "anything your can call with iter()" as "anything you can pass to iter()"

@TJC World 2018-05-28 19:00:22

Iterable:- something that is iterable is iterable; like sequences like lists ,strings etc. Also it has either the __getitem__ method or an __iter__ method. Now if we use iter() function on that object, we'll get an iterator.

Iterator:- When we get the iterator object from the iter() function; we call __next__() method (in python3) or simply next() (in python2) to get elements one by one. This class or instance of this class is called an iterator.

From docs:-

The use of iterators pervades and unifies Python. Behind the scenes, the for statement calls iter() on the container object. The function returns an iterator object that defines the method __next__() which accesses elements in the container one at a time. When there are no more elements, __next__() raises a StopIteration exception which tells the for loop to terminate. You can call the __next__() method using the next() built-in function; this example shows how it all works:

>>> s = 'abc'
>>> it = iter(s)
>>> it
<iterator object at 0x00A1DB50>
>>> next(it)
'a'
>>> next(it)
'b'
>>> next(it)
'c'
>>> next(it)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    next(it)
StopIteration

Ex of a class:-

class Reverse:
    """Iterator for looping over a sequence backwards."""
    def __init__(self, data):
        self.data = data
        self.index = len(data)
    def __iter__(self):
        return self
    def __next__(self):
        if self.index == 0:
            raise StopIteration
        self.index = self.index - 1
        return self.data[self.index]


>>> rev = Reverse('spam')
>>> iter(rev)
<__main__.Reverse object at 0x00A1DB50>
>>> for char in rev:
...     print(char)
...
m
a
p
s

@techkuz 2018-09-06 11:10:36

Iterables have a __iter__ method that instantiates a new iterator every time.

Iterators implement a __next__ method that returns individual items, and a __iter__ method that returns self .

Therefore, iterators are also iterable, but iterables are not iterators.

Luciano Ramalho, Fluent Python.

@AXO 2018-03-25 14:53:31

Here's my cheat sheet:

 sequence
  +
  |
  v
   def __getitem__(self, index: int):
  +    ...
  |    raise IndexError
  |
  |
  |              def __iter__(self):
  |             +     ...
  |             |     return <iterator>
  |             |
  |             |
  +--> or <-----+        def __next__(self):
       +        |       +    ...
       |        |       |    raise StopIteration
       v        |       |
    iterable    |       |
           +    |       |
           |    |       v
           |    +----> and +-------> iterator
           |                               ^
           v                               |
   iter(<iterable>) +----------------------+
                                           |
   def generator():                        |
  +    yield 1                             |
  |                 generator_expression +-+
  |                                        |
  +-> generator() +-> generator_iterator +-+

Quiz: Do you see how...

  • every iterator is an iterable?
  • a container object's __iter__() method can be implemented as a generator?
  • an iterable plus a __next__ method is not necessarily an iterator?

@Shadow 2018-01-28 07:31:42

Before dealing with the iterables and iterator the major factor that decide the iterable and iterator is sequence

Sequence:Sequence is the collection of data

Iterable:Iterable are the sequence type object that support Iter method.

Iter method:Iter method take sequence as an input and create an object which is known as iterator

Iterator:Iterator are the object which call next method and transverse through the sequence.On calling the next method it returns the object that it transversed currently.

example:

x=[1,2,3,4]

x is a sequence which consists of collection of data

y=iter(x)

On calling iter(x) it returns a iterator only when the x object has iter method otherwise it raise an exception.If it returns iterator then y is assign like this:

y=[1,2,3,4]

As y is a iterator hence it support next() method

On calling next method it returns the individual elements of the list one by one.

After returning the last element of the sequence if we again call the next method it raise an StopIteration error

example:

>>> y.next()
1
>>> y.next()
2
>>> y.next()
3
>>> y.next()
4
>>> y.next()
StopIteration

@coelhudo 2018-01-28 07:45:53

Just an observation: y=iter(x) is not exactly y=[1,2,3,4] since y is now an iterator object. Perhaps you should add a comment to clarify that is not a list but an iterator object or change the representation.

@arpan kumar 2017-06-19 12:57:06

iterable = [1, 2] 

iterator = iter(iterable)

print(iterator.__next__())   

print(iterator.__next__())   

so,

  1. iterable is an object that can be looped over. e.g. list , string , tuple etc.

  2. using the iter function on our iterable object will return an iterator object.

  3. now this iterator object has method named __next__ (in Python 3, or just next in Python 2) by which you can access each element of iterable.

so, OUTPUT OF ABOVE CODE WILL BE:

1

2

@Alois Mahdal 2013-09-11 14:59:39

The above answers are great, but as most of what I've seen, don't stress the distinction enough for people like me.

Also, people tend to get "too Pythonic" by putting definitions like "X is an object that has __foo__() method" before. Such definitions are correct--they are based on duck-typing philosophy, but the focus on methods tends to get between when trying to understand the concept in its simplicity.

So I add my version.


In natural language,

  • iteration is the process of taking one element at a time in a row of elements.

In Python,

  • iterable is an object that is, well, iterable, which simply put, means that it can be used in iteration, e.g. with a for loop. How? By using iterator. I'll explain below.

  • ... while iterator is an object that defines how to actually do the iteration--specifically what is the next element. That's why it must have next() method.

Iterators are themselves also iterable, with the distinction that their __iter__() method returns the same object (self), regardless of whether or not its items have been consumed by previous calls to next().


So what does Python interpreter think when it sees for x in obj: statement?

Look, a for loop. Looks like a job for an iterator... Let's get one. ... There's this obj guy, so let's ask him.

"Mr. obj, do you have your iterator?" (... calls iter(obj), which calls obj.__iter__(), which happily hands out a shiny new iterator _i.)

OK, that was easy... Let's start iterating then. (x = _i.next() ... x = _i.next()...)

Since Mr. obj succeeded in this test (by having certain method returning a valid iterator), we reward him with adjective: you can now call him "iterable Mr. obj".

However, in simple cases, you don't normally benefit from having iterator and iterable separately. So you define only one object, which is also its own iterator. (Python does not really care that _i handed out by obj wasn't all that shiny, but just the obj itself.)

This is why in most examples I've seen (and what had been confusing me over and over), you can see:

class IterableExample(object):

    def __iter__(self):
        return self

    def next(self):
        pass

instead of

class Iterator(object):
    def next(self):
        pass

class Iterable(object):
    def __iter__(self):
        return Iterator()

There are cases, though, when you can benefit from having iterator separated from the iterable, such as when you want to have one row of items, but more "cursors". For example when you want to work with "current" and "forthcoming" elements, you can have separate iterators for both. Or multiple threads pulling from a huge list: each can have its own iterator to traverse over all items. See @Raymond's and @glglgl's answers above.

Imagine what you could do:

class SmartIterableExample(object):

    def create_iterator(self):
        # An amazingly powerful yet simple way to create arbitrary
        # iterator, utilizing object state (or not, if you are fan
        # of functional), magic and nuclear waste--no kittens hurt.
        pass    # don't forget to add the next() method

    def __iter__(self):
        return self.create_iterator()

Notes:

  • I'll repeat again: iterator is not iterable. Iterator cannot be used as a "source" in for loop. What for loop primarily needs is __iter__() (that returns something with next()).

  • Of course, for is not the only iteration loop, so above applies to some other constructs as well (while...).

  • Iterator's next() can throw StopIteration to stop iteration. Does not have to, though, it can iterate forever or use other means.

  • In the above "thought process", _i does not really exist. I've made up that name.

  • There's a small change in Python 3.x: next() method (not the built-in) now must be called __next__(). Yes, it should have been like that all along.

  • You can also think of it like this: iterable has the data, iterator pulls the next item

Disclaimer: I'm not a developer of any Python interpreter, so I don't really know what the interpreter "thinks". The musings above are solely demonstration of how I understand the topic from other explanations, experiments and real-life experience of a Python newbie.

@Racing Tadpole 2017-03-21 04:17:57

This is great - but I'm still a little confused. I thought your yellow box was saying that a for loop needs an iterator ("Look, a for loop. Looks like a job for an iterator... Let's get one."). But then you say in the notes at the end that "Iterator cannot be used as a source in a for loop"...?

@nealmcb 2017-04-10 16:34:34

Why do you put just pass in the code for those next definitions? I'll assume you just mean that someone has to implement a way to get the next one, since next has to return something.

@Alois Mahdal 2017-04-12 03:18:30

@nealmcb Yes, I think that's what past me meant. (That's what pass is for, after all.)

@nealmcb 2017-04-21 12:46:32

@AloisMahdal Ahh, I hadn't seen that use before. When I see pass, I think it is there for syntactic reasons. I just ran across the answers at ellipsis object which are quite interesting: you can use ... to indicate a "todo later" block. NotImplemented is also available.

@Rich 2018-11-24 10:15:52

While I like that you're stressing the distinction between an iterator and an iterable, this answer contradicts itself. First you write, 'Iterators are themselves also iterable', (which matches what is written in the Python documentation). But then later on you write: 'iterator is not iterable. Iterator cannot be used as a "source" in for loop'. I get the point of your answer, and like it otherwise, but I think it would benefit from fixing this.

@Nikolay Dudaev 2015-09-30 12:25:25

I don’t know if it helps anybody but I always like to visualize concepts in my head to better understand them. So as I have a little son I visualize iterable/iterator concept with bricks and white paper.

Suppose we are in the dark room and on the floor we have bricks for my son. Bricks of different size, color, does not matter now. Suppose we have 5 bricks like those. Those 5 bricks can be described as an object – let’s say bricks kit. We can do many things with this bricks kit – can take one and then take second and then third, can change places of bricks, put first brick above the second. We can do many sorts of things with those. Therefore this bricks kit is an iterable object or sequence as we can go through each brick and do something with it. We can only do it like my little son – we can play with one brick at a time. So again I imagine myself this bricks kit to be an iterable.

Now remember that we are in the dark room. Or almost dark. The thing is that we don’t clearly see those bricks, what color they are, what shape etc. So even if we want to do something with them – aka iterate through them – we don’t really know what and how because it is too dark.

What we can do is near to first brick – as element of a bricks kit – we can put a piece of white fluorescent paper in order for us to see where the first brick-element is. And each time we take a brick from a kit, we replace the white piece of paper to a next brick in order to be able to see that in the dark room. This white piece of paper is nothing more than an iterator. It is an object as well. But an object with what we can work and play with elements of our iterable object – bricks kit.

That by the way explains my early mistake when I tried the following in an IDLE and got a TypeError:

 >>> X = [1,2,3,4,5]
 >>> next(X)
 Traceback (most recent call last):
    File "<pyshell#19>", line 1, in <module>
      next(X)
 TypeError: 'list' object is not an iterator

List X here was our bricks kit but NOT a white piece of paper. I needed to find an iterator first:

>>> X = [1,2,3,4,5]
>>> bricks_kit = [1,2,3,4,5]
>>> white_piece_of_paper = iter(bricks_kit)
>>> next(white_piece_of_paper)
1
>>> next(white_piece_of_paper)
2
>>>

Don’t know if it helps, but it helped me. If someone could confirm/correct visualization of the concept, I would be grateful. It would help me to learn more.

@glglgl 2012-03-27 06:14:51

An iterable is a object which has a __iter__() method. It can possibly iterated over several times, such as list()s and tuple()s.

An iterator is the object which iterates. It is returned by an __iter__() method, returns itself via its own __iter__() method and has a next() method (__next__() in 3.x).

Iteration is the process of calling this next() resp. __next__() until it raises StopIteration.

Example:

>>> a = [1, 2, 3] # iterable
>>> b1 = iter(a) # iterator 1
>>> b2 = iter(a) # iterator 2, independent of b1
>>> next(b1)
1
>>> next(b1)
2
>>> next(b2) # start over, as it is the first call to b2
1
>>> next(b1)
3
>>> next(b1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>> b1 = iter(a) # new one, start over
>>> next(b1)
1

@thechrishaddad 2012-03-27 06:17:41

So really it's just a object that passes through container's? were would this be useful?

@glglgl 2012-03-27 06:18:42

Often, but not always. A generator, file or database cursor can only be iterated once and thus are their own iterators.

@Bin 2016-11-04 16:11:55

I guess b2 doesn't have to independent of b1 ? for this special case, it's independent, sure I can make it not independent but also a valid Iterable.

@glglgl 2016-11-05 09:41:10

@Bin Yes. As and Iterator is always Iterable and is its own Iterator, two calls of iter() do not necessarily give two independent Iterators.

@agf 2012-03-27 06:15:50

Iteration is a general term for taking each item of something, one after another. Any time you use a loop, explicit or implicit, to go over a group of items, that is iteration.

In Python, iterable and iterator have specific meanings.

An iterable is an object that has an __iter__ method which returns an iterator, or which defines a __getitem__ method that can take sequential indexes starting from zero (and raises an IndexError when the indexes are no longer valid). So an iterable is an object that you can get an iterator from.

An iterator is an object with a next (Python 2) or __next__ (Python 3) method.

Whenever you use a for loop, or map, or a list comprehension, etc. in Python, the next method is called automatically to get each item from the iterator, thus going through the process of iteration.

A good place to start learning would be the iterators section of the tutorial and the iterator types section of the standard types page. After you understand the basics, try the iterators section of the Functional Programming HOWTO.

@jlh 2017-10-24 20:51:11

It confuses me a lot that anything with a __getitem__ method is considered iterable, but any existing __len__ method is completely ignored. Principle of least astonishment failing hard here...

@Janus Troelsen 2018-07-27 09:33:25

Note that collections.abc.AsyncIterator tests for __aiter__ and __anext__ methods. This is a new addition in 3.6.

@shadowtalker 2018-09-19 06:45:54

@jlh why would __len__ be necessarily tied to iteration? How would knowing the length of something help you iterate over it?

@jlh 2018-09-19 10:46:02

@shadowtalker it would help to know which indexes are valid, so you know which indexes can be used with __getitem__.

@shadowtalker 2018-09-19 16:05:15

@jlh it sounds like you are proposing a very opinionated dfeault behavior. Consider that {'a': 'hi', 'b': 'bye'} has length of 2, but cannot be indexed by 0, 1, or 2.

@Rich 2018-11-24 09:59:59

@shadowtalker. But a dict has an __iter__ method. I think jlh is referring to objects that are iterable specifically because they define: "a __getitem__ method that can take sequential indexes starting from zero".

@chepner 2019-04-10 15:58:53

@Rich You can define such a __getitem__ without supporting the length protocol. len simply has nothing to do with iteration.

@Rich 2019-04-10 16:06:46

@chepner Yes. That's what jlh is saying they find odd. A different implementation of Iteration via __getitem__ could check the len to see which numerical indices are valid, and jlh assumed that was how it was implemented ... till they found out it wasn't.

@chepner 2019-04-10 16:42:35

You don't need to know the upper limit, though. Call __getitem__ until it raises an exception, then raise StopIteration in its place. __len__ is ignored because it's not needed with __getitem__ and not useful without it.

@kaya3 2020-01-22 00:05:08

Might be worth noting that "iteration" has another sense when used as a countable noun: it means an individual time that a repeated statement is executed. For example "on each iteration of the outer loop, the inner loop will complete 10 iterations".

@user93097373 2014-07-15 08:59:16

In Python everything is an object. When an object is said to be iterable, it means that you can step through (i.e. iterate) the object as a collection.

Arrays for example are iterable. You can step through them with a for loop, and go from index 0 to index n, n being the length of the array object minus 1.

Dictionaries (pairs of key/value, also called associative arrays) are also iterable. You can step through their keys.

Obviously the objects which are not collections are not iterable. A bool object for example only have one value, True or False. It is not iterable (it wouldn't make sense that it's an iterable object).

Read more. http://www.lepus.org.uk/ref/companion/Iterator.xml

@Mark Amery 2014-10-16 22:58:25

objects which are not collections are not iterable is not generally true. To give just a couple of examples, generators are iterable but are not collections, and the iterator objects created by calling iter() on the standard collection types are iterable but are not, themselves, collections.

@Kimvais 2012-03-27 06:15:39

I don't think that you can get it much simpler than the documentation, however I'll try:

  • Iterable is something that can be iterated over. In practice it usually means a sequence e.g. something that has a beginning and an end and some way to go through all the items in it.
  • You can think Iterator as a helper pseudo-method (or pseudo-attribute) that gives (or holds) the next (or first) item in the iterable. (In practice it is just an object that defines the method next())

  • Iteration is probably best explained by the Merriam-Webster definition of the word :

b : the repetition of a sequence of computer instructions a specified number of times or until a condition is met — compare recursion

Related Questions

Sponsored Content

37 Answered Questions

[SOLVED] What is dependency injection?

18 Answered Questions

[SOLVED] What are metaclasses in Python?

39 Answered Questions

[SOLVED] What does the "yield" keyword do?

42 Answered Questions

[SOLVED] How do I efficiently iterate over each entry in a Java Map?

10 Answered Questions

[SOLVED] Iterating over dictionaries using 'for' loops

10 Answered Questions

[SOLVED] Why is "1000000000000000 in range(1000000000000001)" so fast in Python 3?

29 Answered Questions

[SOLVED] What does if __name__ == "__main__": do?

15 Answered Questions

[SOLVED] What is the meaning of a single and a double underscore before an object name?

14 Answered Questions

[SOLVED] What is the meaning of "POSIX"?

Sponsored Content