So I've been writing iterators for a while, and I thought that I understood them. But I've been struggling with some issues tonight, and the more I play with it, the more confused I become.
I thought that for an iterator you had to implement __iter__ and next (or __next__). And that when you first tried to iterate over the iterator the __iter__ method would be called, and then next would be called until an StopIteration was raised.
When I run this code though
class Iter(object):
def __iter__(self):
return iter([2, 4, 6])
def next(self):
for y in [1, 2, 3]:
return y
iterable = Iter()
for x in iterable:
print(x)
The output is 2 4 6. So __iter__ is being called, but not next. That does seem to match with the documentation that I found here. But then that raises a whole bunch more questions in my mind.
Specifically, what's the difference between a container type and iterator if it's not the implementation of next? How do I know before hand which way my class is going to be treated? And most importantly, if I want to write a class where my next method is called when I use for x in Iter(), how can I do that?
A list is iterable, but it is not an iterator. Compare and contrast:
>>> type([])
list
>>> type(iter([]))
list_iterator
Calling iter on a list creates and returns a new iterator object for iterating the contents of that list.
In your object, you just return a list iterator, specifically an iterator over the list [2, 4, 6], so that object knows nothing about yielding elements 1, 2, 3.
def __iter__(self):
return iter([2, 4, 6]) # <-- you're returning the list iterator, not your own
Here's a more fundamental implementation conforming to the iterator protocol in Python 2, which doesn't confuse matters by relying on list iterators, generators, or anything fancy at all.
class Iter(object):
def __iter__(self):
self.val = 0
return self
def next(self):
self.val += 1
if self.val > 3:
raise StopIteration
return self.val
According to the documentation you link to, an iterator's __iter__ method is supposed to return itself. So your iterator isn't really an iterator at all: when for invokes __iter__ to get the iterator, you're giving it iter([2,4,6]), but you should be giving it self.
(Also, I don't think your next method does what you intend: it returns 1 every time it's called, and never raises StopIteration. So if you fixed __iter__, then your iterator would be an iterator over an infinite stream of ones, rather than over the finite list [1, 2, 3]. But that's a side-issue.)
Related
I have seen a lot of examples where the __iter__ method returns self, and I've done my own example:
class Transactions:
def __init__(self):
self.t = [1,2,9,12.00]
self.idx = 0
def __iter__(self):
return self
def __next__(self):
pos = self.idx
self.idx += 1
try:
return self.t[pos]
except IndexError:
raise StopIteration
>>> list(iter(Transactions()))
[1, 2, 9, 12.0]
How does "returning self" make the object iterable? What exactly does that do?
What you are making is both an Iterator and an Iterable. The instances of the Transactions class will have both the __iter__ which makes it an Iterable and __next__, which makes it an Iterator.
So when you return self from __iter__ you are basically indicating that the returned object is an Iterable as it has the __iter__, and since on calling __iter__, it must return an instance of an Iterator therefore you have defined the __next__ for the same instance so that it behaves like one.
This is highlighted in the documentation:
Iterable
An object capable of returning its members one at a time. Examples of
iterables include all sequence types (such as list, str, and tuple)
and some non-sequence types like dict, file objects, and objects of
any classes you define with an iter() method or with a
getitem() method that implements Sequence semantics.
Iterables can be used in a for loop and in many other places where a sequence is needed (zip(), map(), …). When an iterable object
is passed as an argument to the built-in function iter(), it returns
an iterator for the object. This iterator is good for one pass over
the set of values. When using iterables, it is usually not necessary
to call iter() or deal with iterator objects yourself. The for
statement does that automatically for you, creating a temporary
unnamed variable to hold the iterator for the duration of the loop.
See also iterator, sequence, and generator.
Iterator
An object representing a stream of data. Repeated calls to the
iterator’s __next__() method (or passing it to the built-in function
next()) return successive items in the stream. When no more data are
available a StopIteration exception is raised instead. At this point,
the iterator object is exhausted and any further calls to its
next() method just raise StopIteration again. Iterators are required to have an iter() method that returns the iterator object
itself so every iterator is also iterable and may be used in most
places where other iterables are accepted. One notable exception is
code which attempts multiple iteration passes. A container object
(such as a list) produces a fresh new iterator each time you pass it
to the iter() function or use it in a for loop. Attempting this with
an iterator will just return the same exhausted iterator object used
in the previous iteration pass, making it appear like an empty
container.
A nice answer was given, I'd like to show a practical example.
The Transaction defined in the question is an iterator. It can be iterated over just once.
While the typical iteration has the form for x in ...:, let's continue to use the shorter list() for demonstration:
>>> t=Transactions()
>>> list(t)
[1, 2, 9, 12.0]
>>> list(t)
[]
>>> list(t)
[]
For a real class, this is not what people expect. In order to iterate over the data every time, a new iterator must be created for each iteration, making the original class an iterable:
class TransactionsIterator:
def __init__(self, t):
self.t = t
self.idx = 0
def __iter__(self):
return self
def __next__(self):
pos = self.idx
self.idx += 1
try:
return self.t[pos]
except IndexError:
raise StopIteration
class Transactions:
def __init__(self):
self.t = [1,2,9,12.00]
def __iter__(self):
return TransactionsIterator(self.t)
This behaves as other classes usually do:
>>> t=Transactions()
>>> list(t)
[1, 2, 9, 12.0]
>>> list(t)
[1, 2, 9, 12.0]
>>> list(t)
[1, 2, 9, 12.0]
>>> list(t)
and finally you don't have to reinvent a list iterator, this is all you need:
class Transactions:
def __init__(self):
self.t = [1,2,9,12.00]
def __iter__(self):
return iter(self.t)
Back to the question. We can iterate over data once with an iterator and every time with an iterable. The whole point of iterators returning self in __iter__ is that the iteration code does not have to distinguish between these two cases.
Let's say I have a class which implements an __iter__() function, is it preferred to use iter(obj) or calling obj.__iter__() directly? Are there any real differences besides having to type 5 characters less with the magic function?
In contrast: For next() and __next__() I can see an advantage for having a default value with the magic function.
The difference is mostly just convenience. It's less typing and less symbols to read, and so faster to read. However, the various builtin functions (eg. iter, len et al.) usually do a little type checking to catch errors early. If you wrote a customer __iter__ method and it returned 2, then invoking obj.__iter__() wouldn't catch that, but iter(obj) throws a type error. eg.
>>> class X:
def __iter__(self):
return 2
>>> x = X()
>>> x.__iter__()
2
>>> iter(x)
Traceback (most recent call last):
File "<pyshell#37>", line 1, in <module>
iter(x)
TypeError: iter() returned non-iterator of type 'int'
iter also implements the iterator protocol for objects that have no __iter__, but do implement the sequence protocol. That is, they have a __getitem__ method which implements a sequence starting at index 0 and raises an IndexError for indexes not in bounds. This is an older feature of python and not really something new code should be using. eg.
>>> class Y:
def __getitem__(self, index):
if 0 <= index < 5:
return index ** 2
else:
raise IndexError(index)
>>> list(iter(Y())) # iter not strictly needed here
[0, 1, 4, 9, 16]
When should you use __iter__? This might not be so relevant to __iter__, but if you need access to the implementation of method that the parent class uses then it is best to invoke such methods in the style super().__<dunder_method>__() (using Python 3 style super usage). eg.
>>> class BizzareList(list):
def __iter__(self):
for item in super().__iter__():
yield item * 10
>>> l = BizzareList(range(5))
>>> l # normal access
[0, 1, 2, 3, 4]
>>> l[0] # also normal access
0
>>> tuple(iter(l)) # iter not strictly needed here
(0, 10, 20, 30, 40)
The iter() function (which in turn calls the iter() method) is a python built-in function that returns an
iterator from them. You can opt to use iter(obj) or obj.__iter__(), they are simply the same thing. Its just
that __iter__() is a user-defined function and calling it will just do what the written code asks it to do.
On the other hand, calling the iter() function will in turn call the user-defined __iter__() function and run
the same code. However, iter() will also run a built-in 'type' checking, in which if the customized __iter__()
doesn't return an iterator object, it would throw an error.
*The same thing applies to next() & __next__().
Thus, iter(obj) ~= obj.__iter__(),
next(obj) ~= obj.__next__()
Note that __iter__() and __next__() must be defined by the user when creating a class if the object os intended
to be an iterator.
Source: https://www.programiz.com/python-programming/iterator
Look at below sample:
a = [1, 2, 3, 4]
for i in a:
print(a)
a is the list (iterable) not the iterator.
I'm not asking to know that __iter__ or iter() convert list to iterator!
I'm asking to know if for loop itself convert list implicitly then call __iter__ for iteration keeping list without removing like iterator?
Since stackoverflow identified my question as possible duplicate:
The unique part is that I'm not asking about for loop as concept nor __iter__, I'm asking about the core mechanism of for loop and relationship with iter.
I'm asking to know if for loop itself convert list implicitly then call iter for iteration keeping list without removing like iterator?
The for loop does not convert the list implicitly in the sense that it mutates the list, but it implicitly creates an iterator from the list. The list itself will not change state during iteration, but the created iterator will.
a = [1, 2, 3]
for x in a:
print(x)
is equivalent to
a = [1, 2, 3]
it = iter(a) # calls a.__iter__
while True:
try:
x = next(it)
except StopIteration:
break
print(x)
Here's proof that __iter__ actually gets called:
import random
class DemoIterable(object):
def __iter__(self):
print('__iter__ called')
return DemoIterator()
class DemoIterator(object):
def __iter__(self):
return self
def __next__(self):
print('__next__ called')
r = random.randint(1, 10)
if r == 5:
print('raising StopIteration')
raise StopIteration
return r
Iteration over a DemoIterable:
>>> di = DemoIterable()
>>> for x in di:
... print(x)
...
__iter__ called
__next__ called
9
__next__ called
8
__next__ called
10
__next__ called
3
__next__ called
10
__next__ called
raising StopIteration
Why have an __iter__ method? If an object is an iterator, then it is pointless to have a method which returns itself. If it is not an iterator but is instead an iterable, i.e something with an __iter__ and __getitem__ method, then why would one want to ever define something which returns an iterator but is not an iterator itself? In Python, when would one want to define an iterable that is not itself an iterator? Or, what is an example of something that is an iterable but not an iterator?
Trying to answer your questions one at a time:
Why have an __iter__ method? If an object is an iterator, then it is pointless to have a method which returns itself.
It's not pointless. The iterator protocol demands an __iter__ and __next__ (or next in Python 2) method. All sane iterators I have ever seen just return self in their __iter__ method, but it is still crucial to have that method. Not having it would lead to all kinds of weirdness, for example:
somelist = [1, 2, 3]
it = iter(somelist)
now
iter(it)
or
for x in it: pass
would throw a TypeError and complain that it is not iterable, because when iter(x) is called (which implicitly happens when you employ a for loop) it expects the argument object x to be able to produce an iterator (it just tries to call __iter__ on that object). Concrete example (Python 3):
>>> class A:
... def __iter__(self):
... return B()
...
>>> class B:
... def __next__(self):
... pass
...
>>> iter(iter(A()))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'B' object is not iterable
Consider any functions, escpecially from itertools that expect an iterable, for example dropwhile. Calling it with any object that has an __iter__ method will be fine, regardless of whether it's an iterable that is not an iterator, or an iterator - because you can expect the same result when calling iter with that object as an argument. Making a weird distinction between two kinds of iterables here would go against the principle of duck typing which python strongly embraces.
Neat tricks like
>>> a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> list(zip(*[iter(a)]*3))
[(1, 2, 3), (4, 5, 6), (7, 8, 9)]
would just stop working if you could not pass iterators to zip.
why would one want to ever define something which returns an iterator but is not an iterator itself
Let's consider this simple list iterator:
>>> class MyList(list):
... def __iter__(self):
... return MyListIterator(self)
>>>
>>> class MyListIterator:
... def __init__(self, lst):
... self._lst = lst
... self.index = 0
... def __iter__(self):
... return self
... def __next__(self):
... try:
... n = self._lst[self.index]
... self.index += 1
... return n
... except IndexError:
... raise StopIteration
>>>
>>> a = MyList([1,2,3])
>>> for x in a:
... for x in a:
... x
...
1
2
3
1
2
3
1
2
3
Remember that iter is called with the iterable in question for both for loops, expecting a fresh iterator each time from the object's __iter__ method.
Now, without an iterator being produced each time a for loop is employed, how would you be able to keep track of the current state of any iteration when a MyList object is iterated over an arbitrary number of times at the same time? Oh, that's right, you can't. :)
edit: Bonus and sort of a reply to Tadhg McDonald-Jensen's comment
A resuable iterator is not unthinkable, but of course a bit weird because it would rely on being initialized with a "non-consumable" iterable (i.e. not a classic iterator):
>>> class riter(object):
... def __init__(self, iterable):
... self.iterable = iterable
... self.it = iter(iterable)
... def __next__(self): # python 2: next
... try:
... return next(self.it)
... except StopIteration:
... self.it = iter(self.iterable)
... raise
... def __iter__(self):
... return self
...
>>>
>>> a = [1, 2, 3]
>>> it = riter(a)
>>> for x in it:
... x
...
1
2
3
>>> for x in it:
... x
...
1
2
3
An iterable is something that can be iterated (looped) over, where as an iterator is something that is consumed.
what is an example of something that is an iterable but not an iterator?
Simple, a list. Or any sequence, since you can iterate over a list as many times as you want without destruction to the list:
>>> a = [1,2,3]
>>> for i in a:
print(i,end=" ")
1 2 3
>>> for i in a:
print(i,end=" ")
1 2 3
Where as an iterator (like a generator) can only be used once:
>>> b = (i for i in range(3))
>>> for i in b:
print(i,end=" ")
0 1 2
>>> for i in b:
print(i,end=" ")
>>> #iterator has already been used up, nothing gets printed
For a list to be consumed like an iterator you would need to use something like self.pop(0) to remove the first element of the list for iteration:
class IteratorList(list):
def __iter__(self):
return self #since the current mechanics require this
def __next__(self):
try:
return self.pop(0)
except IndexError: #we need to raise the expected kind of error
raise StopIteration
next = __next__ #for compatibility with python 2
a = IteratorList([1,2,3,4,5])
for i in a:
print(i)
if i==3: # lets stop at three and
break # see what the list is after
print(a)
which gives this output:
1
2
3
[4, 5]
You see? This is what iterators do, once a value is returned from __next__ it has no reason to hang around in the iterator or in memory, so it is removed. That's why we need the __iter__, to define iterators that let us iterate over sequences without destroying them in the process.
In response to #timgeb's comment, I suppose if you added items to an IteratorList then iterated over it again that would make sense:
a = IteratorList([1,2,3,4,5])
for i in a:
print(i)
a.extend([6,7,8,9])
for i in a:
print(i)
But all iterators only make sense to either be consumed or never end. (like itertools.repeat)
You are thinking in the wrong direction. The reason why an iterator has to implement __iter__ is that this way, both containers and iterators can be used in for and in statement.
> # list is a container
> list = [1,2,3]
> dir(list)
[...,
'__iter__',
'__getitem__',
...]
> # let's get its iterator
> it = iter(list)
> dir(it)
[...,
'__iter__',
'__next__',
...]
> # you can use the container directly:
> for i in list:
> print(i)
1
2
3
> # you can also use the iterator directly:
> for i in it:
> print(i)
1
2
3
> # the above will fail if it does not implement '__iter__'
And that is also why you simply need to return self in almost all implementations of an iterator. It is not meant for anything funky, just a little bit easiness on syntax.
Ref: https://docs.python.org/dev/library/stdtypes.html#iterator-types
I don't understand exactly why the __iter__ special method just returns the object it's called on (if it's called on an iterator). Is it essentially just a flag indicating that the object is an iterator?
EDIT: Actually, I discovered that "This is required to allow both containers and iterators to be used with the for and in statements." https://docs.python.org/3/library/stdtypes.html#iterator.iter
Alright, here's how I understand it: When writing a for loop, you're allowed to specify either an iterable or an iterator to loop over. But Python ultimately needs an iterator for the loop, so it calls the __iter__ method on whatever it's given. If it's been given an iterable, the __iter__ method will produce an iterator, and if it's been given an iterator, the __iter__ method will likewise produce an iterator (the original object given).
When you loop over something using for x in something, then the loop actually calls iter(something) first, so it has something to work with. In general, the for loop is approximately equivalent to something like this:
something_iterator = iter(something)
while True:
try:
x = next(something_iterator)
# loop body
except StopIteration:
break
So as you already figured out yourself, in order to be able to loop over an iterator, i.e. when something is already an iterator, iterators should always return themselves when calling iter() on them. So this basically makes sure that iterators are also iterable.
This depends what object you call iter on. If an object is already an iterator, then there is no operation required to convert it to an iterator, because it already is one. But if the object is not an iterator, but is iterable, then an iterator is constructed from the object.
A good example of this is the list object:
>>> x = [1, 2, 3]
>>> iter(x) == x
False
>>> iter(x)
<list_iterator object at 0x7fccadc5feb8>
>>> x
[1, 2, 3]
Lists are iterable, but they are not themselves iterators. The result of list.__iter__ is not the original list.
In Python when ever you try to use loops, or try to iterate over any object like below..
Lets try to understand for list object..
>>> l = [1, 2, 3] # Defined list l
If we iterate over the above list..
>>> for i in l:
... print i
...
1
2
3
When you try to do this iteration over list l, Python for loop checks for l.__iter__() which intern return an iterator object.
>>> for i in l.__iter__():
... print i
...
1
2
3
To understand this more, lets customize the list and create anew list class..
>>> class ListOverride(list):
... def __iter__(self):
... raise TypeError('Not iterable')
...
Here I've created ListOverride class which intern inherited from list and overrided list.__iter__ method to raise TypeError.
>>> ll = ListOverride([1, 2, 3])
>>> ll
[1, 2, 3]
And i've created anew list using ListOverride class, and since it's list object it should iterate in the same way as list does.
>>> for i in ll:
... print i
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __iter__
TypeError: Not iterable
If we try to iterate over ListOverride object ll, we'll endup getting NotIterable exception..