We Keep Coding

Why a return value in a generator don't return me a value? [duplicate]

Since Python 3.3, if a generator function returns a value, that becomes the value for the StopIteration exception that is raised. This can be collected a number of ways:
The value of a yield from expression, which implies the enclosing function is also a generator.
Wrapping a call to next() or .send() in a try/except block.
However, if I'm simply wanting to iterate over the generator in a for loop - the easiest way - there doesn't appear to be a way to collect the value of the StopIteration exception, and thus the return value. Im using a simple example where the generator yields values, and returns some kind of summary at the end (running totals, averages, timing statistics, etc).
for i in produce_values():
do_something(i)
values_summary = ....??
One way is to handle the loop myself:
values_iter = produce_values()
try:
while True:
i = next(values_iter)
do_something(i)
except StopIteration as e:
values_summary = e.value
But this throws away the simplicity of the for loop. I can't use yield from since that requires the calling code to be, itself, a generator. Is there a simpler way than the roll-ones-own for loop shown above?

You can think of the value attribute of StopIteration (and arguably StopIteration itself) as implementation details, not designed to be used in "normal" code.
Have a look at PEP 380 that specifies the yield from feature of Python 3.3: It discusses that some alternatives of using StopIteration to carry the return value where considered.
Since you are not supposed to get the return value in an ordinary for loop, there is no syntax for it. The same way as you are not supposed to catch the StopIteration explicitly.
A nice solution for your situation would be a small utility class (might be useful enough for the standard library):
class Generator:
def __init__(self, gen):
self.gen = gen
def __iter__(self):
self.value = yield from self.gen
This wraps any generator and catches its return value to be inspected later:
>>> def test():
... yield 1
... return 2
...
>>> gen = Generator(test())
>>> for i in gen:
... print(i)
...
1
>>> print(gen.value)
2

You could make a helper wrapper, that would catch the StopIteration and extract the value for you:
from functools import wraps
class ValueKeepingGenerator(object):
def __init__(self, g):
self.g = g
self.value = None
def __iter__(self):
self.value = yield from self.g
def keep_value(f):
#wraps(f)
def g(*args, **kwargs):
return ValueKeepingGenerator(f(*args, **kwargs))
return g
#keep_value
def f():
yield 1
yield 2
return "Hi"
v = f()
for x in v:
print(x)
print(v.value)

A light-weight way to handle the return value (one that doesn't involve instantiating an auxiliary class) is to use dependency injection.
Namely, one can pass in the function to handle / act on the return value using the following wrapper / helper generator function:
def handle_return(generator, func):
returned = yield from generator
func(returned)
For example, the following--
def generate():
yield 1
yield 2
return 3
def show_return(value):
print('returned: {}'.format(value))
for x in handle_return(generate(), show_return):
print(x)
results in--
1
2
returned: 3

The most obvious method I can think of for this would be a user defined type that would remember the summary for you..
>>> import random
>>> class ValueProducer:
... def produce_values(self, n):
... self._total = 0
... for i in range(n):
... r = random.randrange(n*100)
... self._total += r
... yield r
... self.value_summary = self._total/n
... return self.value_summary
...
>>> v = ValueProducer()
>>> for i in v.produce_values(3):
... print(i)
...
25
55
179
>>> print(v.value_summary)
86.33333333333333
>>>

Another light weight way sometimes appropriate is to yield the running summary in every generator step in addition to your primary value in a tuple. The loop stays simple with an extra binding which is still available afterwards:
for i, summary in produce_values():
do_something(i)
show_summary(summary)
This is especially useful if someone could use more than just the last summary value, e. g. updating a progress view.

Iterate twice through an object

I'm trying to create an iterable object, and when I do 1 loop it is okay, but when doing multiple loops, it doesn't work. Here is my simplified code:
class test():
def __init__(self):
self.n = 0
def __iter__(self):
return self
def __next__(self):
if self.n < len(self)-1:
self.n += 1
return self.n
else:
raise StopIteration
def __len__(self):
return 5
#this is an example iteration
test = test()
for i in test:
for j in test:
print(i,j)
#it prints is
1 2
1 3
1 4
#What i expect is
1 1
1 2
1 3
1 4
2 1
2 2
2 3
...
4 3
4 4
How can I make this object (in this case test) to iterate twice and get all the combinations of number i and j in the example loop?

You want an instance of test to be iterable, but not its own iterator. What's the difference?
An iterable is something that, upon request, can supply an iterator. Lists are iterable, because iter([1,2,3]) returns a new listiterator object (not the list itself). To make test iterable, you just need to supply an __iter__ method (more on how to define it in a bit).
An iterator is something that, upon request, can produce a new element. It does this by calling its __next__ method. An iterator can be thought of as two pieces of information: a sequence of items to produce, and a cursor indicating how far along that sequence it currently is. When it reaches the end of its sequence, it raises a StopIteration exception to indicate that the iteration is at an end. To make an instance an iterator, you supply a __next__ method in its class. An iterator should also have a __iter__ method that just returns itself.
So how do you make test iterable without being an iterator? By having its __iter__ method return a new iterator each time it is called, and getting rid of its __next__ method. The simplest way to do that is to make __iter__ a generator function. Define your class something like:
class Test():
def __init__(self):
self._size = 5
def __iter__(self):
n = 0
while n < self._size:
yield n
n += 1
def __len__(self):
return self._size
Now when you write
test = Test()
for i in test: # implicit call to iter(test)
for j in test: # implicit call to iter(test)
print(i, j)
i and j both draw values from separate iterators over the same iterable. Each call to test.__iter__ returns a different generator object that keeps track of its own n.

Take a look at itertools.product.
You should be able to accomplish what you're looking for:
from itertools import product
...
test = test()
for i, j in product(test, repeat=2):
print(i,j)
I love this library!

Python class and __iter__

What are the benefits of using the __iter__ function in a Python class?
In the code below I am just setting up two simple classes. The first class takes in a list as an argument, and I am able to loop over this list without using the __iter__ function. The second bit of code uses the __iter__ function to loop over a list.
What is the benefit of using __iter__ when there are already ways of looping over stuff in a class?
EG 1: no __iter__
class test_class:
def __init__(self, list):
self.container_list = list
def print (self):
a = self.container_list
return a
test_list = test_class([1,2,3,4,5,6,7])
x = test_class.print(test_list)
for i in x:
print (i)
EG 2: yes __iter__
class list_using_iter:
def __init__(self):
self.list = [1,2,3,4]
self.index = -1
def __iter__(self):
return self
def __next__(self):
self.index += 1
if self.index == len(self.list):
raise StopIteration
return self.list [self.index]
r = list_using_iter()
itr = iter(r)
print(next(itr))
print(next(itr))
print(next(itr))
print(next(itr))
print(next(itr)) # Raises the exception!

Your first example is not iterable, but contains an attribute that is. Your second example is iterable, but you iterate simply by "following" another iterable. Here's an example of a iterable that does more work itself:
import itertools
class Fibs:
def __init__(self, a, b):
self.a = a
self.b = b
def __iter__(self):
a = self.a
b = self.b
while True:
yield a
a, b = b, a + b
real_fibs = Fibs(0,1)
for i in itertools.islice(real_fibs, 10):
print(i)
Fibs.__iter__ isn't simply regurgitating values obtained from some other value's __iter__ method; it is computing and yielding new values on demand.
Actually, the preceding is an example of a class that knows how to create its own iterator, rather than having each object be iterable. Here's a version that defines next itself.
class Fibs:
def __init__(self, a, b):
self.a = a
self.b = b
def __iter__(self):
return self
def __next__(self):
rv = self.a
self.a, self.b = self.b, self.a + self.b
return rv

In both cases, the looping works because of __iter__. In your first example, your print function returns a loop.
The implementation of the for keyword will call __iter__ (or the corresponding slot within the C implementation since the code involved is in the C interpreter) in order to loop over the list.
In your second example you could have written
for elt in r:
print(elt)
which would have internally called __iter__ to implement the for loop.
In general you tend to use for rather than iter and next directly. The cases where you use iter and next directly are when you're producing a callback function that will produce an iterator or when you're defining one iterator in terms of another.
In terms of when should you write your own __iter__ or return some object that does its own iteration, that all depends on what functionality you want. For example, your first class is more powerful because two people can be iterating the list at the same time. In your second class, because you store the index in the class itself, only one person can successfully use the iterator at a time.
However, if you had complex enough behavior, the second approach where you define your own __iter__ might make sense.

How to implement "next" for a dictionary object to be iterable?

I've got the following wrapper for a dictionary:
class MyDict:
def __init__(self):
self.container = {}
def __setitem__(self, key, value):
self.container[key] = value
def __getitem__(self, key):
return self.container[key]
def __iter__(self):
return self
def next(self):
pass
dic = MyDict()
dic['a'] = 1
dic['b'] = 2
for key in dic:
print key
My problem is that I don't know how to implement the next method to make MyDict iterable. Any advice would be appreciated.

Dictionaries are themselves not an iterator (which can only be iterated over once). You usually make them an iterable, an object for which you can produce multiple iterators instead.
Drop the next method altogether, and have __iter__ return an iterable object each time it is called. That can be as simple as just returning an iterator for self.container:
def __iter__(self):
return iter(self.container)
If you must make your class an iterator, you'll have to somehow track a current iteration position and raise StopIteration once you reach the 'end'. A naive implementation could be to store the iter(self.container) object on self the first time __iter__ is called:
def __iter__(self):
return self
def next(self):
if not hasattr(self, '_iter'):
self._iter = iter(self.container)
return next(self._iter)
at which point the iter(self.container) object takes care of tracking iteration position for you, and will raise StopIteration when the end is reached. It'll also raise an exception if the underlying dictionary was altered (had keys added or deleted) and iteration order has been broken.
Another way to do this would be to just store in integer position and index into list(self.container) each time, and simply ignore the fact that insertion or deletion can alter the iteration order of a dictionary:
_iter_index = 0
def __iter__(self):
return self
def next(self):
idx = self._iter_index
if idx is None or idx >= len(self.container):
# once we reach the end, all iteration is done, end of.
self._iter_index = None
raise StopIteration()
value = list(self.container)[idx]
self._iter_index = idx + 1
return value
In both cases your object is then an iterator that can only be iterated over once. Once you reach the end, you can't restart it again.

If you want to be able to use your dict-like object inside nested loops, for example, or any other application that requires multiple iterations over the same object, then you need to implement an __iter__ method that returns a newly-created iterator object.
Python's iterable objects all do this:
>>> [1, 2, 3].__iter__()
<listiterator object at 0x7f67146e53d0>
>>> iter([1, 2, 3]) # A simpler equivalent
<listiterator object at 0x7f67146e5390>
The simplest thing for your objects' __iter__ method to do would be to return an iterator on the underlying dict, like this:
def __iter__(self):
return iter(self.container)
For more detail than you probably will ever require, see this Github repository.

How to build a basic iterator?

How would one create an iterative function (or iterator object) in python?

Iterator objects in python conform to the iterator protocol, which basically means they provide two methods: __iter__() and __next__().
The __iter__ returns the iterator object and is implicitly called
at the start of loops.
The __next__() method returns the next value and is implicitly called at each loop increment. This method raises a StopIteration exception when there are no more value to return, which is implicitly captured by looping constructs to stop iterating.
Here's a simple example of a counter:
class Counter:
def __init__(self, low, high):
self.current = low - 1
self.high = high
def __iter__(self):
return self
def __next__(self): # Python 2: def next(self)
self.current += 1
if self.current < self.high:
return self.current
raise StopIteration
for c in Counter(3, 9):
print(c)
This will print:
3
4
5
6
7
8
This is easier to write using a generator, as covered in a previous answer:
def counter(low, high):
current = low
while current < high:
yield current
current += 1
for c in counter(3, 9):
print(c)
The printed output will be the same. Under the hood, the generator object supports the iterator protocol and does something roughly similar to the class Counter.
David Mertz's article, Iterators and Simple Generators, is a pretty good introduction.

There are four ways to build an iterative function:
create a generator (uses the yield keyword)
use a generator expression (genexp)
create an iterator (defines __iter__ and __next__ (or next in Python 2.x))
create a class that Python can iterate over on its own (defines __getitem__)
Examples:
# generator
def uc_gen(text):
for char in text.upper():
yield char
# generator expression
def uc_genexp(text):
return (char for char in text.upper())
# iterator protocol
class uc_iter():
def __init__(self, text):
self.text = text.upper()
self.index = 0
def __iter__(self):
return self
def __next__(self):
try:
result = self.text[self.index]
except IndexError:
raise StopIteration
self.index += 1
return result
# getitem method
class uc_getitem():
def __init__(self, text):
self.text = text.upper()
def __getitem__(self, index):
return self.text[index]
To see all four methods in action:
for iterator in uc_gen, uc_genexp, uc_iter, uc_getitem:
for ch in iterator('abcde'):
print(ch, end=' ')
print()
Which results in:
A B C D E
A B C D E
A B C D E
A B C D E
Note:
The two generator types (uc_gen and uc_genexp) cannot be reversed(); the plain iterator (uc_iter) would need the __reversed__ magic method (which, according to the docs, must return a new iterator, but returning self works (at least in CPython)); and the getitem iteratable (uc_getitem) must have the __len__ magic method:
# for uc_iter we add __reversed__ and update __next__
def __reversed__(self):
self.index = -1
return self
def __next__(self):
try:
result = self.text[self.index]
except IndexError:
raise StopIteration
self.index += -1 if self.index < 0 else +1
return result
# for uc_getitem
def __len__(self)
return len(self.text)
To answer Colonel Panic's secondary question about an infinite lazily evaluated iterator, here are those examples, using each of the four methods above:
# generator
def even_gen():
result = 0
while True:
yield result
result += 2
# generator expression
def even_genexp():
return (num for num in even_gen()) # or even_iter or even_getitem
# not much value under these circumstances
# iterator protocol
class even_iter():
def __init__(self):
self.value = 0
def __iter__(self):
return self
def __next__(self):
next_value = self.value
self.value += 2
return next_value
# getitem method
class even_getitem():
def __getitem__(self, index):
return index * 2
import random
for iterator in even_gen, even_genexp, even_iter, even_getitem:
limit = random.randint(15, 30)
count = 0
for even in iterator():
print even,
count += 1
if count >= limit:
break
print
Which results in (at least for my sample run):
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
How to choose which one to use? This is mostly a matter of taste. The two methods I see most often are generators and the iterator protocol, as well as a hybrid (__iter__ returning a generator).
Generator expressions are useful for replacing list comprehensions (they are lazy and so can save on resources).
If one needs compatibility with earlier Python 2.x versions use __getitem__.

I see some of you doing return self in __iter__. I just wanted to note that __iter__ itself can be a generator (thus removing the need for __next__ and raising StopIteration exceptions)
class range:
def __init__(self,a,b):
self.a = a
self.b = b
def __iter__(self):
i = self.a
while i < self.b:
yield i
i+=1
Of course here one might as well directly make a generator, but for more complex classes it can be useful.

First of all the itertools module is incredibly useful for all sorts of cases in which an iterator would be useful, but here is all you need to create an iterator in python:
yield
Isn't that cool? Yield can be used to replace a normal return in a function. It returns the object just the same, but instead of destroying state and exiting, it saves state for when you want to execute the next iteration. Here is an example of it in action pulled directly from the itertools function list:
def count(n=0):
while True:
yield n
n += 1
As stated in the functions description (it's the count() function from the itertools module...) , it produces an iterator that returns consecutive integers starting with n.
Generator expressions are a whole other can of worms (awesome worms!). They may be used in place of a List Comprehension to save memory (list comprehensions create a list in memory that is destroyed after use if not assigned to a variable, but generator expressions can create a Generator Object... which is a fancy way of saying Iterator). Here is an example of a generator expression definition:
gen = (n for n in xrange(0,11))
This is very similar to our iterator definition above except the full range is predetermined to be between 0 and 10.
I just found xrange() (suprised I hadn't seen it before...) and added it to the above example. xrange() is an iterable version of range() which has the advantage of not prebuilding the list. It would be very useful if you had a giant corpus of data to iterate over and only had so much memory to do it in.

This question is about iterable objects, not about iterators. In Python, sequences are iterable too so one way to make an iterable class is to make it behave like a sequence, i.e. give it __getitem__ and __len__ methods. I have tested this on Python 2 and 3.
class CustomRange:
def __init__(self, low, high):
self.low = low
self.high = high
def __getitem__(self, item):
if item >= len(self):
raise IndexError("CustomRange index out of range")
return self.low + item
def __len__(self):
return self.high - self.low
cr = CustomRange(0, 10)
for i in cr:
print(i)

If you looking for something short and simple, maybe it will be enough for you:
class A(object):
def __init__(self, l):
self.data = l
def __iter__(self):
return iter(self.data)
example of usage:
In [3]: a = A([2,3,4])
In [4]: [i for i in a]
Out[4]: [2, 3, 4]

All answers on this page are really great for a complex object. But for those containing builtin iterable types as attributes, like str, list, set or dict, or any implementation of collections.Iterable, you can omit certain things in your class.
class Test(object):
def __init__(self, string):
self.string = string
def __iter__(self):
# since your string is already iterable
return (ch for ch in self.string)
# or simply
return self.string.__iter__()
# also
return iter(self.string)
It can be used like:
for x in Test("abcde"):
print(x)
# prints
# a
# b
# c
# d
# e

Include the following code in your class code.
def __iter__(self):
for x in self.iterable:
yield x
Make sure that you replace self.iterablewith the iterable which you iterate through.
Here's an example code
class someClass:
def __init__(self,list):
self.list = list
def __iter__(self):
for x in self.list:
yield x
var = someClass([1,2,3,4,5])
for num in var:
print(num)
Output
1
2
3
4
5
Note: Since strings are also iterable, they can also be used as an argument for the class
foo = someClass("Python")
for x in foo:
print(x)
Output
P
y
t
h
o
n

This is an iterable function without yield. It make use of the iter function and a closure which keeps it's state in a mutable (list) in the enclosing scope for python 2.
def count(low, high):
counter = [0]
def tmp():
val = low + counter[0]
if val < high:
counter[0] += 1
return val
return None
return iter(tmp, None)
For Python 3, closure state is kept in an immutable in the enclosing scope and nonlocal is used in local scope to update the state variable.
def count(low, high):
counter = 0
def tmp():
nonlocal counter
val = low + counter
if val < high:
counter += 1
return val
return None
return iter(tmp, None)
Test;
for i in count(1,10):
print(i)
1
2
3
4
5
6
7
8
9

class uc_iter():
def __init__(self):
self.value = 0
def __iter__(self):
return self
def __next__(self):
next_value = self.value
self.value += 2
return next_value
Improving previous answer, one of the advantage of using class is that you can add __call__ to return self.value or even next_value.
class uc_iter():
def __init__(self):
self.value = 0
def __iter__(self):
return self
def __next__(self):
next_value = self.value
self.value += 2
return next_value
def __call__(self):
next_value = self.value
self.value += 2
return next_value
c = uc_iter()
print([c() for _ in range(10)])
print([next(c) for _ in range(5)])
# [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
# [20, 22, 24, 26, 28]
Other example of a class based on Python Random that can be both called and iterated could be seen on my implementation here