Generator with parameters? - python

Is there a way to create a parametrized Generator.
I want a generator/counter that increases if I pass True and decreases if I pass False.
next(gen(True))
1
next(gen(True))
2
next(gen(False))
1
if it falls back below zero I want to end the iteration.
I'm using this generator to implement Backtracking i.e. falling below zero is the case when all options are exhausted.
reaching the to upper bound is dynamic. In case I can not integrate it i can do the check outside.
Having generator allows me to do the cycle logic as simple while-loop.
As mentioned may be I may also explore Callable object.

There is no native syntax for this (yet?), but this behavior can be mimicked:
class IntakeGen(object):
counter = None
def __init__(self, counter=0):
self.counter = counter
def __call__(self, increase):
self.counter += 1 if increase else -1
return self
def __next__(self):
return self.counter
Use:
In [7]: gen = IntakeGen()
In [8]: next(gen(True))
Out[8]: 1
In [9]: next(gen(True))
Out[9]: 2
In [10]: next(gen(False))
Out[10]: 1

If gen is a value of type generator, you can use its send method. next(gen) is essentially the same as gen.send(None) (one of which must be used to start a new generator).
def make_gen():
counter = 0
while True:
if (yield counter):
counter += 1
else:
counter -= 1
Then
>>> gen = make_gen()
>>> next(gen)
0
>>> gen.send(True)
1
>>> gen.send(True)
2
>>> gen.send(False)
1

def gen(v, start_val=[0]):
start_val[0] += 1 if v else -1
yield start_val[0]
print( next(gen(True)) )
print( next(gen(True)) )
print( next(gen(False)) )
Prints:
1
2
1

Related

Can you have member variables in a python function

I'm wondering if there's a way to have member variables in a function instance. For example,
incr_instance = incrementor(1)
incr_instance()
incr_instance()
incr_instance()
would print out
1
2
3
Yes, I know I can just use a class, but this is a question about weird language quirks in the python language.
You can have a function incrementor that returns another function that keeps incrementing the counter n (the inner function uses nonlocal n to access n instead of using incrementor.n):
def incrementor(n):
def inc():
nonlocal n
n += 1
return n - 1
return inc
Test:
incr_instance = incrementor(1)
incr_instance_2 = incrementor(10)
print(incr_instance())
print(incr_instance())
print(incr_instance())
print(incr_instance_2())
print(incr_instance_2())
print(incr_instance_2())
print(incr_instance())
print(incr_instance())
print(incr_instance())
print(incr_instance_2())
print(incr_instance_2())
print(incr_instance_2())
Output:
1
2
3
10
11
12
4
5
6
13
14
15
You can do the following:
def incrementor(n):
def incrementor_aux():
incrementor_aux.x += n
print(incrementor_aux.x)
incrementor_aux.x = 0
return incrementor_aux
You can even create multiple incrementors and have them increment their values independantly:
incr_instance1 = incrementor(1)
incr_instance2 = incrementor(1)
incr_instance1() #prints 1
incr_instance1() #prints 2
incr_instance2() #prints 1
incr_instance1() #prints 3
incr_instance2() #prints 2
What happens is a new instance of incrementor_aux function is created everytime you call incrementor, with its own x local value which stays in each instance's scope.
Here's one that doesn't use function attributes:
from functools import partial
def increment(x):
def gen():
y = x
while True:
y += 1
yield y
g = gen()
return partial(next, g)
f1 = increment(1)
f1()
# 2
f1()
# 3
f2 = increment(0)
f2()
# 1
f2()
# 2
f1()
# 4
def incrementor(i=0):
def inc():
inc.i += 1
return inc.i
inc.i = i - 1
return inc
incr_inst = incrementor(1)
incr_inst() # 1
incr_inst() # 2
But for this actual task
from itertools import count
incr = count(1)
incr() # 1
incr() # 2
Looks like you want a factory that gives you functions with local state, without using a class. Instead of making the state variable a member of the function, I'd use python's nonlocal syntax:
def incrementor(start):
state = start -1
def interior():
nonlocal state
state += 1
return state
return interior
This will print 7, then 8:
inc7 = incrementor(7)
print(inc7())
print(inc7())
This will print 1, then 9 10, then 2 3; each of the functions inc1() and inc7() is going at its own pace.
inc1 = incrementor(1)
print(inc1())
# Yes, I can mix them
print("inc7 again:", inc7(), inc7())
print("inc1 again:", inc1(), inc1())
I fancied having a go at this and tried to make it concise, here's what I ended up with. It's not best practice but you asked for quirky!
The aim here was to not pass the value back into the function, as the OP requested.
index = 0;
incr_instance = lambda: [index, exec("global index; index += 1")]
print(incr_instance()[0])
print(incr_instance()[0])
print(incr_instance()[0])
Will print
0
1
2
Index is our global which keeps track of our progress, the lambda expression executes two statements. The first statement returns the value of index and the second increments it, which had to be done using exec as you cannot assign variables from a lambda as it's for expressions only. The two statements are within square brackets in order to run both; otherwise it will return index only.
The [0] index gives us the value of index rather than the result of the exec expression (always None)

Create a function that will increment by one when called

I am needing a function that will increment by one every time it is called. I have used count but every time I do it resets the count back to the original value plus one count. I have seen lots of code but none of it works. Here is what I have now
I have done lots of looking into loops and iterations
def count_row():
count = 1
while count >= 1:
yield count
count += 1
return count
You can use itertools.count.
from itertools import count
counter = count(1)
next(counter) # 1
next(counter) # 2
Stateful function
If you absolutely want a stateful function instead of calling next, you can wrap the count in a function.
def counter(_count=count(1)):
return next(_count)
counter() # 1
counter() # 2
Class
Alternatively, itertools.count being a class, you can inherit from it to extend it's behaviour and make it a callable.
class CallableCount(count):
def __call__(self):
return next(self)
counter = CallableCount(1)
counter() # 1
counter() # 2
Using a class would be my preferred approach since it allows instantiating multiple counters.
You need a closure. Define a function make_counter which initializes a local variable, then defines and returns a function that increments that variable on each call.
def make_counter():
count = -1
def _():
count += 1
return count
return _
count_row = make_counter()
Now count_row will return a new value on each call:
>>> count_row()
0
>>> count_row()
1
This is sort of the dual of a class. You have a function that "wraps" some data (closes over the variable), instead of a piece of data with an associated method. The class version; note the similarity to make_counter:
class Count:
def __init__(self):
self.count = -1
def __call__(self):
self.count += 1
return count
An instance of this class now behaves like our previous closure.
>>> count_row = Count()
>>> count_row()
0
>>> count_row()
1
You can use a generator here that increments value by one every time it's called using next():
def count_row():
count = 0
while True:
count += 1
yield count
itr = count_row()
print(next(itr)) # 1
print(next(itr)) # 2
If you look closely, this is equivalent to what itertools.count() does.
if I got this right this should work:
count=0
def count_row(count):
count += 1
return count

Python function that produces both generator and aggregate results

What is the Pythonic way to make a generator that also produces aggregate results? In meta code, something like this (but not for real, as my Python version does not support mixing yield and return):
def produce():
total = 0
for item in find_all():
total += 1
yield item
return total
As I see it, I could:
Not make produce() a generator, but pass it a callback function to call on every item.
With every yield, also yield the aggregate results up until now. I'd rather not calculate the intermediate results with every yield, only when finishing.
Send a dict as argument to produce() that will be populated with the aggregate results.
Use a global to store aggregate results.
All of them don't seem very attractive...
NB. total is a simple example, my actual code requires complex aggregations. And I need intermediate results before produce() finishes, hence a generator.
Maybe you shouldn't use a generator but an iterator.
def findall(): # no idea what your "find_all" does so I use this instead. :-)
yield 1
yield 2
yield 3
class Produce(object):
def __init__(self, iterable):
self._it = iterable
self.total = 0
def __iter__(self):
return self
def __next__(self):
self.total += 1
return next(self._it)
next = __next__ # only necessary for python2 compatibility
Maybe better to see this with an example:
>>> it = Produce(findall())
>>> it.total
0
>>> next(it)
1
>>> next(it)
2
>>> it.total
2
you can use enumerate to count stuff, for example
i=0
for i,v in enumerate(range(10), 1 ):
print(v)
print("total",i)
(notice the start value of the enumerate)
for more complex stuff, you can use the same principle, make produce a generator that yield both values and ignore one in the iteration and use it later when finished.
other alternative is passing a modifiable object, for example
def produce(mem):
t=0
for x in range(10):
t+=1
yield x
mem.append(t)
aggregate=[]
for x in produce(aggregate):
print(x)
print("total",aggregate[0])
in either case the result is the same for this example
0
1
2
3
4
5
6
7
8
9
total 10
Am I missing something? Why not:
def produce():
total = 0
for item in find_all():
total += 1
yield item
yield total

Building a python counter with rollover

I need to create a message counter object -not to be confused with Python's Counter class. The specification calls for a counter that is initialized to 0, then increments by 1 until it hits 4294967295, at which point it's supposed to cycle back over to 1.
I've implemented a class to do this, but this is just the naive approach. Is there a better way to achieve this goal ?
class MessageCounter():
def __init__(self):
self.value = 0
def increment(self):
if self.value < 4294967295:
self.value += 1
else:
self.reset()
def reset():
self.value = 1
As an alternative to OO, you could create a generating function that yields numbers in sequence, forever. There are a number of ways to do this. In descending order of size and straightforwardness:
def count_loop(upper_limit):
while True:
for i in range(upper_limit):
yield i
gen = count_loop(4294967295)
import itertools
gen = (i for _ in itertools.count() for i in range(4294967295))
gen = (i for _ in iter(int,1) for i in range(4294967295))
You would then retrieve your values by doing next(gen).
>>> next(gen)
0
>>> next(gen)
1
>>> next(gen)
2
>>> next(gen)
3
(note: Python 2.7 users should use xrange instead of range. However, this may only work for max values smaller than 2^31)
Instead of the reset you can just use the modulo operator:. It will "reset" to 0 instead of one, but that shouldn't matter since you initilized as 0.
def increment(self):
self.value = (value + 1) % 4294967296
I have an example, just more shorter.
class MessageCounter():
def __init__(self):
self.value = 0
def increment(self, reset=False):
self.value = self.value + 1 if self.value < 4294967295 and not reset else 1

How to build a basic iterator?

How would one create an iterative function (or iterator object) in python?
Iterator objects in python conform to the iterator protocol, which basically means they provide two methods: __iter__() and __next__().
The __iter__ returns the iterator object and is implicitly called
at the start of loops.
The __next__() method returns the next value and is implicitly called at each loop increment. This method raises a StopIteration exception when there are no more value to return, which is implicitly captured by looping constructs to stop iterating.
Here's a simple example of a counter:
class Counter:
def __init__(self, low, high):
self.current = low - 1
self.high = high
def __iter__(self):
return self
def __next__(self): # Python 2: def next(self)
self.current += 1
if self.current < self.high:
return self.current
raise StopIteration
for c in Counter(3, 9):
print(c)
This will print:
3
4
5
6
7
8
This is easier to write using a generator, as covered in a previous answer:
def counter(low, high):
current = low
while current < high:
yield current
current += 1
for c in counter(3, 9):
print(c)
The printed output will be the same. Under the hood, the generator object supports the iterator protocol and does something roughly similar to the class Counter.
David Mertz's article, Iterators and Simple Generators, is a pretty good introduction.
There are four ways to build an iterative function:
create a generator (uses the yield keyword)
use a generator expression (genexp)
create an iterator (defines __iter__ and __next__ (or next in Python 2.x))
create a class that Python can iterate over on its own (defines __getitem__)
Examples:
# generator
def uc_gen(text):
for char in text.upper():
yield char
# generator expression
def uc_genexp(text):
return (char for char in text.upper())
# iterator protocol
class uc_iter():
def __init__(self, text):
self.text = text.upper()
self.index = 0
def __iter__(self):
return self
def __next__(self):
try:
result = self.text[self.index]
except IndexError:
raise StopIteration
self.index += 1
return result
# getitem method
class uc_getitem():
def __init__(self, text):
self.text = text.upper()
def __getitem__(self, index):
return self.text[index]
To see all four methods in action:
for iterator in uc_gen, uc_genexp, uc_iter, uc_getitem:
for ch in iterator('abcde'):
print(ch, end=' ')
print()
Which results in:
A B C D E
A B C D E
A B C D E
A B C D E
Note:
The two generator types (uc_gen and uc_genexp) cannot be reversed(); the plain iterator (uc_iter) would need the __reversed__ magic method (which, according to the docs, must return a new iterator, but returning self works (at least in CPython)); and the getitem iteratable (uc_getitem) must have the __len__ magic method:
# for uc_iter we add __reversed__ and update __next__
def __reversed__(self):
self.index = -1
return self
def __next__(self):
try:
result = self.text[self.index]
except IndexError:
raise StopIteration
self.index += -1 if self.index < 0 else +1
return result
# for uc_getitem
def __len__(self)
return len(self.text)
To answer Colonel Panic's secondary question about an infinite lazily evaluated iterator, here are those examples, using each of the four methods above:
# generator
def even_gen():
result = 0
while True:
yield result
result += 2
# generator expression
def even_genexp():
return (num for num in even_gen()) # or even_iter or even_getitem
# not much value under these circumstances
# iterator protocol
class even_iter():
def __init__(self):
self.value = 0
def __iter__(self):
return self
def __next__(self):
next_value = self.value
self.value += 2
return next_value
# getitem method
class even_getitem():
def __getitem__(self, index):
return index * 2
import random
for iterator in even_gen, even_genexp, even_iter, even_getitem:
limit = random.randint(15, 30)
count = 0
for even in iterator():
print even,
count += 1
if count >= limit:
break
print
Which results in (at least for my sample run):
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
How to choose which one to use? This is mostly a matter of taste. The two methods I see most often are generators and the iterator protocol, as well as a hybrid (__iter__ returning a generator).
Generator expressions are useful for replacing list comprehensions (they are lazy and so can save on resources).
If one needs compatibility with earlier Python 2.x versions use __getitem__.
I see some of you doing return self in __iter__. I just wanted to note that __iter__ itself can be a generator (thus removing the need for __next__ and raising StopIteration exceptions)
class range:
def __init__(self,a,b):
self.a = a
self.b = b
def __iter__(self):
i = self.a
while i < self.b:
yield i
i+=1
Of course here one might as well directly make a generator, but for more complex classes it can be useful.
First of all the itertools module is incredibly useful for all sorts of cases in which an iterator would be useful, but here is all you need to create an iterator in python:
yield
Isn't that cool? Yield can be used to replace a normal return in a function. It returns the object just the same, but instead of destroying state and exiting, it saves state for when you want to execute the next iteration. Here is an example of it in action pulled directly from the itertools function list:
def count(n=0):
while True:
yield n
n += 1
As stated in the functions description (it's the count() function from the itertools module...) , it produces an iterator that returns consecutive integers starting with n.
Generator expressions are a whole other can of worms (awesome worms!). They may be used in place of a List Comprehension to save memory (list comprehensions create a list in memory that is destroyed after use if not assigned to a variable, but generator expressions can create a Generator Object... which is a fancy way of saying Iterator). Here is an example of a generator expression definition:
gen = (n for n in xrange(0,11))
This is very similar to our iterator definition above except the full range is predetermined to be between 0 and 10.
I just found xrange() (suprised I hadn't seen it before...) and added it to the above example. xrange() is an iterable version of range() which has the advantage of not prebuilding the list. It would be very useful if you had a giant corpus of data to iterate over and only had so much memory to do it in.
This question is about iterable objects, not about iterators. In Python, sequences are iterable too so one way to make an iterable class is to make it behave like a sequence, i.e. give it __getitem__ and __len__ methods. I have tested this on Python 2 and 3.
class CustomRange:
def __init__(self, low, high):
self.low = low
self.high = high
def __getitem__(self, item):
if item >= len(self):
raise IndexError("CustomRange index out of range")
return self.low + item
def __len__(self):
return self.high - self.low
cr = CustomRange(0, 10)
for i in cr:
print(i)
If you looking for something short and simple, maybe it will be enough for you:
class A(object):
def __init__(self, l):
self.data = l
def __iter__(self):
return iter(self.data)
example of usage:
In [3]: a = A([2,3,4])
In [4]: [i for i in a]
Out[4]: [2, 3, 4]
All answers on this page are really great for a complex object. But for those containing builtin iterable types as attributes, like str, list, set or dict, or any implementation of collections.Iterable, you can omit certain things in your class.
class Test(object):
def __init__(self, string):
self.string = string
def __iter__(self):
# since your string is already iterable
return (ch for ch in self.string)
# or simply
return self.string.__iter__()
# also
return iter(self.string)
It can be used like:
for x in Test("abcde"):
print(x)
# prints
# a
# b
# c
# d
# e
Include the following code in your class code.
def __iter__(self):
for x in self.iterable:
yield x
Make sure that you replace self.iterablewith the iterable which you iterate through.
Here's an example code
class someClass:
def __init__(self,list):
self.list = list
def __iter__(self):
for x in self.list:
yield x
var = someClass([1,2,3,4,5])
for num in var:
print(num)
Output
1
2
3
4
5
Note: Since strings are also iterable, they can also be used as an argument for the class
foo = someClass("Python")
for x in foo:
print(x)
Output
P
y
t
h
o
n
This is an iterable function without yield. It make use of the iter function and a closure which keeps it's state in a mutable (list) in the enclosing scope for python 2.
def count(low, high):
counter = [0]
def tmp():
val = low + counter[0]
if val < high:
counter[0] += 1
return val
return None
return iter(tmp, None)
For Python 3, closure state is kept in an immutable in the enclosing scope and nonlocal is used in local scope to update the state variable.
def count(low, high):
counter = 0
def tmp():
nonlocal counter
val = low + counter
if val < high:
counter += 1
return val
return None
return iter(tmp, None)
Test;
for i in count(1,10):
print(i)
1
2
3
4
5
6
7
8
9
class uc_iter():
def __init__(self):
self.value = 0
def __iter__(self):
return self
def __next__(self):
next_value = self.value
self.value += 2
return next_value
Improving previous answer, one of the advantage of using class is that you can add __call__ to return self.value or even next_value.
class uc_iter():
def __init__(self):
self.value = 0
def __iter__(self):
return self
def __next__(self):
next_value = self.value
self.value += 2
return next_value
def __call__(self):
next_value = self.value
self.value += 2
return next_value
c = uc_iter()
print([c() for _ in range(10)])
print([next(c) for _ in range(5)])
# [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
# [20, 22, 24, 26, 28]
Other example of a class based on Python Random that can be both called and iterated could be seen on my implementation here

Categories