Related
My code uses yield from in python3 in recursive calls and it works perfectly fine. The problem right now is that this was introduced from PEP-380 in python 3.3 and I need it to work in python 2.7. I read up on a few articles and none of them were detailed enough or simple enough.
Few referred articles :
Converting “yield from” statement to Python 2.7 code
yield from and Python 2.7
and few others.
I have recreated a small Sample code (which takes in a multi-level list and returns a flattened list) that is very minimalistic compared to my requirements.
#python 3
def foo(obj):
for ele in obj:
if isinstance(ele, list):
yield from foo(ele)
else:
yield ele
#driver values :
>>> l = [1, [2, 3, [4,5]]]
>>> list(foo(l))
=> [1, 2, 3, 4, 5]
The same converted does not work in python 2.7 due to the non-availability of yield from.
You still need to loop. It doesn't matter that you have recursion here.
You need to loop over the generator produced by the recursive call and yield the results:
def foo(obj):
for ele in obj:
if isinstance(ele, list):
for res in foo(ele):
yield res
else:
yield ele
Your recursive call produces a generator, and you need to pass the results of the generator onwards. You do so by looping over the generator and yielding the individual values.
There are no better options, other than upgrading to Python 3.
yield from essentially passes on the responsibility to loop over to the caller, and passes back any generator.send() and generator.throw() calls to the delegated generator. You don't have any need to pass on .send() or .throw(), so what remains is taking responsibility to do the looping yourself.
Demo:
>>> import sys
>>> sys.version_info
sys.version_info(major=2, minor=7, micro=14, releaselevel='final', serial=0)
>>> def foo(obj):
... for ele in obj:
... if isinstance(ele, list):
... for res in foo(ele):
... yield res
... else:
... yield ele
...
>>> l = [1, [2, 3, [4,5]]]
>>> list(foo(l))
[1, 2, 3, 4, 5]
yield from was introduced in PEP 380 -- Syntax for Delegating to a Subgenerator (not PEP 342), specifically because a loop over the sub-generator would not delegate generator.throw() and generator.send() information.
The PEP explicitly states:
If yielding of values is the only concern, this can be performed without much difficulty using a loop such as
for v in g:
yield v
The Formal Semantics has a Python implementation equivalent that may look intimidating at first, but you can still pick out that it loops (with while 1:, looping ends when there is an exception or StopIteration is handled, new values are retrieved with next() or generator.send(..)), and yields the results (with yield _y).
Why do you say "my code cannot work with loops and needs to be recursive"? You can easily use a loop in a recursive generator:
def foo(obj):
for ele in obj:
if isinstance(ele, list):
#yield from foo(ele)
for t in foo(ele):
yield t
else:
yield ele
l = [1, [2, 3, [4, 5]]]
print list(foo(l))
output
[1, 2, 3, 4, 5]
So I've been writing iterators for a while, and I thought that I understood them. But I've been struggling with some issues tonight, and the more I play with it, the more confused I become.
I thought that for an iterator you had to implement __iter__ and next (or __next__). And that when you first tried to iterate over the iterator the __iter__ method would be called, and then next would be called until an StopIteration was raised.
When I run this code though
class Iter(object):
def __iter__(self):
return iter([2, 4, 6])
def next(self):
for y in [1, 2, 3]:
return y
iterable = Iter()
for x in iterable:
print(x)
The output is 2 4 6. So __iter__ is being called, but not next. That does seem to match with the documentation that I found here. But then that raises a whole bunch more questions in my mind.
Specifically, what's the difference between a container type and iterator if it's not the implementation of next? How do I know before hand which way my class is going to be treated? And most importantly, if I want to write a class where my next method is called when I use for x in Iter(), how can I do that?
A list is iterable, but it is not an iterator. Compare and contrast:
>>> type([])
list
>>> type(iter([]))
list_iterator
Calling iter on a list creates and returns a new iterator object for iterating the contents of that list.
In your object, you just return a list iterator, specifically an iterator over the list [2, 4, 6], so that object knows nothing about yielding elements 1, 2, 3.
def __iter__(self):
return iter([2, 4, 6]) # <-- you're returning the list iterator, not your own
Here's a more fundamental implementation conforming to the iterator protocol in Python 2, which doesn't confuse matters by relying on list iterators, generators, or anything fancy at all.
class Iter(object):
def __iter__(self):
self.val = 0
return self
def next(self):
self.val += 1
if self.val > 3:
raise StopIteration
return self.val
According to the documentation you link to, an iterator's __iter__ method is supposed to return itself. So your iterator isn't really an iterator at all: when for invokes __iter__ to get the iterator, you're giving it iter([2,4,6]), but you should be giving it self.
(Also, I don't think your next method does what you intend: it returns 1 every time it's called, and never raises StopIteration. So if you fixed __iter__, then your iterator would be an iterator over an infinite stream of ones, rather than over the finite list [1, 2, 3]. But that's a side-issue.)
I am wondering if there is there is a simple Pythonic way (maybe using generators) to run a function over each item in a list and result in a list of returns?
Example:
def square_it(x):
return x*x
x_set = [0,1,2,3,4]
squared_set = square_it(x for x in x_set)
I notice that when I do a line by line debug on this, the object that gets passed into the function is a generator.
Because of this, I get an error:
TypeError: unsupported operand type(s) for *: 'generator' and 'generator'
I understand that this generator expression created a generator to be passed into the function, but I am wondering if there is a cool way to accomplish running the function multiple times only by specifying an iterable as the argument? (without modifying the function to expect an iterable).
It seems to me that this ability would be really useful to cut down on lines of code because you would not need to create a loop to fun the function and a variable to save the output in a list.
Thanks!
You want a list comprehension:
squared_set = [square_it(x) for x in x_set]
There's a builtin function, map(), for this common problem.
>>> map(square_it, x_set)
[0,1,4,9,16] # On Python 3, a generator is returned.
Alternatively, one can use a generator expression, which is memory-efficient but lazy (meaning the values will not be computed now, only when needed):
>>> (square_it(x) for x in x_set)
<generator object <genexpr> at ...>
Similarly, one can also use a list comprehension, which computes all the values upon creation, returning a list.
Additionally, here's a comparison of generator expressions and list comprehensions.
You want to call the square_it function inside the generator, not on the generator.
squared_set = (square_it(x) for x in x_set)
As the other answers have suggested, I think it is best (most "pythonic") to call your function explicitly on each element, using a list or generator comprehension.
To actually answer the question though, you can wrap your function that operates over scalers with a function that sniffs the input, and has different behavior depending on what it sees. For example:
>>> import types
>>> def scaler_over_generator(f):
... def wrapper(x):
... if isinstance(x, types.GeneratorType):
... return [f(i) for i in x]
... return f(x)
... return wrapper
>>> def square_it(x):
... return x * x
>>> square_it_maybe_over = scaler_over_generator(square_it)
>>> square_it_maybe_over(10)
100
>>> square_it_maybe_over(x for x in range(5))
[0, 1, 4, 9, 16]
I wouldn't use this idiom in my code, but it is possible to do.
You could also code it up with a decorator, like so:
>>> #scaler_over_generator
... def square_it(x):
... return x * x
>>> square_it(x for x in range(5))
[0, 1, 4, 9, 16]
If you didn't want/need a handle to the original function.
Note that there is a difference between list comprehension returning a list
squared_set = [square_it(x) for x in x_set]
and returning a generator that you can iterate over it:
squared_set = (square_it(x) for x in x_set)
For learning purposes, I'm trying to make a function using Python that takes in another function and two arrays as parameters and calls the function parameter on each index of each array parameter. So this should call add on a1[0] & a2[0], a1[1] & a2[1], etc. But all I'm getting back is a generator object. What's wrong?
def add(a,b):
yield a + b
def generator(add,a1,a2):
for i in range(len(a1)):
yield add(a1[i],a2[i])
g = generator(add,a1,a2)
print g.next()
I've also tried replacing what I have for yield above with
yield map(add,a1[i],a2[i])
But that works even less. I get this:
TypeError: argument 2 to map() must support iteration
Your definition of add() is at least strange (I'm leaning twoards calling it "wrong"). You should return the result, not yield it:
def add(a, b):
return a + b
Now, your generator() will work, though
map(add, a1, a2)
is an easier and faster way to do (almost) the same thing. (If you want an iterator rather than a list, use itertools.imap() instead of map().)
You get a generator because your add is a generator. It should be just return a + b.
I'm trying to make a function using Python that takes in another function and two arrays as parameters and calls the function parameter on each index of each array parameter.
def my_function(func, array_1, array_2):
for e_1,e_2 in zip(array_1, array_2):
yield func(e_1,e_2)
Example:
def add(a, b):
return a + b
for result in my_function(add, [1, 2, 3], [9, 8, 7]):
print(result)
will print:
10
10
10
Now, a couple of notes:
The add function can be found in the operator module.
You see that I used zip, take a look at its the doc.
Even though what you actually need is izip() the generator expression under zip() which basically doesn't return a list but an iterator to each value.
my_function is almost like map(), the only difference is that my_function is a generator while map() gives you a list. Once again the stdlib gives you the generator version of map in the itertools module: imap()
Example, my_fuction is just like imap:
from operator import add
from itertools import imap
for result in imap(add, [1, 2, 3], [9, 8, 7]):
print(result)
#10
#10
#10
I obviously suppose that the add function was just a quick example, otherwise check the built-in sum.
As others have said, you are defining add incorrectly and it should return instead of yield. Also, you could import it:
from operator import add
The reason why this doesn't work:
yield map(add, a1[i], a2[i])
Is because map works on lists/iterables and not single values. If add were defined correctly this could work:
yield map(add, [a[i]], [a2[i]])
But you shouldn't actually do that because it's more complicated than it needs to be for no good reason (as Sven Marnach's answer shows, your generator function is just an attempt to implement map so it really shouldn't use map even if it is a learning exercise). Finally, if the point is to make a function that takes a function as a parameter, I wouldn't call the parameter "add"; otherwise, what's the point of making it at all?
def generator(f, a1, a2):
for x, y in zip(a1, a2):
yield f(x, y)
Speaking of which, take a look at zip.
What I'm trying to do, is, given a list with an arbitrary number of other nested lists, recursively descend through the last value in the nested lists until I've reached the maximum depth, and then append a value to that list. An example might make this clearer:
>>> nested_list1 = [1, 2, 3, [4, 5, 6]]
>>> last_inner_append(nested_list1, 7)
[1, 2, 3, [4, 5, 6, 7]]
>>> nested_list2 = [1, 2, [3, 4], 5, 6]
>>> last_inner_append(nested_list2, 7)
[1, 2, [3, 4], 5, 6, 7]
The following code works, but it seems excessively tricky to me:
def add_to_inner_last(nested, item):
nest_levels = [nested]
try:
nest_levels.append(nested[-1])
except IndexError: # The empty list case
nested.append(item)
return
while type(nest_levels[-1]) == list:
try:
nest_levels.append(nest_levels[-1][-1])
except IndexError: # The empty inner list case
nest_levels[-1].append(item)
return
nest_levels[-2].append(item)
return
Some things I like about it:
It works
It handles the cases of strings at the end of lists, and the cases of empty lists
Some things I don't like about it:
I have to check the type of objects, because strings are also indexable
The indexing system feels too magical--I won't be able to understand this tomorrow
It feels excessively clever to use the fact that appending to a referenced list affects all references
Some general questions I have about it:
At first I was worried that appending to nest_levels was space inefficient, but then I realized that this is probably just a reference, and a new object is not created, right?
This code is purely side effect producing (It always returns None). Should I be concerned about that?
Basically, while this code works (I think...), I'm wondering if there's a better way to do this. By better I mean clearer or more pythonic. Potentially something with more explicit recursion? I had trouble defining a stopping point or a way to do this without producing side effects.
Edit:
To be clear, this method also needs to handle:
>>> last_inner_append([1,[2,[3,[4]]]], 5)
[1,[2,[3,[4,5]]]]
and:
>>> last_inner_append([1,[2,[3,[4,[]]]]], 5)
[1,[2,[3,[4,[5]]]]]
How about this:
def last_inner_append(x, y):
try:
if isinstance(x[-1], list):
last_inner_append(x[-1], y)
return x
except IndexError:
pass
x.append(y)
return x
This function returns the deepest inner list:
def get_deepest_list(lst, depth = 0):
deepest_list = lst
max_depth = depth
for li in lst:
if type(li) == list:
tmp_deepest_list, tmp_max_depth = get_deepest_list(li, depth + 1)
if max_depth < tmp_max_depth: # change to <= to get the rightmost inner list
max_depth = tmp_max_depth
deepest_list = tmp_deepest_list
return deepest_list, max_depth
And then use it as:
def add_to_deepest_inner(lst, item):
inner_lst, depth = get_deepest_list(lst)
inner_lst.append(item)
Here is my take:
def last_inner_append(cont, el):
if type(cont) == list:
if not len(cont) or type(cont[-1]) != list:
cont.append(el)
else:
last_inner_append(cont[-1], el)
I think it's nice and clear, and passes all your tests.
It is also pure side-effect; if you want to change this, I suggest you go with BasicWolf's approach and create a 'selector' and an 'update' function, where the latter uses the former.
It's the same recursion scheme as Phil H's, but handles empty lists.
I don't think there is a good way around the two type tests, however you approach them (e.g. with 'type' or checking for 'append'...).
You can test if append is callable, rather than using try/catch, and recursing:
def add_to_inner_last(nested, item):
if callable(nested,append):
if callable(nested[-1],append):
return add_to_inner_last(nested[-1],item)
else:
nested.append(item)
return true
else:
return false
It's slightly annoying to have to have two callable tests, but the alternative is to pass a reference to the parent as well as the child.
def last_inner_append(sequence, element):
def helper(tmp, seq, elem=element):
if type(seq) != list:
tmp.append(elem)
elif len(seq):
helper(seq, seq[-1])
else:
seq.append(elem)
helper(sequence, sequence)