def fib(n):
if n <= 1:
return n
else:
return fib(n - 2) + fib(n - 1)
def count(f):
def counted(*args):
counted.call_count += 1
return f(*args)
counted.call_count = 0
return counted
>>> fib = count(fib)
>>> fib(5)
5
>>> fib.call_count
15
>>> counted.call_count
NameError
I understand that fib is now actually counted. However, I cannot figure out why, when I want to call counted.call_count, I have to call fib.call_count.
I think that counted is an instance of the class Function. But what does counted.call_count exactly mean?
More specifically, I do not understand an attribute defined outside the code class ****:. In that case defining an attribute is like self.call_count is clearer, because I can understand self to refer to the instance. In my example, it seems that counted in counted.call_count is just like self in self.call_count, but is it?
The issue here is that the counted function isn't called counted in your main program code, it's called fib. (count returns counted, which you bind to fib with an assignment), so that's the name you should use.
Your reliance on the decorated function name is a little tenuous. You will observe that you rely on that name. If you change the calling code to read
fibfun = count(fib)
fibfun(5)
fibfun.call_count
you will see it only counts the outer call.
You are in fact writing a decorator, though you may not know it. A more conventional way to write this might be
def count(f):
def counted(*args):
counted.call_count += 1
return f(*args)
counted.call_count = 0
return counted
#count
def fib(n):
if n <= 1:
return n
else:
return fib(n - 2) + fib(n - 1)
There's then no need to wrap the function manually (the decorator syntax simply does the call for you), and you can just write
>>> fib(5)
5
>>> fib.call_count
15
The problem is that you can only call this decorated function once, unless you take into account that the count continues to accumulate as further calls are made:
>>> fib(5)
5
>>> fib.call_count
30
This may be acceptable. Otherwise, further complexity may be required.
Thinking about how the python interpreter works under the hood may help you understand what's going on here. When source code is passed through the lexer, and turned into bytecode we can see references to "load" and "store" commands indicating that the interpreter works primarily as a stack based machine:
>>> dis.dis(count)
2 0 LOAD_CLOSURE 0 (counted)
2 LOAD_CLOSURE 1 (f)
4 BUILD_TUPLE 2
6 LOAD_CONST 1 (<code object counted at 0x00000193D034C8A0, file "<ipython-input-2-a31b910d61de>", line 2>)
8 LOAD_CONST 2 ('count.<locals>.counted')
10 MAKE_FUNCTION 8
12 STORE_DEREF 0 (counted)
5 14 LOAD_CONST 3 (0)
16 LOAD_DEREF 0 (counted)
18 STORE_ATTR 0 (call_count)
6 20 LOAD_DEREF 0 (counted)
22 RETURN_VALUE
We can also see that the call_count attribute is bound to the new code object that is created, but the name: counted is explicitly de-referenced as it is no longer in scope and only the code object is returned.
Related
I am researching the lambda function, and I came across this code sample:
def myfunc(n):
return lambda a : a * n
tripler = myfunc(3)
print(tripler(10)) # Output: 30
I understand that when tripler is assigned to the return value from the myfunc function, the value of 3 is passed as an argument and from within the function, it is represented as 'n'. But lambda uses a bound variable labeled 'a'. So at this point, ignoring the final print line, I'm not sure how 'a' is factored into this. When I run this code, I receive no errors.
I would be interested in understanding how this code is processed and how it comes up with its conclusion without generating errors.
The return value of myfunc is a closure, which retains a reference to the argument passed to myfunc.
>>> x = myfunc(3)
>>> x.__closure__[0].cell_contents
3
When you call x, the value is used.
>>> x(10) # (lambda a: a * n)(10), with n == 3
30
You can see this in the byte code generated for x:
>>> import dis
>>> dis.dis(x)
2 0 LOAD_FAST 0 (a)
2 LOAD_DEREF 0 (n)
4 BINARY_MULTIPLY
6 RETURN_VALUE
LOAD_FAST references the local variables (in this case, the only local variable is the parameter a), while LOAD_DEREF references the closure. The argument 0 is an index into the tuple stored in x.__closure__.
I write the following code in Python:
def fib(n):
a, b = 0, 1
while b < n:
print(b, end=' ')
a, b = b, a + b
print(fib, fib(10))
I think the correct output should be:
<function fib at 0x000001DF4BB13E18> 1 1 2 3 5 8
But The output is:
1 1 2 3 5 8 <function fib at 0x000001C60F823E18> None
The code prints None and its behavior is odd.
Why behavior of print function is odd?
The behavior is correct. It's just that fib(10) is executed before you call print. That's because the arguments for the function call have to be executed before they are passed to the function.
So in fact you do calculate fib(10) (including all the prints inside the function) and then print fib and the result from the fib(10) call (which is None because your fib function contains no explicit return).
You can also invoke dis.dis to see the order of evaluation:
def f():
print(fib, fib(10))
import dis
dis.dis(f)
Result:
10 0 LOAD_GLOBAL 0 (print)
2 LOAD_GLOBAL 1 (fib)
4 LOAD_GLOBAL 1 (fib)
6 LOAD_CONST 1 (10)
8 CALL_FUNCTION 1 ----> the fib(10) call
10 CALL_FUNCTION 2 ----> the print call
^------- number of arguments!
12 POP_TOP
14 LOAD_CONST 0 (None)
16 RETURN_VALUE
Both arguments to print() are fully evaluated before print() is actually called. Since the second argument to your main print() is fib(10), fib is called with argument 10 during this process. And since it prints things, these get printed during the evaluation process. The function itself doesn't contain a return statement, so it returns None, which is why that's printed.
Let's break this down.
First we'll look at the ordering:
print(fib, fib(10))
The call to fib will be evaluated first and pass its returned value into the print function.
fib(10) will do some printing of it's own and then exit. In this case it doesn't explicitly return a value and so gets treated as None
The above call to print can therefore be seen as
print(fib, None)
Which will result in printing the function address and then None
Here's a simple file depicting some inconsistent Python (3.6) behavior. Why is it possible that Case 1 and Case 2 run but Case 3 fails, even though Case 3 is just a merger of the first two cases?
I have provided the dis output of the first two cases.
import dis # Python bytecode disassembler
class A(object):
def __init__(self):
self.x # In case 2 (and 3), getting x results in a function call (because they are #properties), which fails when instantiating A because y is undefined. Case 1 evaluates the reference to a function without calling it and so it does not raise an exception.
# CASE 1: Legal
def x(self):
y
pass
'''
# CASE 2: Legal
#property
def x(self):
pass
'''
'''
# CASE 3: Illegal:
#property
def x(self):
y
pass
'''
if __name__ == '__main__':
a = A()
dis.dis(A)
Case 1 bytecode:
Disassembly of __init__:
5 0 LOAD_FAST 0 (self)
2 LOAD_ATTR 0 (x)
4 POP_TOP
6 LOAD_CONST 0 (None)
8 RETURN_VALUE
Disassembly of x:
9 0 LOAD_GLOBAL 0 (y)
2 POP_TOP
10 4 LOAD_CONST 0 (None)
6 RETURN_VALUE
Case 2 bytecode:
Disassembly of __init__:
5 0 LOAD_FAST 0 (self)
2 LOAD_ATTR 0 (x)
4 POP_TOP
6 LOAD_CONST 0 (None)
8 RETURN_VALUE
There is no inconsistency here.
When you instantiate a = A(), __init__ is called, which calls self.x, which will execute the body of x. At that point, there is no y inscope, so you get an exception.
Thanks to #chepner's comment:
In case 1, you aren't calling anything; self.x is a function reference
that isn't used. In case 3, self.x actually calls the defined getter
for x, which presumably is then trying to access an undefined global
name.
The behavior caused by the line self.x in case 3 is fundamentally different from case 1 because case 1 doesn't call anything -- it just evaluates a reference to a function.
On the other hand, self.x in case 3 executes the body of the x method, resulting in the undefined y error.
In order to confirm #chepner's comment, I ran a.x() with case 1 and got the same error as in case 3.
A generator function can be defined by putting the yield keyword in the function’s body:
def gen():
for i in range(10):
yield i
How to define an empty generator function?
The following code doesn’t work, since Python cannot know that it is supposed to be a generator function instead of a normal function:
def empty():
pass
I could do something like this:
def empty():
if False:
yield
But that would be very ugly. Is there a nicer way?
You can use return once in a generator; it stops iteration without yielding anything, and thus provides an explicit alternative to letting the function run out of scope. So use yield to turn the function into a generator, but precede it with return to terminate the generator before yielding anything.
>>> def f():
... return
... yield
...
>>> list(f())
[]
I'm not sure it's that much better than what you have -- it just replaces a no-op if statement with a no-op yield statement. But it is more idiomatic. Note that just using yield doesn't work.
>>> def f():
... yield
...
>>> list(f())
[None]
Why not just use iter(())?
This question asks specifically about an empty generator function. For that reason, I take it to be a question about the internal consistency of Python's syntax, rather than a question about the best way to create an empty iterator in general.
If question is actually about the best way to create an empty iterator, then you might agree with Zectbumo about using iter(()) instead. However, it's important to observe that iter(()) doesn't return a function! It directly returns an empty iterable. Suppose you're working with an API that expects a callable that returns an iterable each time it's called, just like an ordinary generator function. You'll have to do something like this:
def empty():
return iter(())
(Credit should go to Unutbu for giving the first correct version of this answer.)
Now, you may find the above clearer, but I can imagine situations in which it would be less clear. Consider this example of a long list of (contrived) generator function definitions:
def zeros():
while True:
yield 0
def ones():
while True:
yield 1
...
At the end of that long list, I'd rather see something with a yield in it, like this:
def empty():
return
yield
or, in Python 3.3 and above (as suggested by DSM), this:
def empty():
yield from ()
The presence of the yield keyword makes it clear at the briefest glance that this is just another generator function, exactly like all the others. It takes a bit more time to see that the iter(()) version is doing the same thing.
It's a subtle difference, but I honestly think the yield-based functions are more readable and maintainable.
See also this great answer from user3840170 that uses dis to show another reason why this approach is preferable: it emits the fewest instructions when compiled.
iter(())
You don't require a generator. C'mon guys!
Python 3.3 (because I'm on a yield from kick, and because #senderle stole my first thought):
>>> def f():
... yield from ()
...
>>> list(f())
[]
But I have to admit, I'm having a hard time coming up with a use case for this for which iter([]) or (x)range(0) wouldn't work equally well.
Another option is:
(_ for _ in ())
Like #senderle said, use this:
def empty():
return
yield
I’m writing this answer mostly to share another justification for it.
One reason for choosing this solution above the others is that it is optimal as far as the interpreter is concerned.
>>> import dis
>>> def empty_yield_from():
... yield from ()
...
>>> def empty_iter():
... return iter(())
...
>>> def empty_return():
... return
... yield
...
>>> def noop():
... pass
...
>>> dis.dis(empty_yield_from)
2 0 LOAD_CONST 1 (())
2 GET_YIELD_FROM_ITER
4 LOAD_CONST 0 (None)
6 YIELD_FROM
8 POP_TOP
10 LOAD_CONST 0 (None)
12 RETURN_VALUE
>>> dis.dis(empty_iter)
2 0 LOAD_GLOBAL 0 (iter)
2 LOAD_CONST 1 (())
4 CALL_FUNCTION 1
6 RETURN_VALUE
>>> dis.dis(empty_return)
2 0 LOAD_CONST 0 (None)
2 RETURN_VALUE
>>> dis.dis(noop)
2 0 LOAD_CONST 0 (None)
2 RETURN_VALUE
As we can see, the empty_return has exactly the same bytecode as a regular empty function; the rest perform a number of other operations that don’t change the behaviour anyway. The only difference between empty_return and noop is that the former has the generator flag set:
>>> dis.show_code(noop)
Name: noop
Filename: <stdin>
Argument count: 0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals: 0
Stack size: 1
Flags: OPTIMIZED, NEWLOCALS, NOFREE
Constants:
0: None
>>> dis.show_code(empty_return)
Name: empty_return
Filename: <stdin>
Argument count: 0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals: 0
Stack size: 1
Flags: OPTIMIZED, NEWLOCALS, GENERATOR, NOFREE
Constants:
0: None
The above disassembly is outdated as of CPython 3.11, but empty_return still comes out on top, with only two more opcodes (four bytes) than a no-op function:
>>> dis.dis(empty_yield_from)
1 0 RETURN_GENERATOR
2 POP_TOP
4 RESUME 0
2 6 LOAD_CONST 1 (())
8 GET_YIELD_FROM_ITER
10 LOAD_CONST 0 (None)
>> 12 SEND 3 (to 20)
14 YIELD_VALUE
16 RESUME 2
18 JUMP_BACKWARD_NO_INTERRUPT 4 (to 12)
>> 20 POP_TOP
22 LOAD_CONST 0 (None)
24 RETURN_VALUE
>>> dis.dis(empty_iter)
1 0 RESUME 0
2 2 LOAD_GLOBAL 1 (NULL + iter)
14 LOAD_CONST 1 (())
16 PRECALL 1
20 CALL 1
30 RETURN_VALUE
>>> dis.dis(empty_return)
1 0 RETURN_GENERATOR
2 POP_TOP
4 RESUME 0
2 6 LOAD_CONST 0 (None)
8 RETURN_VALUE
>>> dis.dis(noop)
1 0 RESUME 0
2 2 LOAD_CONST 0 (None)
4 RETURN_VALUE
Of course, the strength of this argument is very dependent on the particular implementation of Python in use; a sufficiently smart alternative interpreter may notice that the other operations amount to nothing useful and optimise them out. However, even if such optimisations are present, they require the interpreter to spend time performing them and to safeguard against optimisation assumptions being broken, like the iter identifier at global scope being rebound to something else (even though that would most likely indicate a bug if it actually happened). In the case of empty_return there is simply nothing to optimise, as bytecode generation stops after a return statement, so even the relatively naïve CPython will not waste time on any spurious operations.
Must it be a generator function? If not, how about
def f():
return iter(())
The "standard" way to make an empty iterator appears to be iter([]).
I suggested to make [] the default argument to iter(); this was rejected with good arguments, see http://bugs.python.org/issue25215
- Jurjen
I want to give a class based example since we haven't had any suggested yet. This is a callable iterator that generates no items. I believe this is a straightforward and descriptive way to solve the issue.
class EmptyGenerator:
def __iter__(self):
return self
def __next__(self):
raise StopIteration
>>> list(EmptyGenerator())
[]
generator = (item for item in [])
Nobody has mentioned it yet, but calling the built-in function zip with no arguments returns an empty iterator:
>>> it = zip()
>>> next(it)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
Suppose I want to execute code, for example
value += 5
inside a namespace of my own (so the result is essentially mydict['value'] += 5). There's a function exec(), but I have to pass a string there:
exec('value += 5', mydict)
and passing statements as strings seems strange (e.g. it's not colorized that way).
Can it be done like:
def block():
value += 5
???(block, mydict)
? The obvious candidate for last line was exec(block.__code__, mydict), but no luck: it raises UnboundLocalError about value. I believe it basically executes block(), not the code inside block, so assignments aren't easy – is that correct?
Of course, another possible solution would be to disassembly block.__code__...
FYI, I got the question because of this thread. Also, this is why some (me undecided) call for new syntax
using mydict:
value += 5
Note how this doesn't throw error but doesn't change mydict either:
def block(value = 0):
value += 5
block(**mydict)
You can pass bytecode instead of a string to exec, you just need to make the right bytecode for the purpose:
>>> bytecode = compile('value += 5', '<string>', 'exec')
>>> mydict = {'value': 23}
>>> exec(bytecode, mydict)
>>> mydict['value']
28
Specifically, ...:
>>> import dis
>>> dis.dis(bytecode)
1 0 LOAD_NAME 0 (value)
3 LOAD_CONST 0 (5)
6 INPLACE_ADD
7 STORE_NAME 0 (value)
10 LOAD_CONST 1 (None)
13 RETURN_VALUE
the load and store instructions must be of the _NAME persuasion, and this compile makes them so, while...:
>>> def f(): value += 5
...
>>> dis.dis(f.func_code)
1 0 LOAD_FAST 0 (value)
3 LOAD_CONST 1 (5)
6 INPLACE_ADD
7 STORE_FAST 0 (value)
10 LOAD_CONST 0 (None)
13 RETURN_VALUE
...code in a function is optimized to use the _FAST versions, and those don't work on a dict passed to exec. If you started somehow with a bytecode using the _FAST instructions, you could patch it to use the _NAME kind instead, e.g. with bytecodehacks or some similar approach.
Use the global keyword to force dynamic scoping on any variables you want to modify from within the block:
def block():
global value
value += 5
mydict = {"value": 42}
exec(block.__code__, mydict)
print(mydict["value"])
Here is a crazy decorator to create such a block that uses "custom locals". In reality it is a quick hack to turn all variable access inside the function to global access, and evaluate the result with the custom locals dictionary as environment.
import dis
import functools
import types
import string
def withlocals(func):
"""Decorator for executing a block with custom "local" variables.
The decorated function takes one argument: its scope dictionary.
>>> #withlocals
... def block():
... counter += 1
... luckynumber = 88
>>> d = {"counter": 1}
>>> block(d)
>>> d["counter"]
2
>>> d["luckynumber"]
88
"""
def opstr(*opnames):
return "".join([chr(dis.opmap[N]) for N in opnames])
translation_table = string.maketrans(
opstr("LOAD_FAST", "STORE_FAST"),
opstr("LOAD_GLOBAL", "STORE_GLOBAL"))
c = func.func_code
newcode = types.CodeType(c.co_argcount,
0, # co_nlocals
c.co_stacksize,
c.co_flags,
c.co_code.translate(translation_table),
c.co_consts,
c.co_varnames, # co_names, name of global vars
(), # co_varnames
c.co_filename,
c.co_name,
c.co_firstlineno,
c.co_lnotab)
#functools.wraps(func)
def wrapper(mylocals):
return eval(newcode, mylocals)
return wrapper
if __name__ == '__main__':
import doctest
doctest.testmod()
This is just a monkey-patching adaption of someone's brilliant recipe for a goto decorator
From S.Lott's comment above I think I get the idea for an answer using creation of new class.
class _(__metaclass__ = change(mydict)):
value += 1
...
where change is a metaclass whose __prepare__ reads dictionary and whose __new__ updates dictionary.
For reuse, the snippet below would work, but it's kind of ugly:
def increase_value(d):
class _(__metaclass__ = change(d)):
value += 1
...
increase_value(mydict)