def fib(n):
if n <= 1:
return n
else:
return fib(n - 2) + fib(n - 1)
def count(f):
def counted(*args):
counted.call_count += 1
return f(*args)
counted.call_count = 0
return counted
>>> fib = count(fib)
>>> fib(5)
5
>>> fib.call_count
15
>>> counted.call_count
NameError
I understand that fib is now actually counted. However, I cannot figure out why, when I want to call counted.call_count, I have to call fib.call_count.
I think that counted is an instance of the class Function. But what does counted.call_count exactly mean?
More specifically, I do not understand an attribute defined outside the code class ****:. In that case defining an attribute is like self.call_count is clearer, because I can understand self to refer to the instance. In my example, it seems that counted in counted.call_count is just like self in self.call_count, but is it?
The issue here is that the counted function isn't called counted in your main program code, it's called fib. (count returns counted, which you bind to fib with an assignment), so that's the name you should use.
Your reliance on the decorated function name is a little tenuous. You will observe that you rely on that name. If you change the calling code to read
fibfun = count(fib)
fibfun(5)
fibfun.call_count
you will see it only counts the outer call.
You are in fact writing a decorator, though you may not know it. A more conventional way to write this might be
def count(f):
def counted(*args):
counted.call_count += 1
return f(*args)
counted.call_count = 0
return counted
#count
def fib(n):
if n <= 1:
return n
else:
return fib(n - 2) + fib(n - 1)
There's then no need to wrap the function manually (the decorator syntax simply does the call for you), and you can just write
>>> fib(5)
5
>>> fib.call_count
15
The problem is that you can only call this decorated function once, unless you take into account that the count continues to accumulate as further calls are made:
>>> fib(5)
5
>>> fib.call_count
30
This may be acceptable. Otherwise, further complexity may be required.
Thinking about how the python interpreter works under the hood may help you understand what's going on here. When source code is passed through the lexer, and turned into bytecode we can see references to "load" and "store" commands indicating that the interpreter works primarily as a stack based machine:
>>> dis.dis(count)
2 0 LOAD_CLOSURE 0 (counted)
2 LOAD_CLOSURE 1 (f)
4 BUILD_TUPLE 2
6 LOAD_CONST 1 (<code object counted at 0x00000193D034C8A0, file "<ipython-input-2-a31b910d61de>", line 2>)
8 LOAD_CONST 2 ('count.<locals>.counted')
10 MAKE_FUNCTION 8
12 STORE_DEREF 0 (counted)
5 14 LOAD_CONST 3 (0)
16 LOAD_DEREF 0 (counted)
18 STORE_ATTR 0 (call_count)
6 20 LOAD_DEREF 0 (counted)
22 RETURN_VALUE
We can also see that the call_count attribute is bound to the new code object that is created, but the name: counted is explicitly de-referenced as it is no longer in scope and only the code object is returned.
Sometimes, some values/strings are hard-coded in functions. For example in the following function, I define a "constant" comparing string and check against it.
def foo(s):
c_string = "hello"
if s == c_string:
return True
return False
Without discussing too much about why it's bad to do this, and how it should be defined in the outer scope, I'm wondering what happens behind the scenes when it is defined this way.
Does the string get created each call?
If instead of the string "hello" it was the list: [1,2,3] (or a list with mutable content if it matters) would the same happen?
Because the string is immutable (as would a tuple), it is stored with the bytecode object for the function. It is loaded by a very simple and fast index lookup. This is actually faster than a global lookup.
You can see this in a disassembly of the bytecode, using the dis.dis() function:
>>> import dis
>>> def foo(s):
... c_string = "hello"
... if s == c_string:
... return True
... return False
...
>>> dis.dis(foo)
2 0 LOAD_CONST 1 ('hello')
3 STORE_FAST 1 (c_string)
3 6 LOAD_FAST 0 (s)
9 LOAD_FAST 1 (c_string)
12 COMPARE_OP 2 (==)
15 POP_JUMP_IF_FALSE 22
4 18 LOAD_GLOBAL 0 (True)
21 RETURN_VALUE
5 >> 22 LOAD_GLOBAL 1 (False)
25 RETURN_VALUE
>>> foo.__code__.co_consts
(None, 'hello')
The LOAD_CONST opcode loads the string object from the co_costs array that is part of the code object for the function; the reference is pushed to the top of the stack. The STORE_FAST opcode takes the reference from the top of the stack and stores it in the locals array, again a very simple and fast operation.
For mutable literals ({..}, [..]) special opcodes build the object, with the contents still treated as constants as much as possible (more complex structures just follow the same building blocks):
>>> def bar(): return ['spam', 'eggs']
...
>>> dis.dis(bar)
1 0 LOAD_CONST 1 ('spam')
3 LOAD_CONST 2 ('eggs')
6 BUILD_LIST 2
9 RETURN_VALUE
The BUILD_LIST call creates the new list object, using two constant string objects.
Interesting fact: If you used a list object for a membership test (something in ['option1', 'option2', 'option3'] Python knows the list object will never be mutated and will convert it to a tuple for you at compile time (a so-called peephole optimisation). The same applies to a set literal, which is converted to a frozenset() object, but only in Python 3.2 and newer. See Tuple or list when using 'in' in an 'if' clause?
Note that your sample function is using booleans rather verbosely; you could just have used:
def foo(s):
c_string = "hello"
return s == c_string
for the exact same result, avoiding the LOAD_GLOBAL calls in Python 2 (Python 3 made True and False keywords so the values can also be stored as constants).
This came up in a recent PyCon talk.
The statement
[] = []
does nothing meaningful, but it does not throw an exception either. I have the feeling this must be due to unpacking rules. You can do tuple unpacking with lists too, e.g.,
[a, b] = [1, 2]
does what you would expect. As logical consequence, this also should work, when the number of elements to unpack is 0, which would explain why assigning to an empty list is valid. This theory is further supported by what happens when you try to assign a non-empty list to an empty list:
>>> [] = [1]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: too many values to unpack
I would be happy with this explanation, if the same would also be true for tuples. If we can unpack to a list with 0 elements, we should also be able to unpack to a tuple with 0 elements, no? However:
>>> () = ()
File "<stdin>", line 1
SyntaxError: can't assign to ()
It seems like unpacking rules are not applied for tuples as they are for lists. I cannot think of any explanation for this inconsistency. Is there a reason for this behavior?
The comment by #user2357112 that this seems to be coincidence appears to be correct. The relevant part of the Python source code is in Python/ast.c:
switch (e->kind) {
# several cases snipped
case List_kind:
e->v.List.ctx = ctx;
s = e->v.List.elts;
break;
case Tuple_kind:
if (asdl_seq_LEN(e->v.Tuple.elts)) {
e->v.Tuple.ctx = ctx;
s = e->v.Tuple.elts;
}
else {
expr_name = "()";
}
break;
# several more cases snipped
}
/* Check for error string set by switch */
if (expr_name) {
char buf[300];
PyOS_snprintf(buf, sizeof(buf),
"can't %s %s",
ctx == Store ? "assign to" : "delete",
expr_name);
return ast_error(c, n, buf);
}
tuples have an explicit check that the length is not zero and raise an error when it is. lists do not have any such check, so there's no exception raised.
I don't see any particular reason for allowing assignment to an empty list when it is an error to assign to an empty tuple, but perhaps there's some special case that I'm not considering. I'd suggest that this is probably a (trivial) bug and that the behaviors should be the same for both types.
I decided to try to use dis to figure out what's going on here, when I tripped over something curious:
>>> def foo():
... [] = []
...
>>> dis.dis(foo)
2 0 BUILD_LIST 0
3 UNPACK_SEQUENCE 0
6 LOAD_CONST 0 (None)
9 RETURN_VALUE
>>> def bar():
... () = ()
...
File "<stdin>", line 2
SyntaxError: can't assign to ()
Somehow the Python compiler special-cases an empty tuple on the LHS. This difference varies from the specification, which states:
Assignment of an object to a single target is recursively defined as follows.
...
If the target is a target list enclosed in parentheses or in square brackets: The object must be an iterable with the same number of items as there are targets in the target list, and its items are assigned, from left to right, to the corresponding targets.
So it looks like you've found a legitimate, although ultimately inconsequential, bug in CPython (2.7.8 and 3.4.1 tested).
IronPython 2.6.1 exhibits the same difference, but Jython 2.7b3+ has a stranger behavior, with () = () starting a statement with seemingly no way to end it.
It's a bug.
http://bugs.python.org/issue23275
However, it seems to be harmless so I doubt it would get fixed for fear of breaking working code.
“Assigning to a list” is the wrong way to think about it.
In all cases you are unpacking: The Python interpreter creates an unpacking instruction from all three ways to write it, there are no lists or tuples involved on the left hand side (code courtesy of /u/old-man-prismo):
>>> def f():
... iterable = [1, 2]
... a, b = iterable
... (c, d) = iterable
... [e, f] = iterable
...
>>> from dis import dis
>>> dis(f)
2 0 LOAD_CONST 1 (1)
3 LOAD_CONST 2 (2)
6 BUILD_LIST 2
9 STORE_FAST 0 (iterable)
3 12 LOAD_FAST 0 (iterable)
15 UNPACK_SEQUENCE 2
18 STORE_FAST 1 (a)
21 STORE_FAST 2 (b)
4 24 LOAD_FAST 0 (iterable)
27 UNPACK_SEQUENCE 2
30 STORE_FAST 3 (c)
33 STORE_FAST 4 (d)
5 36 LOAD_FAST 0 (iterable)
39 UNPACK_SEQUENCE 2
42 STORE_FAST 5 (e)
45 STORE_FAST 6 (f)
48 LOAD_CONST 0 (None)
51 RETURN_VALUE
As you can see, all three statements are exactly the same.
What unpacking does now is basically:
_iterator = iter(some_iterable)
a = next(_iterator)
b = next(_iterator)
for superfluous_element in _iterator:
# this only happens if there’s something left
raise SyntaxError('Expected some_iterable to have 2 elements')
Analoguously for more or less names on the left side.
Now as #blckknght said: The compiler for some reason checks if the left hand side is an empty tuple and disallows that, but not if it’s an empty list.
It’s only consistent and logical to allow assigning to 0 names: Why not? You basically just assert that the iterable on the right hand side is empty. That opinion also seems to emerge as consensus in the bug report #gecko mentioned: Let’s allow () = iterable.
I know how to use generators but I don't know anything about their internals. I tried this:
In [4]: def f(): yield 1
In [6]: type(f())
Out[6]: generator
Now I disassemble it:
In [7]: dis.dis(f)
1 0 LOAD_CONST 1 (1)
3 YIELD_VALUE
4 POP_TOP
5 LOAD_CONST 0 (None)
8 RETURN_VALUE
Why does the opcodes suggest return None while f actually returns a generator?
All functions return None at their end if there is no explicit return, generators are no exception. As of Python 3.3, generators can return a final value when ending, but in Python 2.7 a blank return is mandatory if you use return to end the function early.
This is a 'limitation' of how Python frames are implemented; you have to have a RETURN_VALUE opcode to exit the frame cleanly and unwind the stack, and that opcode requires an operand, always.
Calling the function still produces a generator, but the byte code isn't executed until you actually call the .next() method on the generator.
I'd like to point to a function that does nothing:
def identity(*args)
return args
my use case is something like this
try:
gettext.find(...)
...
_ = gettext.gettext
else:
_ = identity
Of course, I could use the identity defined above, but a built-in would certainly run faster (and avoid bugs introduced by my own).
Apparently, map and filter use None for the identity, but this is specific to their implementations.
>>> _=None
>>> _("hello")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not callable
Doing some more research, there is none, a feature was asked in issue 1673203 And from Raymond Hettinger said there won't be:
Better to let people write their own trivial pass-throughs
and think about the signature and time costs.
So a better way to do it is actually (a lambda avoids naming the function):
_ = lambda *args: args
advantage: takes any number of parameters
disadvantage: the result is a boxed version of the parameters
OR
_ = lambda x: x
advantage: doesn't change the type of the parameter
disadvantage: takes exactly 1 positional parameter
An identity function, as defined in https://en.wikipedia.org/wiki/Identity_function, takes a single argument and returns it unchanged:
def identity(x):
return x
What you are asking for when you say you want the signature def identity(*args) is not strictly an identity function, as you want it to take multiple arguments. That's fine, but then you hit a problem as Python functions don't return multiple results, so you have to find a way of cramming all of those arguments into one return value.
The usual way of returning "multiple values" in Python is to return a tuple of the values - technically that's one return value but it can be used in most contexts as if it were multiple values. But doing that here means you get
>>> def mv_identity(*args):
... return args
...
>>> mv_identity(1,2,3)
(1, 2, 3)
>>> # So far, so good. But what happens now with single arguments?
>>> mv_identity(1)
(1,)
And fixing that problem quickly gives other issues, as the various answers here have shown.
So, in summary, there's no identity function defined in Python because:
The formal definition (a single argument function) isn't that useful, and is trivial to write.
Extending the definition to multiple arguments is not well-defined in general, and you're far better off defining your own version that works the way you need it to for your particular situation.
For your precise case,
def dummy_gettext(message):
return message
is almost certainly what you want - a function that has the same calling convention and return as gettext.gettext, which returns its argument unchanged, and is clearly named to describe what it does and where it's intended to be used. I'd be pretty shocked if performance were a crucial consideration here.
yours will work fine. When the number of parameters is fix you can use an anonymous function like this:
lambda x: x
There is no a built-in identity function in Python. An imitation of the Haskell's id function would be:
identity = lambda x, *args: (x,) + args if args else x
Example usage:
identity(1)
1
identity(1,2)
(1, 2)
Since identity does nothing except returning the given arguments, I do not think that it is slower than a native implementation would be.
No, there isn't.
Note that your identity:
is equivalent to lambda *args: args
Will box its args - i.e.
In [6]: id = lambda *args: args
In [7]: id(3)
Out[7]: (3,)
So, you may want to use lambda arg: arg if you want a true identity function.
NB: This example will shadow the built-in id function (which you will probably never use).
If the speed does not matter, this should handle all cases:
def identity(*args, **kwargs):
if not args:
if not kwargs:
return None
elif len(kwargs) == 1:
return next(iter(kwargs.values()))
else:
return (*kwargs.values(),)
elif not kwargs:
if len(args) == 1:
return args[0]
else:
return args
else:
return (*args, *kwargs.values())
Examples of usage:
print(identity())
None
$identity(1)
1
$ identity(1, 2)
(1, 2)
$ identity(1, b=2)
(1, 2)
$ identity(a=1, b=2)
(1, 2)
$ identity(1, 2, c=3)
(1, 2, 3)
Stub of a single-argument function
gettext.gettext (the OP's example use case) accepts a single argument, message. If one needs a stub for it, there's no reason to return [message] instead of message (def identity(*args): return args). Thus both
_ = lambda message: message
def _(message):
return message
fit perfectly.
...but a built-in would certainly run faster (and avoid bugs introduced by my own).
Bugs in such a trivial case are barely relevant. For an argument of predefined type, say str, we can use str() itself as an identity function (because of string interning it even retains object identity, see id note below) and compare its performance with the lambda solution:
$ python3 -m timeit -s "f = lambda m: m" "f('foo')"
10000000 loops, best of 3: 0.0852 usec per loop
$ python3 -m timeit "str('foo')"
10000000 loops, best of 3: 0.107 usec per loop
A micro-optimisation is possible. For example, the following Cython code:
test.pyx
cpdef str f(str message):
return message
Then:
$ pip install runcython3
$ makecython3 test.pyx
$ python3 -m timeit -s "from test import f" "f('foo')"
10000000 loops, best of 3: 0.0317 usec per loop
Build-in object identity function
Don't confuse an identity function with the id built-in function which returns the 'identity' of an object (meaning a unique identifier for that particular object rather than that object's value, as compared with == operator), its memory address in CPython.
Lots of good answers and discussion are in this topic. I just want to note that, in OP's case where there is a single argument in the identity function, compile-wise it doesn't matter if you use a lambda or define a function (in which case you should probably define the function to stay PEP8 compliant). The bytecodes are functionally identical:
import dis
function_method = compile("def identity(x):\n return x\ny=identity(Type('x', (), dict()))", "foo", "exec")
dis.dis(function_method)
1 0 LOAD_CONST 0 (<code object identity at 0x7f52cc30b030, file "foo", line 1>)
2 LOAD_CONST 1 ('identity')
4 MAKE_FUNCTION 0
6 STORE_NAME 0 (identity)
3 8 LOAD_NAME 0 (identity)
10 LOAD_NAME 1 (Type)
12 LOAD_CONST 2 ('x')
14 LOAD_CONST 3 (())
16 LOAD_NAME 2 (dict)
18 CALL_FUNCTION 0
20 CALL_FUNCTION 3
22 CALL_FUNCTION 1
24 STORE_NAME 3 (y)
26 LOAD_CONST 4 (None)
28 RETURN_VALUE
Disassembly of <code object identity at 0x7f52cc30b030, file "foo", line 1>:
2 0 LOAD_FAST 0 (x)
2 RETURN_VALUE
And lambda
import dis
lambda_method = compile("identity = lambda x: x\ny=identity(Type('x', (), dict()))", "foo", "exec")
dis.dis(lambda_method)
1 0 LOAD_CONST 0 (<code object <lambda> at 0x7f52c9fbbd20, file "foo", line 1>)
2 LOAD_CONST 1 ('<lambda>')
4 MAKE_FUNCTION 0
6 STORE_NAME 0 (identity)
2 8 LOAD_NAME 0 (identity)
10 LOAD_NAME 1 (Type)
12 LOAD_CONST 2 ('x')
14 LOAD_CONST 3 (())
16 LOAD_NAME 2 (dict)
18 CALL_FUNCTION 0
20 CALL_FUNCTION 3
22 CALL_FUNCTION 1
24 STORE_NAME 3 (y)
26 LOAD_CONST 4 (None)
28 RETURN_VALUE
Disassembly of <code object <lambda> at 0x7f52c9fbbd20, file "foo", line 1>:
1 0 LOAD_FAST 0 (x)
2 RETURN_VALUE
Adding to all answers:
Notice there is an implicit convention in Python stdlib, where a HOF defaulting it's key parameter function to the identity function, interprets None as such.
E.g. sorted, heapq.merge, max, min, etc.
So, it is not bad idea to consider your HOF expecting key to following the same pattern.
That is, instead of:
def my_hof(x, key=lambda _: _):
...
(whis is totally right)
You could write:
def my_hof(x, key=None):
if key is None: key = lambda _: _
...
If you want.
The thread is pretty old. But still wanted to post this.
It is possible to build an identity method for both arguments and objects. In the example below, ObjOut is an identity for ObjIn. All other examples above haven't dealt with dict **kwargs.
class test(object):
def __init__(self,*args,**kwargs):
self.args = args
self.kwargs = kwargs
def identity (self):
return self
objIn=test('arg-1','arg-2','arg-3','arg-n',key1=1,key2=2,key3=3,keyn='n')
objOut=objIn.identity()
print('args=',objOut.args,'kwargs=',objOut.kwargs)
#If you want just the arguments to be printed...
print(test('arg-1','arg-2','arg-3','arg-n',key1=1,key2=2,key3=3,keyn='n').identity().args)
print(test('arg-1','arg-2','arg-3','arg-n',key1=1,key2=2,key3=3,keyn='n').identity().kwargs)
$ py test.py
args= ('arg-1', 'arg-2', 'arg-3', 'arg-n') kwargs= {'key1': 1, 'keyn': 'n', 'key2': 2, 'key3': 3}
('arg-1', 'arg-2', 'arg-3', 'arg-n')
{'key1': 1, 'keyn': 'n', 'key2': 2, 'key3': 3}