In both Python2 and Python3, in the stack trace the __name__ of a function is not used, the original name (the one that is specified after def) is used instead.
Consider the example:
import traceback
def a():
return b()
def b():
return c()
def c():
print("\n".join(line.strip() for line in traceback.format_stack()))
a.__name__ = 'A'
b.__name__ = 'B'
c.__name__ = 'C'
a();
The output is:
File "test.py", line 16, in <module>
a();
File "test.py", line 4, in a
return b()
File "test.py", line 7, in b
return c()
File "test.py", line 10, in c
print("\n".join(line.strip() for line in traceback.format_stack()))
Why so? How do I change the name that is used in the stack trace? Where is the __name__ attribute used then?
So, basically every function has three things that can be considered being name of the function:
The original name of the code block
It's stored in the f.__code__.co_name (where f is the function object). If you use def orig_name to create function, orig_name is that name. For lambas it's <lambda>.
This attribute is readonly and can't be changed. So the only way to create function with the custom name in runtime I'm aware of is exec:
exec("""def {name}():
print '{name}'
""".format(name='any')) in globals()
any() # prints 'any'
(There is also more low-level way to do this that was mentioned in a comment to the question.)
The immutability of co_name actually makes sense: with that you can be sure that the name you see in the debugger (or just stack trace) is exactly the same you see in the source code (along with the filename and line number).
The __name__ attribute of the function object
It's also aliased to func_name.
You can modify it (orig_name.__name__ = 'updated name') and you surely do on a daily basis: #functools.wraps copies the __name__ of the decorated function to the new one.
__name__ is used by tools like pydoc, that's why you need #functools.wraps: so you don't see the technical details of every decorator in your documentation. Look at the example:
from functools import wraps
def decorator1(f):
def decorated(*args, **kwargs):
print 'start1'
f(*args, **kwargs)
return decorated
def decorator2(f):
#wraps(f)
def decorated(*args, **kwargs):
print 'start2'
f(*args, **kwargs)
return decorated
#decorator1
def test1():
print 'test1'
#decorator2
def test2():
print 'test2'
Here is the pydoc output:
FUNCTIONS
decorator1(f)
decorator2(f)
test1 = decorated(*args, **kwargs)
test2(*args, **kwargs)
With wraps there is no sign of decorated in the documentation.
Name of the reference
One more thing that can be called function name (though it hardly is) is the name of a variable or an attribute where reference to that function is stored.
If you create function with def name, the name attribute will be added to the current scope. In case of lambda you should assign the result to some variable: name = lambda: None.
Obviously you can create more than one reference to the same function and all that references can have different names.
The only way all that three things are connected to each other is the def foo statement that creates function object with both __name__ and __code__.co_name equal to foo and assign it to the foo attribute of the current scope. But they are not bound in any way and can be different from each other:
import traceback
def make_function():
def orig_name():
"""Docstring here
"""
traceback.print_stack()
return orig_name
globals()['name_in_module'] = make_function()
name_in_module.__name__ = 'updated name'
name_in_module()
Output:
File "my.py", line 13, in <module>
name_in_module()
File "my.py", line 7, in orig_name
traceback.print_stack()
Pydoc:
FUNCTIONS
make_function()
name_in_module = updated name()
Docstring here
I thank other people for comments and answers, they helped me to organize my thoughts and knowledge.
Tried to explore the CPython implementation, definitely not an expert. As pointed out in the comments, when the stack entry of f is printed, the attribute f.__code__.co_name is used. Also, f.__name__ is initially set to f.__code__.co_name, but when you modify the former, the latter is not modified accordingly.
Therefore, I tried to modify that directly, but it is not possible:
>>> f.__code__.co_name = 'g'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: readonly attribute
>>>
Why are there two ways to say a function's name? Well, according to the documentation, __name__ is defined for "class, function, method, descriptor, or generator instance", so in the case of functions it maps to that attribute, for other objects it will map to something else.
Related
I saw a code snippet in Python 3.6.5 that can be replicated with this simplified example below and I do not understand if this is something concerning or not. I am surprised it works honestly...
class Foo:
def bar(numb):
return numb
A1 = bar(1)
print(Foo)
print(Foo.A1)
print(Foo.bar(17))
In all python guides that I have seen, self appears as the first argument for all the purposes we know and love. When it is not, the methods are decorated with a static decorator and all is well. This case works as it is, however. If I were to use the static decorator on bar, I get a TypeError when setting A1:
Traceback (most recent call last):
File "/home/user/dir/understanding_classes.py", line 1, in <module>
class Foo:
File "/home/user/dir/understanding_classes.py", line 7, in Foo
A1 = bar(1)
TypeError: 'staticmethod' object is not callable
Is this something that is OK keeping in the code or is this a potential problem? I hope the question is not too broad, but how and why does this work?
The first parameter of the method will be set to the receiver. We call it self by convention, but self isn't a keyword; any valid parameter name would work just as well.
There's two different ways to invoke a method that are relevant here. Let's say we have a simple Person class with a name and a say_hi method
class Person:
def __init__(self, name):
self.name = name
def say_hi(self):
print(f'Hi my name is {self.name}')
p = Person('J.K.')
If we call the method on p, we'll get a call to say_hi with self=p
p.say_hi() # self=p, prints 'Hi my name is J.K.'
What you're doing in your example is calling the method via the class, and passing that first argument explicitly. The equivalent call here would be
Person.say_hi(p) # explicit self=p, also prints 'Hi my name is J.K.'
In your example you're using a non-static method then calling it through the class, then explicitly passing the first parameter. It happens to work but it doesn't make a lot of sense because you should be able to invoke a non-static method by saying
f = Foo()
f.bar() # numb = f, works, but numb isn't a number it's a Foo
If you want to put a function inside of a class that doesn't have a receiver, that's when you want to use #staticmethod (or, #classmethod more often)
class Person:
def __init__(self, name):
self.name = name
def say_hi(self):
print(f'Hi my name is {self.name}')
#staticmethod
def say_hello():
print('hello')
p = Person('J.K.')
Person.say_hello()
p.say_hello()
Let's say I have this code:
class StaticParent:
def _print(self):
print("I'm StaticParent")
class StaticChild(StaticParent):
def _print(self):
print('StaticChild saying: ')
super()._print()
def _parent_print_proto(self):
print("I'm DynamicParent")
def _child_print_proto(self):
print('DynamicChild saying: ')
super()._print()
DynamicParent = type('DynamicParent', tuple([]), {"_print": _parent_print_proto})
DynamicChild = type('DynamicChild', (DynamicParent,), {"_print": _child_print_proto})
sc = StaticChild()
sc._print()
dc = DynamicChild()
dc._print()
And it's output is:
StaticChild saying:
I'm StaticParent
DynamicChild saying:
Traceback (most recent call last):
File "/tmp/example.py", line 28, in <module>
dc._print()
File "/tmp/example.py", line 17, in _child_print_proto
super()._print()
RuntimeError: super(): __class__ cell not found
So question is how to create prototype method for many classes that calling super()?
PS I tried to implement method with lambda but it is also not working:
DynamicChildWithLambda = type('DynamicChild', (DynamicParent,), {"_print": lambda self : print('Lambda saying: ', super()._print())})
Traceback (most recent call last):
File "/tmp/example.py", line 30, in <module>
dcwl._print()
File "/tmp/example.py", line 23, in <lambda>
DynamicChildWithLambda = type('DynamicChild', (DynamicParent,), {"_print": lambda self : print('Lambda saying: ', super()._print())})
RuntimeError: super(): __class__ cell not found
PS2 Also I tried this way:
class StaticParent:
def _print(self):
print("I'm StaticParent")
def _child_print_proto(self):
print('DynamicChild saying: ')
super(StaticParent, self)._print()
DynamicChild = type('DynamicChild', (StaticParent,), {"_print": _child_print_proto})
dc = DynamicChild()
dc._print()
DynamicChild saying:
Traceback (most recent call last):
File "/tmp/example.py", line 13, in <module>
dc._print()
File "/tmp/example.py", line 8, in _child_print_proto
super(StaticParent, self)._print()
AttributeError: 'super' object has no attribute '_print'
Methods defined within a class get a fake closure scope that automatically provides the class it was defined in to no-arg super(). When defined outside a class, it can't do this (because clearly no class is being defined at the time you define the method). But you can still make a closure the old-fashioned way, by actually writing a closure function that you manually define __class__ appropriately in:
class StaticParent:
def _print(self):
print("I'm StaticParent")
class StaticChild(StaticParent):
def _print(self):
print('StaticChild saying: ')
super()._print()
def _parent_print_proto(self):
print("I'm DynamicParent")
# Nesting allows us to make the inner function have an appropriate __class__
# defined for use by no-arg super
def _make_child_print_proto(cls):
__class__ = cls
def _child_print_proto(self):
print('DynamicChild saying: ')
super()._print()
return _child_print_proto
DynamicParent = type('DynamicParent', tuple([]), {"_print": _parent_print_proto})
DynamicChild = type('DynamicChild', (DynamicParent,), {})
# Need DynamicChild to exist to use it as __class__, bind after creation
DynamicChild._print = _make_child_print_proto(DynamicChild)
sc = StaticChild()
sc._print()
dc = DynamicChild()
dc._print()
Try it online!
Yes, it's hacky and awful. In real code, I'd just use the less common explicit two-arg super:
def _child_print_proto(self):
print('DynamicChild saying: ')
super(DynamicChild, self)._print()
That's all super() does anyway; Python hides the class it was defined in in closure scope as __class__, super() pulls it and the first positional argument, assumed to be self, and implicitly does the same thing the two-arg form did explicitly.
To be clear: You cannot pass self.__class__ manually as the first argument to two-arg super(), to simulate the __class__ that is bound in closure scope. It appears to work, and it does work, until you actually make a child of the class with that method and try to call the method on it (and the whole point of super() is you might have an arbitrarily complex class hierarchy to navigate; you can't just say "Oh, but my class is special enough to never be subclassed again"). If you do something as simple as adding:
class DynamicGrandChild(DynamicChild):
pass
dgc = DynamicGrandChild()
dgc._print()
to the self.__class__-using code from Epsi95's answer, you're going to see:
StaticChild saying:
I'm StaticParent
DynamicChild saying:
I'm DynamicParent
DynamicChild saying:
DynamicChild saying:
DynamicChild saying:
DynamicChild saying:
DynamicChild saying:
DynamicChild saying:
DynamicChild saying:
DynamicChild saying:
... repeats a thousand times or so ...
DynamicChild saying:
DynamicChild saying:
Traceback (most recent call last):
File ".code.tio", line 31, in <module>
dgc._print()
File ".code.tio", line 15, in _child_print_proto
super(self.__class__, self)._print()
File ".code.tio", line 15, in _child_print_proto
super(self.__class__, self)._print()
File ".code.tio", line 15, in _child_print_proto
super(self.__class__, self)._print()
[Previous line repeated 994 more times]
File ".code.tio", line 14, in _child_print_proto
print('DynamicChild saying: ')
RecursionError: maximum recursion depth exceeded while calling a Python object
super() is used when you're designing for inheritance. super(self.__class__, self) is used only when you're sabotaging inheritance. The only safe ways to do this involve some sort of static linkage to the class (the one that the method will be attached to, not the run time type of whatever instance it is called with), and my two solutions above are the two reasonable ways to do this (there's only one other option really, which is making the closure, and still passing the class and self explicitly, e.g. instead of defining __class__, just use super(cls, self) to explicitly use a closure variable; not sufficiently distinct, and kinda the worst of both worlds).
So, I said "there's three ways", but in fact there is a slightly nicer (but also less portable, because the API for types.FunctionType has changed over time, even though closures defined normally haven't) solution, which lets you make a utility function to bind arbitrary functions to arbitrary classes, instead of requiring you to wrap each such function in a closure-maker. And that's to literally rebuild the function as a closure directly:
import types
# Define function normally, with super(). The function can't actually be used as is though
def _child_print_proto(self):
print('DynamicChild saying: ')
super()._print()
# Utility function binding arbitrary function to arbitrary class
def bind_to_class(cls, func):
# This translates as:
# Make a new function using the same code object as the passed function,
# but tell it it has one closure scoped variable named __class__, and
# provide the class as the value to associate with it.
# This requires Python 3.8 or later (code objects didn't have a replace method
# until then, so doing this would be even uglier than it already is)
return types.FunctionType(func.__code__.replace(co_freevars=('__class__',)), func.__globals__, closure=(types.CellType(cls),))
DynamicParent = type('DynamicParent', tuple([]), {"_print": _parent_print_proto})
DynamicChild = type('DynamicChild', (DynamicParent,), {})
# Rebind the function to a new function that believes it was defined in DynamicChild
DynamicChild._print = bind_to_class(DynamicChild, _child_print_proto)
Like I said, it's super-ugly and would need rewrites pre-3.8, but it does have the mild advantage of allowing you to use bind_to_class with arbitrary super() using functions to bind them to arbitrary classes. I still think manually calling two-arg super is the safe/obvious way to go, but now you've got all the options.
Is it possible to delete an object form inside its class?
class A():
def __init__(self):
print("init")
self.b="c"
def __enter__(self):
print("enter")
return self
def __exit__(self, type, value, traceback):
print("exit")
with A() as a:
print(a.b)
print(a.b)
returns:
init
enter
c
exit
c
How comes I still have access to the a object after exiting the with ? Is there a way to auto-delete the object in __exit__?
Yes and no. Use del a after the with clause. This will remove the variable a who is the last reference holder on the object.
The object itself (i. e. in __exit__()) cannot make the ones who know about it and hold a reference (i. e. the code at the with clause) forget this. As long as the reference exists, the object will exist.
Of course, your object can empty itself in the __exit__() and remain as a hollow thing (e. g. by del self.b in this case).
Short answer: it is (to some extent) possible, but not advisable at all.
The with part in Python has no dedicated scope, so that means that variables defined in the with statement are not removed. This is frequently wanted behavior. For example if you load a file, you can write it like:
with open('foo.txt') as f:
data = list(f)
print(data)
You do not want to remove the data variable: the with is here used to ensure that the file handler is properly closed (and the handler is also closed if an exception occurs in the body of the with).
Strictly speaking you can delete local variables that refer to the A() object, by a "hackish" solution: we inspect the call stack, and remove references to self (or another object), like:
import inspect
class A(object):
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
locs = inspect.stack()[1][0].f_locals
ks = [k for k, v in locs.items() if v is self]
for k in ks:
del locs[k]
Then it will delete it like:
>>> with A() as a:
... pass
...
>>> a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
But I would strongly advice against that. First of all, if the variabe is global, or located outside the local scope, it will not get removed here (we can fix this, but it will introduce a lot of extra logic).
Furthermore it is not said that the variable even exists, if the variable is iterable, one can define it like:
# If A.__enter__ returns an iterable with two elements
with A() as (foo, bar):
pass
So then these elements will not get recycled. Finally if the __enter__ returns self, it is possible that it "removes too much", since one could write with foo as bar, and then both foo and bar will be removed.
Most IDEs will probably not be able to understand the logic in the __exit__, anyway, and hence still will include a in the autocompletion.
In general, it is better to simply mark the object as closed, like:
import inspect
class A(object):
def __init__(self):
self.closed = False
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
self.closed = True
def some_method(self):
if self.closed:
raise Exception('A object is closed')
# process request
The above is also the way it is handled for a file handler.
class A():
def __init__(self):
print("init")
self.b="c"
def __enter__(self):
print("enter")
return self
def __exit__(self, type, value, traceback):
print("exit")
del self.b
with A() as a:
print(a.b)
print(a.b)
You can't delete instance of class A itself within __exit__. The best you can do is delete the property b.
init
enter
c
exit
Traceback (most recent call last):
File "main.py", line 14, in <module>
print(a.b)
AttributeError: A instance has no attribute 'b'
I am very new to python : I want to serialize and deserialize my custom object in python. Please guide me on the same. I have a sample class :
import pickle
import json
class MyClass():
variable = "blah"
num = 10
def function(self):
print("this is a message inside the class.")
def get_variable():
return variable
def get_num():
return num
def main():
myObj = MyClass()
with open('/opt/infi/deeMyObj.txt', 'w') as output:
pickle.dump(myObj, output,pickle.HIGHEST_PROTOCOL)
with open('/opt/infi/deeMyObj.txt', 'r') as input:
myObjread = pickle.load(input)
print myObjread.get_variable()
print myObjread.get_num()
main()
I am getting following error :
Traceback (most recent call last):
File "sample.py", line 30, in
main()
File "sample.py", line 27, in main
print myObjread.get_variable()
TypeError: get_variable() takes no arguments (1 given)
Main intention is to read the object back.
To expand on jasonharper's comment, your get_variable and get_num methods aren't referring to the class's member variables. They should take the object as their first argument, e.g.
class MyClass:
...
def get_variable(self):
return self.variable
I think your serialization code is OK, but I might be wrong.
(Aside)
This is a bit off-topic, but another thing to note: when you define variables directly within the class block, they're defined on the class, not on objects of that class. That happens to work out in this case, since Python will look for a class-level variable of the same name if it can't find one on the object. However, if you store, say, a list in one of them and start modifying it, you'd end up sharing it between objects, which is probably not what you want. Instead you want to define them on in an __init__ method:
class MyClass:
def __init__(self):
self.variable = "blah"
I tried to use the property definition from PythonDecoratorLibrary (example 3).
=> https://wiki.python.org/moin/PythonDecoratorLibrary#Property_Definition
import sys
def property(function):
keys = 'fget', 'fset', 'fdel'
[...]
After additionally importing sys I get this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in Angle
File "<stdin>", line 12, in property
TypeError: property() got an unexpected keyword argument 'doc'
Line 12 is: function() because of the sys import :)
My python version is 3.4.1 on Windows.
First of all: That is one ugly way to allow for local functions to define the 3 property functions.
The sample decorator masks the property built-in, but then tries to use it still to produce the property object. Oops.
You can still access the orginal built-in with:
import builtins
def property(function):
keys = 'fget', 'fset', 'fdel'
func_locals = {'doc':function.__doc__}
def probe_func(frame, event, arg):
if event == 'return':
locals = frame.f_locals
func_locals.update(dict((k, locals.get(k)) for k in keys))
sys.settrace(None)
return probe_func
sys.settrace(probe_func)
function()
return builtins.property(**func_locals)
The builtins module lets you access the builtins even when a local name has overridden it.
I've updated the wiki page to reflect this.
The example is relying on the built-in property function, while also naming itself property:
def property(function):
keys = 'fget', 'fset', 'fdel'
func_locals = {'doc':function.__doc__}
def probe_func(frame, event, arg):
if event == 'return':
locals = frame.f_locals
func_locals.update(dict((k, locals.get(k)) for k in keys))
sys.settrace(None)
return probe_func
sys.settrace(probe_func)
function()
return property(**func_locals) # This is supposed to be the built-in property
So it ends up calling itself (which doesn't take a doc keyword argument), instead of the built-in property (which takes a doc keyword argument). So yes, the example is broken. The function it's calling property should be named something else, or it should save a reference to the built-in property and call that internally.
Edit: Using builtins.property is obviously much nicer than saving a reference to property before masking it. So go with that.