I've got a bunch of functions (outside of any class) where I've set attributes on them, like funcname.fields = 'xxx'. I was hoping I could then access these variables from inside the function with self.fields, but of course it tells me:
global name 'self' is not defined
So... what can I do? Is there some magic variable I can access? Like __this__.fields?
A few people have asked "why?". You will probably disagree with my reasoning, but I have a set of functions that all must share the same signature (accept only one argument). For the most part, this one argument is enough to do the required computation. However, in a few limited cases, some additional information is needed. Rather than forcing every function to accept a long list of mostly unused variables, I've decided to just set them on the function so that they can easily be ignored.
Although, it occurs to me now that you could just use **kwargs as the last argument if you don't care about the additional args. Oh well...
Edit: Actually, some of the functions I didn't write, and would rather not modify to accept the extra args. By "passing in" the additional args as attributes, my code can work both with my custom functions that take advantage of the extra args, and with third party code that don't require the extra args.
Thanks for the speedy answers :)
self isn't a keyword in python, its just a normal variable name. When creating instance methods, you can name the first parameter whatever you want, self is just a convention.
You should almost always prefer passing arguments to functions over setting properties for input, but if you must, you can do so using the actual functions name to access variables within it:
def a:
if a.foo:
#blah
a.foo = false
a()
see python function attributes - uses and abuses for when this comes in handy. :D
def foo():
print(foo.fields)
foo.fields=[1,2,3]
foo()
# [1, 2, 3]
There is nothing wrong with adding attributes to functions. Many memoizers use this to cache results in the function itself.
For example, notice the use of func.cache:
from decorator import decorator
#decorator
def memoize(func, *args, **kw):
# Author: Michele Simoniato
# Source: http://pypi.python.org/pypi/decorator
if not hasattr(func, 'cache'):
func.cache = {}
if kw: # frozenset is used to ensure hashability
key = args, frozenset(kw.iteritems())
else:
key = args
cache = func.cache # attribute added by memoize
if key in cache:
return cache[key]
else:
cache[key] = result = func(*args, **kw)
return result
You can't do that "function accessing its own attributes" correctly for all situations - see for details here how can python function access its own attributes? - but here is a quick demonstration:
>>> def f(): return f.x
...
>>> f.x = 7
>>> f()
7
>>> g = f
>>> g()
7
>>> del f
>>> g()
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
File "<interactive input>", line 1, in f
NameError: global name 'f' is not defined
Basically most methods directly or indirectly rely on accessing the function object through lookup by name in globals; and if original function name is deleted, this stops working. There are other kludgey ways of accomplishing this, like defining class, or factory - but thanks to your explanation it is clear you don't really need that.
Just do the mentioned keyword catch-all argument, like so:
def fn1(oneArg):
// do the due
def fn2(oneArg, **kw):
if 'option1' in kw:
print 'called with option1=', kw['option1']
//do the rest
fn2(42)
fn2(42, option1='something')
Not sure what you mean in your comment of handling TypeError - that won't arise when using **kw. This approach works very well for some python system functions - check min(), max(), sort(). Recently sorted(dct,key=dct.get,reverse=True) came very handy to me in CodeGolf challenge :)
Example:
>>> def x(): pass
>>> x
<function x at 0x100451050>
>>> x.hello = "World"
>>> x.hello
"World"
You can set attributes on functions, as these are just plain objects, but I actually never saw something like this in real code.
Plus. self is not a keyword, just another variable name, which happens to be the particular instance of the class. self is passed implicitly, but received explicitly.
if you want globally set parameters for a callable 'thing' you could always create a class and implement the __call__ method?
There is no special way, within a function's body, to refer to the function object whose code is executing. Simplest is just to use funcname.field (with funcname being the function's name within the namespace it's in, which you indicate is the case -- it would be harder otherwise).
This isn't something you should do. I can't think of any way to do what you're asking except some walking around on the call stack and some weird introspection -- which isn't something that should happen in production code.
That said, I think this actually does what you asked:
import inspect
_code_to_func = dict()
def enable_function_self(f):
_code_to_func[f.func_code] = f
return f
def get_function_self():
f = inspect.currentframe()
code_obj = f.f_back.f_code
return _code_to_func[code_obj]
#enable_function_self
def foo():
me = get_function_self()
print me
foo()
While I agree with the the rest that this is probably not good design, the question did intrigue me. Here's my first solution, which I may update once I get decorators working. As it stands, it relies pretty heavily on being able to read the stack, which may not be possible in all implementations (something about sys._getframe() not necessarily being present...)
import sys, inspect
def cute():
this = sys.modules[__name__].__dict__.get(inspect.stack()[0][3])
print "My face is..." + this.face
cute.face = "very cute"
cute()
What do you think? :3
You could use the following (hideously ugly) code:
class Generic_Object(object):
pass
def foo(a1, a2, self=Generic_Object()):
self.args=(a1,a2)
print "len(self.args):", len(self.args)
return None
... as you can see it would allow you to use "self" as you described. You can't use an "object()" directly because you can't "monkey patch(*)" values into an object() instance. However, normal subclasses of object (such as the Generic_Object() I've shown here) can be "monkey patched"
If you wanted to always call your function with a reference to some object as the first argument that would be possible. You could put the defaulted argument first, followed by a *args and optional **kwargs parameters (through which any other arguments or dictionaries of options could be passed during calls to this function).
This is, as I said hideously ugly. Please don't ever publish any code like this or share it with anyone in the Python community. I'm only showing it here as a sort of strange educational exercise.
An instance method is like a function in Python. However, it exists within the namespace of a class (thus it must be accessed via an instance ... myobject.foo() for example) and it is called with a reference to "self" (analagous to the "this" pointer in C++) as the first argument. Also there's a method resolution process which causes the interpreter to search the namespace of the instance, then it's class, and then each of the parent classes and so on ... up through the inheritance tree.
An unbound function is called with whatever arguments you pass to it. There can't bee any sort of automatically pre-pended object/instance reference to the argument list. Thus, writing a function with an initial argument named "self" is meaningless. (It's legal because Python doesn't place any special meaning on the name "self." But meaningless because callers to your function would have to manually supply some sort of object reference to the argument list and it's not at all clear what that should be. Just some bizarre "Generic_Object" which then floats around in the global variable space?).
I hope that clarifies things a bit. It sounds like you're suffering from some very fundamental misconceptions about how Python and other object-oriented systems work.
("Monkey patching" is a term used to describe the direct manipulation of an objects attributes -- or "instance variables" by code that is not part of the class hierarchy of which the object is an instance).
As another alternative, you can make the functions into bound class methods like so:
class _FooImpl(object):
a = "Hello "
#classmethod
def foo(cls, param):
return cls.a + param
foo = _FooImpl.foo
# later...
print foo("World") # yes, Hello World
# and if you have to change an attribute:
foo.im_self.a = "Goodbye "
If you want functions to share attribute namespaecs, you just make them part of the same class. If not, give each its own class.
What exactly are you hoping "self" would point to, if the function is defined outside of any class? If your function needs some global information to execute properly, you need to send this information to the function in the form of an argument.
If you want your function to be context aware, you need to declare it within the scope of an object.
Related
EDIT2：
A minimal demonstration is:
code = """\
a=1
def f1():
print(a)
print(f1.__closure__)
f1()
"""
def foo():
exec(code)
foo()
Which gives:
None
Traceback (most recent call last):
File "D:/workfiles/test_eval_rec.py", line 221, in <module>
foo()
File "D:/workfiles//test_eval_rec.py", line 219, in foo
exec(code)
File "<string>", line 5, in <module>
File "<string>", line 3, in f1
NameError: name 'a' is not defined
It can be seen that the __closure__ attribute of function defined inside code str passed to exec() is None, making calling the function fails.
Why does this happen and how can I define a function successfully?
I find several questions that may be related.
Closure lost during callback defined in exec()
Using exec() with recursive functions
Why exec() works differently when invoked inside of function and how to avoid it
Why are closures broken within exec?
NameError: name 'self' is not defined IN EXEC/EVAL
These questions are all related to "defining a function insdie exec()". I think the fourth question here is closest to the essence of these problems. The common cause of these problems is that when defining a function in exec(), the __closure__ attribute of the function object can not be set correctly and will always be None. However, many existing answers to this question didn't realize this point.
Why these questions are caused by wrong __closure__:
When defining a function, __closure__ attribute is set to a dict that contains all local symbols (at the place where the keyword def is used) that is used inside the newly defined funtion. When calling a function, local symbol tables will be retrived from the __closure__ attribute. Since the __closure__ is set to None, the local symbol tables can not be retrived as expected, making the function call fail.
These answers work by making None a correct __closure__ attribute:
Existing solutions to the questions listed above solve these problems by getting the function definition rid of the usage of local symbol, i.e, they make the local symbols used(variable, function definition) global by passing globals() as locals of exec or by using keyword global explicitly in the code string.
Why existing solution unsatisfying:
These solutions I think is just an escape of the core problem of setting __closure__ correctly when define a functioni inside exec(). And as symbols used in the function definition is made global, these solutions will produce redundant global symbol which I don't want.
Original Questions：
(You May ignore this session, I have figured something out, and what I currently want to ask is described as the session EDIT2. The original question can be viewed as a sepecial case of the question described in session EDIT2)
original title of this question is: Wrapping class function to new function with exec() raise NameError that ‘self’ is not defined
I want to wrap an existing member function to a new class function. However, exec() function failed with a NameError that ‘self’ is not defined.
I did some experiment with the following codes. I called globals() and locals() in the execed string, it seems that the locals() is different in the function definition scope when exec() is executed. "self" is in the locals() when in exec(), however, in the function definition scope inside the exec(), "self" is not in the locals().
class test_wrapper_function():
def __init__(self):
# first wrapper
def temp_func():
print("locals() inside the function definition without exec:")
print(locals())
return self.func()
print("locals() outside the function definition without exec:")
print(locals())
self.wrappered_func1 = temp_func
# third wrapper using eval
define_function_str = '''def temp_func():
print("locals() inside the function definition:")
print(locals())
print("globals() inside the function definition:")
print(globals())
return self.func()
print("locals() outside the function definition:")
print(locals())
print("globals() outside the function definition:")
print(globals())
self.wrappered_func2 = temp_func'''
exec(define_function_str)
# call locals() here, it will contains temp_func
def func(self):
print("hi!")
t = test_wrapper_function()
print("**********************************************")
t.wrappered_func1()
t.wrappered_func2()
I have read this link. In the exec(), memeber function, attribute of "self" can be accessed without problem, while in the function difinition in the exec(), "self" is not available any more. Why does this happen?
Why I want to do this:
I am building a PyQt program. I want to create several similar slots(). These slots can be generated by calling one member function with different arguments. I decided to generate these slots using exec() function of python. I also searched with the keyword "nested name scope in python exec", I found this question may be related, but there is no useful answer.
To be more specific. I want to define a family of slots like func_X (X can be 'a', 'b', 'c'...), each do something like self.do_something_on(X). Here, do_something is a member function of my QWidget. So I use a for loop to create these slots function. I used codes like this:
class MyWidget():
def __init__(self):
self.create_slots_family()
def do_something(self, character):
# in fact, this function is much more complex. Do some simplification.
print(character)
def create_slots_i(self, character):
# want to define a function like this:
# if character is 'C', define self.func_C such that self.func_C() works like self.do_something(C)
create_slot_command_str = "self.func_" + character + " = lambda:self.do_something('" + character + "')"
print(create_slot_command_str)
exec(create_slot_command_str)
def create_slots_family(self):
for c in ["A", "B", "C", "D"]:
self.create_slots_i(c)
my_widget = MyWidget()
my_widget.func_A()
Note that, as far as I know, the Qt slots should not accept any parameter, so I have to wrap self.do_something(character) to be a series function self.func_A, self.func_C and so on for all the possible characters.
So the above is what I want to do orignially.
EDIT1：
(You May ignore this session, I have figured something out, and what I currently want to ask is described as the session EDIT2. This simplified version of original question can also be viewed as a sepecial case of the question described in session EDIT2)
As #Mad Physicist suggested. I provide a simplified version here, deleting some codes used for experiments.
class test_wrapper_function():
def __init__(self):
define_function_str = '''\
def temp_func():
return self.func()
self.wrappered_func2 = temp_func'''
exec(define_function_str)
def func(self):
print("hi!")
t = test_wrapper_function()
t.wrappered_func2()
I expected this to print a "hi". However, I got the following exception:
Traceback (most recent call last):
File "D:/workfiles/test_eval_class4.py", line 12, in <module>
t.wrappered_func2()
File "<string>", line 2, in temp_func
NameError: name 'self' is not defined
Using Exec
You've already covered most of the problems and workarounds with exec, but I feel that there is still value in adding a summary.
The key issue is that exec only knows about globals and locals, but not about free variables and the non-local namespace. That is why the docs say
If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.
There is no way to make it run as though it were in a method body. However, as you've already noted, you can make exec create a closure and use that instead of the internal namespace by adding a method body to your snippet. However, there are still a couple of subtle restrictions there.
Your example of what you are trying to do showcases the issues perfectly, so I will use a modified version of that. The goal is to make a method that binds to self and has a variable argument in the exec string.
class Test:
def create_slots_i(self, c):
create_slot_command_str = f"self.func_{c} = lambda: self.do_something('{c}')"
exec(create_slot_command_str)
def do_something(self, c):
print(f'I did {c}!')
There are different ways of getting exec to "see" variables: literals, globals, and internal closures.
Literals. This works robustly, but only for simple types that can be easily instantiated from a string. The usage of c above is a perfect example. This will not help you with a complex object like self:
>>> t = Test()
>>> t.create_slots_i('a')
>>> t.func_a()
...
NameError: name 'self' is not defined
This happens exactly because exec has no concept of free variables. Since self is passed to it via the default locals(), it does not bind the reference to a closure.
globals. You can pass in a name self to exec via globals. There are a couple of ways of doing this, each with its own issues. Remember that globals are accessed by a function through its __globals__ (look at the table under "Callable types") attribute. Normally __globals__ refers to the __dict__ of the module in which a function is defined. In exec, this is the case by default as well, since that's what globals() returns.
Add to globals: You can create a global variable named self, which will make your problem go away, sort of:
>>> self = t
>>> t.func_a()
I did a!
But of course this is a house of cards that falls apart as soon as you delete, self, modify it, or try to run this on multiple instances:
>>> del self
>>> t.func_a()
...
NameError: name 'self' is not defined
Copy globals. A much more versatile solution, on the surface of it, is to copy globals() when you run exec in create_slots_i:
def create_slots_i(self, c):
create_slot_command_str = f"self.func_{c} = lambda: self.do_something('{c}')"
g = globals().copy()
g['self'] = self
exec(create_slot_command_str, g)
This appears to work normally, and for a very limited set of cases, it actually does:
>>> t = Test()
>>> t.create_slots_i('a')
>>> t.func_a()
I did a!
But now, your function's __globals__ attribute is no longer bound to the module you created it in. If it uses any other global values, especially ones that might change, you will not be able to see the changes. For limited functionality, this is OK, but in the general case, it can be a severe handicap.
Internal Closures. This is the solution you already hit upon, where you create a closure within the exec string to let it know that you have a free variable by artificial means. For example:
class Test:
def create_slots_i(self, c):
create_slot_command_str = f"""def make_func(self):
def func_{c}():
self.do_something('{c}')
return func_{c}
self.func_{c} = make_func(self)"""
g = globals().copy()
g['self'] = self
exec(create_slot_command_str, g)
def do_something(self, c):
print(f'I did {c}!')
This approach works completely:
>>> t = Test()
>>> t.create_slots_i('a')
>>> t.func_a()
I did a!
The only real drawbacks here are security, which is always a problem with exec, and the sheer awkwardness of this monstrosity.
A Better Way
Since you are already creating closures, there is really no need to use exec at all. In fact, the only thing you are really doing is creating methods so that self.func_... will bind the method for you, since you need a function with the signature of your slot and access to self. You can write a simple method that will generate functions that you can assign to your slots directly. The advantage of doing it this way is that (a) you avoid calling exec entirely, and (b) you don't need to have a bunch of similarly named auto-generated methods polluting your class namespace. The slot generator would look something like this:
def create_slots_i(self, c):
def slot_func():
self.do_something(c) # This is a real closure now
slot_func.__name__ = f'func_{c}'
return slot_func
Since you will not be referring to these function objects anywhere except your slots, __name__ is the only way to get the "name" under which they were stored. That is the same thing that def does for you under the hood.
You can now assign slots directly:
some_widget.some_signal.connect(self.create_slots_i('a'))
Note
I originally had a more complex approach in mind for you, since I thought you cared about generating bound methods, instead of just setting __name__. In case you have a sufficiently complex scenario where it still applies, here is my original blurb:
A quick recap of the descriptor protocol: when you bind a function with the dot operator, e.g., t.func_a, python looks at the class for descriptors with that name. If your class has a data descriptor (like property, but not functions), then that descriptor will shadow anything you may have placed in the instance __dict__. However, if you have a non-data descriptor (one a __get__ method but without a __set__ method, like a function object), then it will only be bound if an instance attribute does not shadow it. Once this decision has been made, actually invoking the descriptor protocol involves calling type(t).func_a.__get__(t). That's how a bound method knows about self.
Now you can return a bound method from within your generator:
def create_slots_i(self, c):
def slot_func(self):
self.do_something(c) # This is a closure on `c`, but not on `self` until you bind it
slot_func.__name__ = f'func_{c}'
return slot_func.__get__(self)
Why this phenomena happen:
Actually the answer of the question 4 listed above can answer this question.
When call exec() on one code string, the code string is first compiled. I suppose that during compiling, the provided globals and locals is not considered. The symbol in the exec()ed code str is compiled to be in the globals. So the function defined in the code str will be considered using global variables, and thus __closure__ is set to None.
Refer to this answer for more information about what the func exec does.
How to deal with this phenomena:
Imitating the solutions provided in the previous questions, for the minimal demostration the question, it can also be modified this way to work:
a=1 # moving out of the variable 'code'
code = """\
def f1():
print(a)
print(f1.__closure__)
f1()
"""
def foo():
exec(code)
foo()
Although the __closure__ is still None, the exception can be avoided because now only the global symbol is needed and __closure__ should also be None if correctly set. You can read the part The reason why the solutions work in the question body for more information.
This was originally added in Revision 4 of the question.
TL;DR
To set correct __closure__ attribute of function defined in the code string passed to exec() function. Just wrap the total code string with a function definition.
I provide an example here to demonstrate all possible situations. Suppose you want to define a function named foo inside a code string used by exec(). The foo use function, variables that defined inside and outside the code string:
def f1():
outside_local_variable = "this is local variable defined outside code str"
def outside_local_function():
print("this is function defined outside code str")
code = """\
local_variable = "this is local variable defined inside code str"
def local_function():
print("this is function defined inside code str")
def foo():
print(local_variable)
local_function()
print(outside_local_variable)
outside_local_function()
foo()
"""
exec(code)
f1()
It can be wrapper like this:
def f1():
outside_local_variable = "this is local variable defined outside code str"
def outside_local_function():
print("this is function defined outside code str")
code = """\
def closure_helper_func(outside_local_variable, outside_local_function):
local_variable = "this is local variable defined inside code str"
def local_function():
print("this is function defined inside code str")
def foo():
print(local_variable)
local_function()
print(outside_local_variable)
outside_local_function()
foo()
closure_helper_func(outside_local_variable, outside_local_function)
"""
exec(code)
f1()
Detailed explanation:
Why the __closure__ attribute is not corretly set:
please refer to The community wiki answer.
How to set the __closure__ attribute to what's expected:
Just wrap the whole code str with a helper function definition and call the helper function once, then during compiling, the variables are considered to be local, and will be stored in the __closure__ attribute.
For the minimal demonstration in the question, it can be modified to following:
code = """\
def closure_helper_func():
a=1
def f1():
print(a)
print(f1.__closure__)
f1()
closure_helper_func()
"""
def foo():
exec(code)
foo()
This output as expected
(<cell at 0x0000019CE6239A98: int object at 0x00007FFF42BFA1A0>,)
1
The example above provide a way to add symbols that defined in the code str to the __closure__ For example, in the minimal demo, a=1 is a defined inside the code str. But what if one want to add the local symbols defined outside the code str? For example, in the code snippet in EDIT1 session, the self symbol needs to be added to the __closure__, and the symbol is provided in the locals() when exec() is called. Just add the name of these symbols to the arguments of helper function and you can handle this situation.
The following shows how to fix the problem in EDIT1 session.
class test_wrapper_function():
def __init__(self):
define_function_str = '''\
def closure_helper_func(self):
def temp_func():
return self.func()
self.wrappered_func2 = temp_func
closure_helper_func(self)
'''
exec(define_function_str)
def func(self):
print("hi!")
t = test_wrapper_function()
t.wrappered_func2()
The following shows how to fix the codes in the session "Why I want to do this"
class MyWidget():
def __init__(self):
self.create_slots_family()
def do_something(self, character):
# in fact, this function is much more complex. Do some simplification.
print(character)
def create_slots_i(self, character):
# want to define a function like this:
# if character is 'C', define self.func_C such that self.func_C() works like self.do_something(C)
# create_slot_command_str = "self.func_" + character + " = lambda:self.do_something('" + character + "')"
create_slot_command_str = """
def closure_helper_func(self):
self.func_""" + character + " = lambda:self.do_something('" + character + """')
closure_helper_func(self)
"""
# print(create_slot_command_str)
exec(create_slot_command_str)
def create_slots_family(self):
for c in ["A", "B", "C", "D"]:
self.create_slots_i(c)
my_widget = MyWidget()
my_widget.func_A()
This solution seems to be too tricky. However, I can not find a more elegant way to declare that some variables should be local symbol during compiling.
This question already has answers here:
What is the purpose of the `self` parameter? Why is it needed?
(26 answers)
Closed 6 months ago.
When defining a method on a class in Python, it looks something like this:
class MyClass(object):
def __init__(self, x, y):
self.x = x
self.y = y
But in some other languages, such as C#, you have a reference to the object that the method is bound to with the "this" keyword without declaring it as an argument in the method prototype.
Was this an intentional language design decision in Python or are there some implementation details that require the passing of "self" as an argument?
I like to quote Peters' Zen of Python. "Explicit is better than implicit."
In Java and C++, 'this.' can be deduced, except when you have variable names that make it impossible to deduce. So you sometimes need it and sometimes don't.
Python elects to make things like this explicit rather than based on a rule.
Additionally, since nothing is implied or assumed, parts of the implementation are exposed. self.__class__, self.__dict__ and other "internal" structures are available in an obvious way.
It's to minimize the difference between methods and functions. It allows you to easily generate methods in metaclasses, or add methods at runtime to pre-existing classes.
e.g.
>>> class C:
... def foo(self):
... print("Hi!")
...
>>>
>>> def bar(self):
... print("Bork bork bork!")
...
>>>
>>> c = C()
>>> C.bar = bar
>>> c.bar()
Bork bork bork!
>>> c.foo()
Hi!
>>>
It also (as far as I know) makes the implementation of the python runtime easier.
I suggest that one should read Guido van Rossum's blog on this topic - Why explicit self has to stay.
When a method definition is decorated, we don't know whether to automatically give it a 'self' parameter or not: the decorator could turn the function into a static method (which has no 'self'), or a class method (which has a funny kind of self that refers to a class instead of an instance), or it could do something completely different (it's trivial to write a decorator that implements '#classmethod' or '#staticmethod' in pure Python). There's no way without knowing what the decorator does whether to endow the method being defined with an implicit 'self' argument or not.
I reject hacks like special-casing '#classmethod' and '#staticmethod'.
Python doesn't force you on using "self". You can give it whatever name you want. You just have to remember that the first argument in a method definition header is a reference to the object.
Also allows you to do this: (in short, invoking Outer(3).create_inner_class(4)().weird_sum_with_closure_scope(5) will return 12, but will do so in the craziest of ways.
class Outer(object):
def __init__(self, outer_num):
self.outer_num = outer_num
def create_inner_class(outer_self, inner_arg):
class Inner(object):
inner_arg = inner_arg
def weird_sum_with_closure_scope(inner_self, num)
return num + outer_self.outer_num + inner_arg
return Inner
Of course, this is harder to imagine in languages like Java and C#. By making the self reference explicit, you're free to refer to any object by that self reference. Also, such a way of playing with classes at runtime is harder to do in the more static languages - not that's it's necessarily good or bad. It's just that the explicit self allows all this craziness to exist.
Moreover, imagine this: We'd like to customize the behavior of methods (for profiling, or some crazy black magic). This can lead us to think: what if we had a class Method whose behavior we could override or control?
Well here it is:
from functools import partial
class MagicMethod(object):
"""Does black magic when called"""
def __get__(self, obj, obj_type):
# This binds the <other> class instance to the <innocent_self> parameter
# of the method MagicMethod.invoke
return partial(self.invoke, obj)
def invoke(magic_self, innocent_self, *args, **kwargs):
# do black magic here
...
print magic_self, innocent_self, args, kwargs
class InnocentClass(object):
magic_method = MagicMethod()
And now: InnocentClass().magic_method() will act like expected. The method will be bound with the innocent_self parameter to InnocentClass, and with the magic_self to the MagicMethod instance. Weird huh? It's like having 2 keywords this1 and this2 in languages like Java and C#. Magic like this allows frameworks to do stuff that would otherwise be much more verbose.
Again, I don't want to comment on the ethics of this stuff. I just wanted to show things that would be harder to do without an explicit self reference.
I think it has to do with PEP 227:
Names in class scope are not accessible. Names are resolved in the
innermost enclosing function scope. If a class definition occurs in a
chain of nested scopes, the resolution process skips class
definitions. This rule prevents odd interactions between class
attributes and local variable access. If a name binding operation
occurs in a class definition, it creates an attribute on the resulting
class object. To access this variable in a method, or in a function
nested within a method, an attribute reference must be used, either
via self or via the class name.
I think the real reason besides "The Zen of Python" is that Functions are first class citizens in Python.
Which essentially makes them an Object. Now The fundamental issue is if your functions are object as well then, in Object oriented paradigm how would you send messages to Objects when the messages themselves are objects ?
Looks like a chicken egg problem, to reduce this paradox, the only possible way is to either pass a context of execution to methods or detect it. But since python can have nested functions it would be impossible to do so as the context of execution would change for inner functions.
This means the only possible solution is to explicitly pass 'self' (The context of execution).
So i believe it is a implementation problem the Zen came much later.
As explained in self in Python, Demystified
anything like obj.meth(args) becomes Class.meth(obj, args). The calling process is automatic while the receiving process is not (its explicit). This is the reason the first parameter of a function in class must be the object itself.
class Point(object):
def __init__(self,x = 0,y = 0):
self.x = x
self.y = y
def distance(self):
"""Find distance from origin"""
return (self.x**2 + self.y**2) ** 0.5
Invocations:
>>> p1 = Point(6,8)
>>> p1.distance()
10.0
init() defines three parameters but we just passed two (6 and 8). Similarly distance() requires one but zero arguments were passed.
Why is Python not complaining about this argument number mismatch?
Generally, when we call a method with some arguments, the corresponding class function is called by placing the method's object before the first argument. So, anything like obj.meth(args) becomes Class.meth(obj, args). The calling process is automatic while the receiving process is not (its explicit).
This is the reason the first parameter of a function in class must be the object itself. Writing this parameter as self is merely a convention. It is not a keyword and has no special meaning in Python. We could use other names (like this) but I strongly suggest you not to. Using names other than self is frowned upon by most developers and degrades the readability of the code ("Readability counts").
...
In, the first example self.x is an instance attribute whereas x is a local variable. They are not the same and lie in different namespaces.
Self Is Here To Stay
Many have proposed to make self a keyword in Python, like this in C++ and Java. This would eliminate the redundant use of explicit self from the formal parameter list in methods. While this idea seems promising, it's not going to happen. At least not in the near future. The main reason is backward compatibility. Here is a blog from the creator of Python himself explaining why the explicit self has to stay.
The 'self' parameter keeps the current calling object.
class class_name:
class_variable
def method_name(self,arg):
self.var=arg
obj=class_name()
obj.method_name()
here, the self argument holds the object obj. Hence, the statement self.var denotes obj.var
There is also another very simple answer: according to the zen of python, "explicit is better than implicit".
I am not sure if the title will match the question I am about to ask but please feel free to update it if you know a better title which can help everyone.
So let's say we have the following definition:
>>> def helloFunction():
name = "Hello World"
so when I type in the following code, that returns an empty dictionary.
>>> helloFunction.__dict__
{}
I am not sure if this is how it should be but let's continue. Interestingly, I can do the following:
>>> helloFunction.hello = "world"
>>> helloFunction.__dict__
{'hello': 'world'}
and when I type in the following code, it tells me helloFunction is indeed a function.
>>> type(helloFunction)
<type 'function'>
I am coming from C# and this behavior is little odd to me. How come Python works like this? Is a function an object? How should I interpret this situation? And also where would I need this type of functionality?
Update
While I was composing this question, I realized __class__ is defined on helloFunction.
>>> helloFunction.__class__
<type 'function'>
So it seems like function is indeed a class type?
Pep 232 added "function attributes" to the language. You can read that if you want all the official reasoning. The reality of the situation boils down to this sentence in the intro:
func_doc has the
interesting property that there is special syntax in function (and
method) definitions for implicitly setting the attribute. This
convenience has been exploited over and over again, overloading
docstrings with additional semantics.
Emphasis mine. People were using the __doc__ attribute to smuggle all sorts of function metadata; it seemed more natural to provide a real place to do that.
As for some more specific questions:
Is a function an object?
Oh yes. Functions are first-class objects in python. You can pass references to them as arguments to other functions all you like. Were they not first-class, I couldn't do this:
def increment(x):
return x+1
map(increment,[1,2,3]) # python2 `map`, for brevity
Out[3]: [2, 3, 4]
And also where would I need this type of functionality?
You generally don't. Not often. But it can be useful when you want to store metadata about a function.
Say I wanted to wrap a function with a decorator that records how many times it's been called. That's easy since we can plop that info into the function's __dict__.
def count_calls(func):
def _inner(*args, **kwargs):
before = getattr(func,'times_called',0)
func.times_called = before + 1
print('func has now been called {} times'.format(func.times_called))
return func(*args,**kwargs)
return _inner
#count_calls
def add(x,y):
return x+y
add(3,4)
func has now been called 1 times
Out[7]: 7
add(2,3)
func has now been called 2 times
Out[8]: 5
A function is an object and - like most objects in Python - it has a dictionary. One usage example I've seen in the wild is with the web framework CherryPy, where it's used to indicate which methods are to web access:
import cherrypy
class HelloWorld(object):
def index(self):
return "Hello World!"
index.exposed = True
When a path is accessed, the dispatcher can check that the corresponding handler method has its exposed attribute set to True and respond to it, allowing for both accessible and private methods to be safely defined on the controller.
Another use I've seen was a decorator that counted the number of times a function was called:
def call_counter(fn):
fn.count = 0
def _fn(*args, **kwargs):
fn.count += 1
return fn(*arg, **kwargs)
return _fn
Partly quote from Learning Python (Mark Lutz):
Like everything else in Python, functions are just objects; they are
recorded explicitly in memory at program execution time. In fact,
besides calls, functions allow arbitrary attributes to be attached to
record information for later use.
def func(): ... # Create function object
func() # Call object
func.attr = value # Attach attributes
I know a ton has been written on this subject. I cannot, however, absorb much of it. Perhaps because I'm a complete novice teaching myself without the benefit of any training in computer science. Regardless, maybe if some of you big brains chime in on this specific example, you'll help other beginners like me.
So, I've written the following function which works just fine when I call it (as a module?) as it's own file called 'funky.py':
I type the following into my terminal:
python classy.py
and it runs fine.
def load_deck():
suite = ('Spades', 'Hearts')
rank = ('2', '3')
full_deck = {}
i = 0
for s in suite:
for r in rank:
full_deck[i] = "%s of %s" % (r, s)
i += 1
return full_deck
print load_deck()
When I put the same function in a class, however, I get an error.
Here's my code for 'classy.py':
class GAME():
def load_deck():
suite = ('Spades', 'Hearts')
rank = ('2', '3')
full_deck = {}
i = 0
for s in suite:
for r in rank:
full_deck[i] = "%s of %s" % (r, s)
i += 1
return full_deck
MyGame = GAME()
print MyGame.load_deck()
I get the following error:
Traceback (most recent call last):
File "classy.py", line 15, in <module>
print MyGame.load_deck()
TypeError: load_deck() takes no arguments (1 given)
So, I changed the definition line to the following and it works fine:
def load_deck(self):
What is it about putting a function in a class that demands the use of 'self'. I understand that 'self' is just a convention. So, why is any argument needed at all? Do functions behave differently when they are called from within a class?
Also, and this is almost more important, why does my class work without the benefit of using init ? What would using init do for my class?
Basically, if someone has the time to explain this to me like i'm a 6 year-old, it would help. Thanks in advance for any help.
Defining a function in a class definition invokes some magic that turns it into a method descriptor. When you access foo.method it will automatically create a bound method and pass the object instance as the first parameter. You can avoid this by using the #staticmethod decorator.
__init__ is simply a method called when your class is created to do optional setup. __new__ is what actually creates the object.
Here are some examples
>>> class Foo(object):
def bar(*args, **kwargs):
print args, kwargs
>>> foo = Foo()
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x01C9FEB0>>
>>> Foo.bar
<unbound method Foo.bar>
>>> foo.bar()
(<__main__.Foo object at 0x01C9FEB0>,) {}
>>> Foo.bar()
Traceback (most recent call last):
File "<pyshell#29>", line 1, in <module>
Foo.bar()
TypeError: unbound method bar() must be called with Foo instance as first argument (got nothing instead)
>>> Foo.bar(foo)
(<__main__.Foo object at 0x01C9FEB0>,) {}
So, why is any argument needed at all?
To access attributes on the current instance of the class.
Say you have a class with two methods, load_deck and shuffle. At the end of load_deck you want to shuffle the deck (by calling the shuffle method)
In Python you'd do something like this:
class Game(object):
def shuffle(self, deck):
return random.shuffle(deck)
def load_deck(self):
# ...
return self.shuffle(full_deck)
Compare this to the roughly-equivalent C++ code:
class Game {
shuffle(deck) {
return random.shuffle(deck);
}
load_deck() {
// ...
return shuffle(full_deck)
}
}
On shuffle(full_deck) line, first it looks for a local variable called shuffle - this doesn't exist, to next it checks one level higher, and finds an instance-method called shuffle (if this doesn't exist, it would check for a global variable with the right name)
This is okay, but it's not clear if shuffle refers to some local variable, or the instance method. To address this ambiguity, instance-methods or instance-attributes can also be accessed via this:
...
load_deck() {
// ...
return this->shuffle(full_deck)
}
this is almost identical to Python's self, except it's not passed as an argument.
Why is it useful to have self as an argument useful? The FAQ lists several good reasons - these can be summarised by a line in "The Zen of Python":
Explicit is better than implicit.
This is backed up by a post in The History of Python blog,
I decided to give up on the idea of implicit references to instance variables. Languages like C++ let you write this->foo to explicitly reference the instance variable foo (in case there’s a separate local variable foo). Thus, I decided to make such explicit references the only way to reference instance variables. In addition, I decided that rather than making the current object ("this") a special keyword, I would simply make "this" (or its equivalent) the first named argument to a method. Instance variables would just always be referenced as attributes of that argument.
With explicit references, there is no need to have a special syntax for method definitions nor do you have to worry about complicated semantics concerning variable lookup. Instead, one simply defines a function whose first argument corresponds to the instance, which by convention is named "self."
If you don't intent to use self you should probably declare the method to be a staticmethod.
class Game:
#staticmethod
def load_deck():
....
This undoes the automatic default packing that ordinarily happens to turn a function in a class scope into a method taking the instance as an argument.
Passing arguments you don't use is disconcerting to others trying to read your code.
Most classes have members. Yours doesn't, so all of its methods should be static. As your project develops, you will probably find data that should be accessible to all of the functions in it, and you will put those in self, and pass it around to all of them.
In this context, where the application itself is your primary object, __init__ is just the function that would initialize all of those shared values.
This is the first step toward an object-oriented style, wherein smaller pieces of data get used as objects themselves. But this is a normal stage in moving from straight scripting to OO programming.
This question already has answers here:
What is the purpose of the `self` parameter? Why is it needed?
(26 answers)
Closed 6 months ago.
When defining a method on a class in Python, it looks something like this:
class MyClass(object):
def __init__(self, x, y):
self.x = x
self.y = y
But in some other languages, such as C#, you have a reference to the object that the method is bound to with the "this" keyword without declaring it as an argument in the method prototype.
Was this an intentional language design decision in Python or are there some implementation details that require the passing of "self" as an argument?
I like to quote Peters' Zen of Python. "Explicit is better than implicit."
In Java and C++, 'this.' can be deduced, except when you have variable names that make it impossible to deduce. So you sometimes need it and sometimes don't.
Python elects to make things like this explicit rather than based on a rule.
Additionally, since nothing is implied or assumed, parts of the implementation are exposed. self.__class__, self.__dict__ and other "internal" structures are available in an obvious way.
It's to minimize the difference between methods and functions. It allows you to easily generate methods in metaclasses, or add methods at runtime to pre-existing classes.
e.g.
>>> class C:
... def foo(self):
... print("Hi!")
...
>>>
>>> def bar(self):
... print("Bork bork bork!")
...
>>>
>>> c = C()
>>> C.bar = bar
>>> c.bar()
Bork bork bork!
>>> c.foo()
Hi!
>>>
It also (as far as I know) makes the implementation of the python runtime easier.
I suggest that one should read Guido van Rossum's blog on this topic - Why explicit self has to stay.
When a method definition is decorated, we don't know whether to automatically give it a 'self' parameter or not: the decorator could turn the function into a static method (which has no 'self'), or a class method (which has a funny kind of self that refers to a class instead of an instance), or it could do something completely different (it's trivial to write a decorator that implements '#classmethod' or '#staticmethod' in pure Python). There's no way without knowing what the decorator does whether to endow the method being defined with an implicit 'self' argument or not.
I reject hacks like special-casing '#classmethod' and '#staticmethod'.
Python doesn't force you on using "self". You can give it whatever name you want. You just have to remember that the first argument in a method definition header is a reference to the object.
Also allows you to do this: (in short, invoking Outer(3).create_inner_class(4)().weird_sum_with_closure_scope(5) will return 12, but will do so in the craziest of ways.
class Outer(object):
def __init__(self, outer_num):
self.outer_num = outer_num
def create_inner_class(outer_self, inner_arg):
class Inner(object):
inner_arg = inner_arg
def weird_sum_with_closure_scope(inner_self, num)
return num + outer_self.outer_num + inner_arg
return Inner
Of course, this is harder to imagine in languages like Java and C#. By making the self reference explicit, you're free to refer to any object by that self reference. Also, such a way of playing with classes at runtime is harder to do in the more static languages - not that's it's necessarily good or bad. It's just that the explicit self allows all this craziness to exist.
Moreover, imagine this: We'd like to customize the behavior of methods (for profiling, or some crazy black magic). This can lead us to think: what if we had a class Method whose behavior we could override or control?
Well here it is:
from functools import partial
class MagicMethod(object):
"""Does black magic when called"""
def __get__(self, obj, obj_type):
# This binds the <other> class instance to the <innocent_self> parameter
# of the method MagicMethod.invoke
return partial(self.invoke, obj)
def invoke(magic_self, innocent_self, *args, **kwargs):
# do black magic here
...
print magic_self, innocent_self, args, kwargs
class InnocentClass(object):
magic_method = MagicMethod()
And now: InnocentClass().magic_method() will act like expected. The method will be bound with the innocent_self parameter to InnocentClass, and with the magic_self to the MagicMethod instance. Weird huh? It's like having 2 keywords this1 and this2 in languages like Java and C#. Magic like this allows frameworks to do stuff that would otherwise be much more verbose.
Again, I don't want to comment on the ethics of this stuff. I just wanted to show things that would be harder to do without an explicit self reference.
I think it has to do with PEP 227:
Names in class scope are not accessible. Names are resolved in the
innermost enclosing function scope. If a class definition occurs in a
chain of nested scopes, the resolution process skips class
definitions. This rule prevents odd interactions between class
attributes and local variable access. If a name binding operation
occurs in a class definition, it creates an attribute on the resulting
class object. To access this variable in a method, or in a function
nested within a method, an attribute reference must be used, either
via self or via the class name.
I think the real reason besides "The Zen of Python" is that Functions are first class citizens in Python.
Which essentially makes them an Object. Now The fundamental issue is if your functions are object as well then, in Object oriented paradigm how would you send messages to Objects when the messages themselves are objects ?
Looks like a chicken egg problem, to reduce this paradox, the only possible way is to either pass a context of execution to methods or detect it. But since python can have nested functions it would be impossible to do so as the context of execution would change for inner functions.
This means the only possible solution is to explicitly pass 'self' (The context of execution).
So i believe it is a implementation problem the Zen came much later.
As explained in self in Python, Demystified
anything like obj.meth(args) becomes Class.meth(obj, args). The calling process is automatic while the receiving process is not (its explicit). This is the reason the first parameter of a function in class must be the object itself.
class Point(object):
def __init__(self,x = 0,y = 0):
self.x = x
self.y = y
def distance(self):
"""Find distance from origin"""
return (self.x**2 + self.y**2) ** 0.5
Invocations:
>>> p1 = Point(6,8)
>>> p1.distance()
10.0
init() defines three parameters but we just passed two (6 and 8). Similarly distance() requires one but zero arguments were passed.
Why is Python not complaining about this argument number mismatch?
Generally, when we call a method with some arguments, the corresponding class function is called by placing the method's object before the first argument. So, anything like obj.meth(args) becomes Class.meth(obj, args). The calling process is automatic while the receiving process is not (its explicit).
This is the reason the first parameter of a function in class must be the object itself. Writing this parameter as self is merely a convention. It is not a keyword and has no special meaning in Python. We could use other names (like this) but I strongly suggest you not to. Using names other than self is frowned upon by most developers and degrades the readability of the code ("Readability counts").
...
In, the first example self.x is an instance attribute whereas x is a local variable. They are not the same and lie in different namespaces.
Self Is Here To Stay
Many have proposed to make self a keyword in Python, like this in C++ and Java. This would eliminate the redundant use of explicit self from the formal parameter list in methods. While this idea seems promising, it's not going to happen. At least not in the near future. The main reason is backward compatibility. Here is a blog from the creator of Python himself explaining why the explicit self has to stay.
The 'self' parameter keeps the current calling object.
class class_name:
class_variable
def method_name(self,arg):
self.var=arg
obj=class_name()
obj.method_name()
here, the self argument holds the object obj. Hence, the statement self.var denotes obj.var
There is also another very simple answer: according to the zen of python, "explicit is better than implicit".