EDIT2：
A minimal demonstration is:
code = """\
a=1
def f1():
print(a)
print(f1.__closure__)
f1()
"""
def foo():
exec(code)
foo()
Which gives:
None
Traceback (most recent call last):
File "D:/workfiles/test_eval_rec.py", line 221, in <module>
foo()
File "D:/workfiles//test_eval_rec.py", line 219, in foo
exec(code)
File "<string>", line 5, in <module>
File "<string>", line 3, in f1
NameError: name 'a' is not defined
It can be seen that the __closure__ attribute of function defined inside code str passed to exec() is None, making calling the function fails.
Why does this happen and how can I define a function successfully?
I find several questions that may be related.
Closure lost during callback defined in exec()
Using exec() with recursive functions
Why exec() works differently when invoked inside of function and how to avoid it
Why are closures broken within exec?
NameError: name 'self' is not defined IN EXEC/EVAL
These questions are all related to "defining a function insdie exec()". I think the fourth question here is closest to the essence of these problems. The common cause of these problems is that when defining a function in exec(), the __closure__ attribute of the function object can not be set correctly and will always be None. However, many existing answers to this question didn't realize this point.
Why these questions are caused by wrong __closure__:
When defining a function, __closure__ attribute is set to a dict that contains all local symbols (at the place where the keyword def is used) that is used inside the newly defined funtion. When calling a function, local symbol tables will be retrived from the __closure__ attribute. Since the __closure__ is set to None, the local symbol tables can not be retrived as expected, making the function call fail.
These answers work by making None a correct __closure__ attribute:
Existing solutions to the questions listed above solve these problems by getting the function definition rid of the usage of local symbol, i.e, they make the local symbols used(variable, function definition) global by passing globals() as locals of exec or by using keyword global explicitly in the code string.
Why existing solution unsatisfying:
These solutions I think is just an escape of the core problem of setting __closure__ correctly when define a functioni inside exec(). And as symbols used in the function definition is made global, these solutions will produce redundant global symbol which I don't want.
Original Questions：
(You May ignore this session, I have figured something out, and what I currently want to ask is described as the session EDIT2. The original question can be viewed as a sepecial case of the question described in session EDIT2)
original title of this question is: Wrapping class function to new function with exec() raise NameError that ‘self’ is not defined
I want to wrap an existing member function to a new class function. However, exec() function failed with a NameError that ‘self’ is not defined.
I did some experiment with the following codes. I called globals() and locals() in the execed string, it seems that the locals() is different in the function definition scope when exec() is executed. "self" is in the locals() when in exec(), however, in the function definition scope inside the exec(), "self" is not in the locals().
class test_wrapper_function():
def __init__(self):
# first wrapper
def temp_func():
print("locals() inside the function definition without exec:")
print(locals())
return self.func()
print("locals() outside the function definition without exec:")
print(locals())
self.wrappered_func1 = temp_func
# third wrapper using eval
define_function_str = '''def temp_func():
print("locals() inside the function definition:")
print(locals())
print("globals() inside the function definition:")
print(globals())
return self.func()
print("locals() outside the function definition:")
print(locals())
print("globals() outside the function definition:")
print(globals())
self.wrappered_func2 = temp_func'''
exec(define_function_str)
# call locals() here, it will contains temp_func
def func(self):
print("hi!")
t = test_wrapper_function()
print("**********************************************")
t.wrappered_func1()
t.wrappered_func2()
I have read this link. In the exec(), memeber function, attribute of "self" can be accessed without problem, while in the function difinition in the exec(), "self" is not available any more. Why does this happen?
Why I want to do this:
I am building a PyQt program. I want to create several similar slots(). These slots can be generated by calling one member function with different arguments. I decided to generate these slots using exec() function of python. I also searched with the keyword "nested name scope in python exec", I found this question may be related, but there is no useful answer.
To be more specific. I want to define a family of slots like func_X (X can be 'a', 'b', 'c'...), each do something like self.do_something_on(X). Here, do_something is a member function of my QWidget. So I use a for loop to create these slots function. I used codes like this:
class MyWidget():
def __init__(self):
self.create_slots_family()
def do_something(self, character):
# in fact, this function is much more complex. Do some simplification.
print(character)
def create_slots_i(self, character):
# want to define a function like this:
# if character is 'C', define self.func_C such that self.func_C() works like self.do_something(C)
create_slot_command_str = "self.func_" + character + " = lambda:self.do_something('" + character + "')"
print(create_slot_command_str)
exec(create_slot_command_str)
def create_slots_family(self):
for c in ["A", "B", "C", "D"]:
self.create_slots_i(c)
my_widget = MyWidget()
my_widget.func_A()
Note that, as far as I know, the Qt slots should not accept any parameter, so I have to wrap self.do_something(character) to be a series function self.func_A, self.func_C and so on for all the possible characters.
So the above is what I want to do orignially.
EDIT1：
(You May ignore this session, I have figured something out, and what I currently want to ask is described as the session EDIT2. This simplified version of original question can also be viewed as a sepecial case of the question described in session EDIT2)
As #Mad Physicist suggested. I provide a simplified version here, deleting some codes used for experiments.
class test_wrapper_function():
def __init__(self):
define_function_str = '''\
def temp_func():
return self.func()
self.wrappered_func2 = temp_func'''
exec(define_function_str)
def func(self):
print("hi!")
t = test_wrapper_function()
t.wrappered_func2()
I expected this to print a "hi". However, I got the following exception:
Traceback (most recent call last):
File "D:/workfiles/test_eval_class4.py", line 12, in <module>
t.wrappered_func2()
File "<string>", line 2, in temp_func
NameError: name 'self' is not defined
Using Exec
You've already covered most of the problems and workarounds with exec, but I feel that there is still value in adding a summary.
The key issue is that exec only knows about globals and locals, but not about free variables and the non-local namespace. That is why the docs say
If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.
There is no way to make it run as though it were in a method body. However, as you've already noted, you can make exec create a closure and use that instead of the internal namespace by adding a method body to your snippet. However, there are still a couple of subtle restrictions there.
Your example of what you are trying to do showcases the issues perfectly, so I will use a modified version of that. The goal is to make a method that binds to self and has a variable argument in the exec string.
class Test:
def create_slots_i(self, c):
create_slot_command_str = f"self.func_{c} = lambda: self.do_something('{c}')"
exec(create_slot_command_str)
def do_something(self, c):
print(f'I did {c}!')
There are different ways of getting exec to "see" variables: literals, globals, and internal closures.
Literals. This works robustly, but only for simple types that can be easily instantiated from a string. The usage of c above is a perfect example. This will not help you with a complex object like self:
>>> t = Test()
>>> t.create_slots_i('a')
>>> t.func_a()
...
NameError: name 'self' is not defined
This happens exactly because exec has no concept of free variables. Since self is passed to it via the default locals(), it does not bind the reference to a closure.
globals. You can pass in a name self to exec via globals. There are a couple of ways of doing this, each with its own issues. Remember that globals are accessed by a function through its __globals__ (look at the table under "Callable types") attribute. Normally __globals__ refers to the __dict__ of the module in which a function is defined. In exec, this is the case by default as well, since that's what globals() returns.
Add to globals: You can create a global variable named self, which will make your problem go away, sort of:
>>> self = t
>>> t.func_a()
I did a!
But of course this is a house of cards that falls apart as soon as you delete, self, modify it, or try to run this on multiple instances:
>>> del self
>>> t.func_a()
...
NameError: name 'self' is not defined
Copy globals. A much more versatile solution, on the surface of it, is to copy globals() when you run exec in create_slots_i:
def create_slots_i(self, c):
create_slot_command_str = f"self.func_{c} = lambda: self.do_something('{c}')"
g = globals().copy()
g['self'] = self
exec(create_slot_command_str, g)
This appears to work normally, and for a very limited set of cases, it actually does:
>>> t = Test()
>>> t.create_slots_i('a')
>>> t.func_a()
I did a!
But now, your function's __globals__ attribute is no longer bound to the module you created it in. If it uses any other global values, especially ones that might change, you will not be able to see the changes. For limited functionality, this is OK, but in the general case, it can be a severe handicap.
Internal Closures. This is the solution you already hit upon, where you create a closure within the exec string to let it know that you have a free variable by artificial means. For example:
class Test:
def create_slots_i(self, c):
create_slot_command_str = f"""def make_func(self):
def func_{c}():
self.do_something('{c}')
return func_{c}
self.func_{c} = make_func(self)"""
g = globals().copy()
g['self'] = self
exec(create_slot_command_str, g)
def do_something(self, c):
print(f'I did {c}!')
This approach works completely:
>>> t = Test()
>>> t.create_slots_i('a')
>>> t.func_a()
I did a!
The only real drawbacks here are security, which is always a problem with exec, and the sheer awkwardness of this monstrosity.
A Better Way
Since you are already creating closures, there is really no need to use exec at all. In fact, the only thing you are really doing is creating methods so that self.func_... will bind the method for you, since you need a function with the signature of your slot and access to self. You can write a simple method that will generate functions that you can assign to your slots directly. The advantage of doing it this way is that (a) you avoid calling exec entirely, and (b) you don't need to have a bunch of similarly named auto-generated methods polluting your class namespace. The slot generator would look something like this:
def create_slots_i(self, c):
def slot_func():
self.do_something(c) # This is a real closure now
slot_func.__name__ = f'func_{c}'
return slot_func
Since you will not be referring to these function objects anywhere except your slots, __name__ is the only way to get the "name" under which they were stored. That is the same thing that def does for you under the hood.
You can now assign slots directly:
some_widget.some_signal.connect(self.create_slots_i('a'))
Note
I originally had a more complex approach in mind for you, since I thought you cared about generating bound methods, instead of just setting __name__. In case you have a sufficiently complex scenario where it still applies, here is my original blurb:
A quick recap of the descriptor protocol: when you bind a function with the dot operator, e.g., t.func_a, python looks at the class for descriptors with that name. If your class has a data descriptor (like property, but not functions), then that descriptor will shadow anything you may have placed in the instance __dict__. However, if you have a non-data descriptor (one a __get__ method but without a __set__ method, like a function object), then it will only be bound if an instance attribute does not shadow it. Once this decision has been made, actually invoking the descriptor protocol involves calling type(t).func_a.__get__(t). That's how a bound method knows about self.
Now you can return a bound method from within your generator:
def create_slots_i(self, c):
def slot_func(self):
self.do_something(c) # This is a closure on `c`, but not on `self` until you bind it
slot_func.__name__ = f'func_{c}'
return slot_func.__get__(self)
Why this phenomena happen:
Actually the answer of the question 4 listed above can answer this question.
When call exec() on one code string, the code string is first compiled. I suppose that during compiling, the provided globals and locals is not considered. The symbol in the exec()ed code str is compiled to be in the globals. So the function defined in the code str will be considered using global variables, and thus __closure__ is set to None.
Refer to this answer for more information about what the func exec does.
How to deal with this phenomena:
Imitating the solutions provided in the previous questions, for the minimal demostration the question, it can also be modified this way to work:
a=1 # moving out of the variable 'code'
code = """\
def f1():
print(a)
print(f1.__closure__)
f1()
"""
def foo():
exec(code)
foo()
Although the __closure__ is still None, the exception can be avoided because now only the global symbol is needed and __closure__ should also be None if correctly set. You can read the part The reason why the solutions work in the question body for more information.
This was originally added in Revision 4 of the question.
TL;DR
To set correct __closure__ attribute of function defined in the code string passed to exec() function. Just wrap the total code string with a function definition.
I provide an example here to demonstrate all possible situations. Suppose you want to define a function named foo inside a code string used by exec(). The foo use function, variables that defined inside and outside the code string:
def f1():
outside_local_variable = "this is local variable defined outside code str"
def outside_local_function():
print("this is function defined outside code str")
code = """\
local_variable = "this is local variable defined inside code str"
def local_function():
print("this is function defined inside code str")
def foo():
print(local_variable)
local_function()
print(outside_local_variable)
outside_local_function()
foo()
"""
exec(code)
f1()
It can be wrapper like this:
def f1():
outside_local_variable = "this is local variable defined outside code str"
def outside_local_function():
print("this is function defined outside code str")
code = """\
def closure_helper_func(outside_local_variable, outside_local_function):
local_variable = "this is local variable defined inside code str"
def local_function():
print("this is function defined inside code str")
def foo():
print(local_variable)
local_function()
print(outside_local_variable)
outside_local_function()
foo()
closure_helper_func(outside_local_variable, outside_local_function)
"""
exec(code)
f1()
Detailed explanation:
Why the __closure__ attribute is not corretly set:
please refer to The community wiki answer.
How to set the __closure__ attribute to what's expected:
Just wrap the whole code str with a helper function definition and call the helper function once, then during compiling, the variables are considered to be local, and will be stored in the __closure__ attribute.
For the minimal demonstration in the question, it can be modified to following:
code = """\
def closure_helper_func():
a=1
def f1():
print(a)
print(f1.__closure__)
f1()
closure_helper_func()
"""
def foo():
exec(code)
foo()
This output as expected
(<cell at 0x0000019CE6239A98: int object at 0x00007FFF42BFA1A0>,)
1
The example above provide a way to add symbols that defined in the code str to the __closure__ For example, in the minimal demo, a=1 is a defined inside the code str. But what if one want to add the local symbols defined outside the code str? For example, in the code snippet in EDIT1 session, the self symbol needs to be added to the __closure__, and the symbol is provided in the locals() when exec() is called. Just add the name of these symbols to the arguments of helper function and you can handle this situation.
The following shows how to fix the problem in EDIT1 session.
class test_wrapper_function():
def __init__(self):
define_function_str = '''\
def closure_helper_func(self):
def temp_func():
return self.func()
self.wrappered_func2 = temp_func
closure_helper_func(self)
'''
exec(define_function_str)
def func(self):
print("hi!")
t = test_wrapper_function()
t.wrappered_func2()
The following shows how to fix the codes in the session "Why I want to do this"
class MyWidget():
def __init__(self):
self.create_slots_family()
def do_something(self, character):
# in fact, this function is much more complex. Do some simplification.
print(character)
def create_slots_i(self, character):
# want to define a function like this:
# if character is 'C', define self.func_C such that self.func_C() works like self.do_something(C)
# create_slot_command_str = "self.func_" + character + " = lambda:self.do_something('" + character + "')"
create_slot_command_str = """
def closure_helper_func(self):
self.func_""" + character + " = lambda:self.do_something('" + character + """')
closure_helper_func(self)
"""
# print(create_slot_command_str)
exec(create_slot_command_str)
def create_slots_family(self):
for c in ["A", "B", "C", "D"]:
self.create_slots_i(c)
my_widget = MyWidget()
my_widget.func_A()
This solution seems to be too tricky. However, I can not find a more elegant way to declare that some variables should be local symbol during compiling.
I'm trying to understand when Python code will refer to module-level variables versus class level variables. I have the following code in a module, main.py'
## main.py
# global variable x
x = "I am global and I love it!"
class SomeClass:
x = "I am a class attribute, and life is good!"
print(x) # prints "I am a class attribute, and life is good!"
def __init__(self):
print(x) # prints "I am global and I love it!" Why?
print(x) # prints "I am global and I love it!"
SomeClass()
When I import this module, the output is:
I am a class attribute, and life is good!
I am global and I love it!
I am global and I love it!
Why does the print inside the SomeClass.__init__ method print the global variable, while the print inside the class body prints the class attribute x?
This was paraphrased from a question on the Python mailing list: https://mail.python.org/pipermail/python-list/2015-December/701167.html
Class definitions in Python create a namespace, but they do not create a new scope. This means that functions defined inside the class cannot access variables of the containing class directly. Instead they must access them as attributes of the class or an instance.
So, rather than accessing x in your example's __init__ function, you can get the class variable by using SomeClass.x or self.x.
The reason that print(x) works in the class body is that that code is running with the class namespace as its local namespace. There are a number of issues you can run into if you try to do more complicated stuff at class level though, as the scopeless namespace is a rather weird environment. For example, this will not work:
class Foo:
x = 1
dct = {i: x for i in range(10)}
You'll get a NameError about the x variable because the dictionary comprehension runs in its own scope and cannot see the class variable x, which is defined in the scopeless class namespace.
Importing a module
Whenever you import a module, Python executes all the module-level code in the order the module is written. It attaches all the defined names to the module object, and the importer of the module can access all the defined names through the module object. That way you can import the module, and the names in the module don't clobber your namespace.
import main # executes the script main.py, then creates a module
print(main.x)
Defining a class
You can think of the way that Python creates the class as similar to how it creates a module: it executes all the code in the class body, then assigns the defined names to the class. The __init__ method just becomes an attribute of the class, as well; by the time the function is called, the class has been constructed and within the function you can only refer to the x class attribute as SomeClass.x (or self.x inside of __init__). The namespace for the class definition is different than the namespace for the module, so the print(x) inside the class finds the class namespace first, since the print is in the same namespace.
Aside
Just as you can't refer to the current module when you're in, you cannot refer to the current class either. For example:
class SomeOtherClass:
x = 5
SomeOtherClass.x = 6 # This fails!
You can't refer to the class for the same reason you cannot refer to the module you are in from within a module: the name has not been created yet.
This answer was inspired by ChrisA's answer on the Python mailing list.
I've been playing with my codes a little for a while, and this one is not about a bug or anything, but i just don't understand why class main() runs without needing to initialize it...
class vars():
var1 = "Universe!"
var2 = "Oscar!"
var3 = "Rainbow!"
class main():
print (vars.var1)
def __init__(self):
print (vars.var2)
print (vars.var3)
But yes, thank you very much for reading.
Unlike many other languages, class body is an executable statement in Python and is executed immediately as the interpreter reaches the class line. When you run this "program":
class Foo:
print("hey")
it just prints "hey" without any Foo object being created.
The same applies to the function definition statement def (but not to function bodies). When you run this:
def foo(arg=print("hi")):
print("not yet")
it prints "hi", but not "not yet".
When a class is created, Python executes all of the code directly inside the class declaration in a new namespace. This is so that any variables created in the class (most commonly methods, created by ordinary function declarations like def foo(self...)) are attached to the class rather than being global.
But the code still runs immediately. If it calls print() or does something else which creates a visible side effect, that will happen now, not when the class is instantiated (called to create a new instance). If you need something to happen when the class is instantiated, write an __init__() method instead.
main is a class not a function. Thus the code contained in the class declaration runs immediately because all statements are executed as they appear in code. As a method declaration is reached, it's bound to the class as a member, so in a way methods execute as well but are not called.
When Python read your code, it looked into class vars and defined all the variables. Then, it went into class main and executed the code there, as well as defining init. Python just executes whatever which is not in a function definition.
This should be relatively simple, but I'm just missing something. I am trying to utilize a function from another module which is contained within a class. I can do it easily when there is no class involved.
# a.py
import b
b.name()
--
# b.py
def name():
print "What is your name?"
class details(object):
def age():
print "What is your age?"
When I run a i get the expected result of
What is your name?
However when i try to access "def age()" from another module it keeps giving me trouble.
Some of what I have tried so far...
# c.py
import b
b.details.age()
= TypeError: unbound method age() must be called with details instance as first argument (got nothing instead)
# c.py
from b import details
details.age()
= TypeError: unbound method age() must be called with details instance as first argument (got nothing instead)
# c.py
from b import details
b.details(age)
= NameError: name 'b' is not defined
I have tried a few others as well but too many to reasonably post. What am i doing wrong? What is the syntax to do do this? Is it even possible to execute a function when it is contained within a class in another module?
Thanks in advance
EDIT: Fixed all tabs to spaces as suggested by Mike Graham
The first parameter of all class methods in Python is a reference to the current object (normally this is called self). However, that said, you seem to be trying to use it as a static method and not as an instance method, so perhaps you meant to use the #staticmethod decorator:
class Details: # class names in Python should generally be CamelCased.
# please note the comments below
#staticmethod
def age():
print 'What is your age?'
Or, if you really want it to be an instance method, then you need to add self and change how you're referencing it:
class Details:
def age(self):
print 'What is your age?'
# c.py
from b import Details
#you must create an instance of the class before you can call methods on it.
d = Details()
d.age()
EDIT
As noted in the comments, it is rare that #staticmethod has a genuine use case (it is often better to organize your code with modules, for example). You will often come across #classmethod as an alternative. Please note, though, that methods decorated with #classmethod have a reference to the current class as the first parameter. This question addresses the major differences.
my problem is about i have a file that contain class and inside this class there is bunch of code will be executed
so whenever i import that file it will executed ! without creating an object of the class ! , here is the example
FILE X
class d:
def __init__(self):
print 'print this will NOT be printed'
print "this will be printed"
file B
import x
output is this will be printed, so my question is how to skip executing it until creating a new object?
You can't do that in Python, in Python every class is a first level object, a Python class is an object too and an class attribute can exist even if there is no instances of that class. If you just want to suppress the output of the print statement you can redirect the output of your print statements on the importing moment or create a context like the one provided in this first answer and use the __import__ statement manually.
If all you want to do is suppress the print (or any other executable statements) during import, surround them with a check for top module execution:
if __name__ == '__main__':
print 'this will be printed'
This will prevent the print during import, but allow it when the module is
executed interactively.
As others have pointed out, the second print statment is executing because it's one of the suite of statements making up the class declaration -- all of which are executed when the module they're in is imported because the declaration is part of its top-level code verses it being nested inside a function or method.
The first print statement isn't executed because it's part of a method definition, whose statements don't execute until it's called --- unlike those within a class definition. Typically a class's __init__() method is called indirectly when an instance of the class is created using the class's name, which would be d() for one named d like yours.
So, although it contradicts what's in the text of the strings being displayed, to make that second print statement only execute when instances of the class are created (just like with the first one) you'd need to also make it part of the same method (or called by it). In other words, after doing so, neither of them will execute when the file the class is in is imported, but both will when any instances of the class are created. Here's what I mean:
File x.py:
class d:
def __init__(self):
print 'print this will NOT be printed' # not true
print "this will be printed when object is created"
File b.py:
import x # no print statements execute
obj = d() # both print statements will be executed now
Your question is like: I have a function
def f():
print(1)
print(2)
How do I make print(1) executed, but not print(2)? There is really no easy way. What you have to understand is that def __init__(self) is also a statement. Your class consists of that statement and print statement. There is no easy way to execute one but not the other. Of course, if you can change the source of the class, just put the print inside __init__, where it will be called after instance creation.
(Copied from a comment above in case it is useful to future readers)
Agree with #mgilson and #EmmettJButler - this code is likely best-placed in the __init__. When Python imports a module, it executes the module-level code, including building the class definition with the class methods, etc. Therefore when you import X, and class d's definition gets built (so you can call it from B), it executes the code inside of the class. Usually this means you'll have class-level variables set and unbound methods ready to be attached to instances, but in your case it means that your statement will be printed.
As suggested by the others, refactoring the code is likely your best bet.