Say we have two scripts, script1 and script2.
script1 is defined as:
class Foo(object):
def __init__(self, name):
self.name = name
class bar(object):
def __init__(self, name):
self.name = name
def test(givenString):
return eval(givenString)
and script2 is defined as:
from .script1 import test
x = "Foo('me')"
print test(x)
script2's print statement for test(x) successfully tells me that I have a Foo object, but that doesn't make sense to me because I only imported test from script1, not Foo. I looked at the eval documentation but that didn't clear up much for me. How is it possible that a Foo object is created even when I never imported the class Foo?
eval() uses the globals of the module it is executed in. test 'lives' in the script1 global namespace, so any expression executed by eval() uses the same namespace as that function and thus can resolve Foo, bar and test.
Importing a function does not alter its namespace; the globals for test don't change merely by being called from script2. If it did, any imports in script1 would also need to be imported into script2, for each and every function you ever wanted to use. That would be incredibly impractical.
You can even see the globals for functions you import; print test.func_globals will show you the exact namespace of script1.
Related
I have file main.py which contains the following code:
class A:
def __init__(self, a):
self.a = a
def run(self):
return self.a+10
a = A(4)
print(a.run())
In file test.py, I tried to monkey patch class A in main.py as follows:
import main
class A:
def __init__(self, a):
self.a = a
def run(self):
return self.a+5
main.A = A
Unfortunately, when I run import test from a python interpreter, the module still prints out 14 as opposed to my expected output which is 9.
Is there a way to monkey patch a class inside a module before the module body is executed?
The problem here is that when you imported the main.py file, it executed the code a = A(4) using the real implementation of the class A. Then the rest of your test.py was executed and you replaced the A reference, but it was too late.
You can check that by adding in your test :
print(__name__) # __main__
print(main.__name__) # so70731368_main
print(A.__module__) # __main__
print(main.A.__module__) # __main__
print(main.a.__class__.__module__) # so70731368_main
Here, __main__ is a bit confusing but that's how Python call the first file you run (in your case test.py). The a instance is declared in the so70731368_main module, and it used the A class from the same module, you just changed A after the fact with the definition from the test file (__main__).
The fact that you need to patch two definitions (A and a) defined in the same file is very tricky. unittest.mock.patch is not powerful enough to patch inside an import (it patches after the import).
You can not, in a clean and simple way, prevent a to be instantiated as a main.A (real) class and get printed. What you can do is patch it after, for later uses, that is what you showed.
To answer directly your question : "patching" means replacing one reference by another, so the reference has to already be defined. In your example, it would require to patch between the class definition and the class instantiation (for the print to not use the real a), which is not supported.
There is no simple solution to this problem. If you have control over the code of the main.py file, then try to change it so that it does not instantiate a at import time.
I'm trying to understand when Python code will refer to module-level variables versus class level variables. I have the following code in a module, main.py'
## main.py
# global variable x
x = "I am global and I love it!"
class SomeClass:
x = "I am a class attribute, and life is good!"
print(x) # prints "I am a class attribute, and life is good!"
def __init__(self):
print(x) # prints "I am global and I love it!" Why?
print(x) # prints "I am global and I love it!"
SomeClass()
When I import this module, the output is:
I am a class attribute, and life is good!
I am global and I love it!
I am global and I love it!
Why does the print inside the SomeClass.__init__ method print the global variable, while the print inside the class body prints the class attribute x?
This was paraphrased from a question on the Python mailing list: https://mail.python.org/pipermail/python-list/2015-December/701167.html
Class definitions in Python create a namespace, but they do not create a new scope. This means that functions defined inside the class cannot access variables of the containing class directly. Instead they must access them as attributes of the class or an instance.
So, rather than accessing x in your example's __init__ function, you can get the class variable by using SomeClass.x or self.x.
The reason that print(x) works in the class body is that that code is running with the class namespace as its local namespace. There are a number of issues you can run into if you try to do more complicated stuff at class level though, as the scopeless namespace is a rather weird environment. For example, this will not work:
class Foo:
x = 1
dct = {i: x for i in range(10)}
You'll get a NameError about the x variable because the dictionary comprehension runs in its own scope and cannot see the class variable x, which is defined in the scopeless class namespace.
Importing a module
Whenever you import a module, Python executes all the module-level code in the order the module is written. It attaches all the defined names to the module object, and the importer of the module can access all the defined names through the module object. That way you can import the module, and the names in the module don't clobber your namespace.
import main # executes the script main.py, then creates a module
print(main.x)
Defining a class
You can think of the way that Python creates the class as similar to how it creates a module: it executes all the code in the class body, then assigns the defined names to the class. The __init__ method just becomes an attribute of the class, as well; by the time the function is called, the class has been constructed and within the function you can only refer to the x class attribute as SomeClass.x (or self.x inside of __init__). The namespace for the class definition is different than the namespace for the module, so the print(x) inside the class finds the class namespace first, since the print is in the same namespace.
Aside
Just as you can't refer to the current module when you're in, you cannot refer to the current class either. For example:
class SomeOtherClass:
x = 5
SomeOtherClass.x = 6 # This fails!
You can't refer to the class for the same reason you cannot refer to the module you are in from within a module: the name has not been created yet.
This answer was inspired by ChrisA's answer on the Python mailing list.
I am writing a Python script where some of the core functionalities can be done by another existing library. Unfortunately, while that library has more features, it is also slower, so I'd like if the user could at runtime select whether they want to use that library or my own fast and simple implementation. Unfortunately I'm stuck at a point where I don't understand some of the workings of Python's module system.
Suppose that my main program was main.py, that the (optional) external module is in module_a.py and that my own fast and simple implementation of module_a together with the actual program code that uses either my own implementation or the one of module_a is in the file module_x.py:
main.py:
import module_x
module_x.test(True)
module_x.test(False)
module_a.py:
class myclass():
def __init__(self):
print("i'm myclass in module_a")
module_x.py:
class myclass():
def __init__(self):
print("i'm myclass in module_x")
def test(enable_a):
if enable_a:
try:
from module_a import myclass
except ImportError:
global myclass
enable_a = False
else:
global myclass
i = myclass()
When I now execute main.py I get:
$ python3 main.py
i'm myclass in module_a
i'm myclass in module_a
But why is this? If False is passed to test() then the import of the module_a implementation should never happen. Instead it should only see myclass from the local file. Why doesn't it? How do I make test() use the local definition of myclass conditionally?
My solution is supposed to run in Python3 but I see the same effect when I use Python2.7.
An import statement is permanent within the thread of execution unless it is explicitly undone. Furthermore, once the from ... import statement is executed in this case, it replaces the variable myclass in the global scope (at which point the class it was previously referencing defined in the same file is no longer referenced and can in theory be garbage collected)
So what is happening here is whenever you run test(True) the first time, your myclass in module_x is effectively deleted and replaced with the myclass from module_a. All subsequent calls to test(False) then call global myclass which is effectively a no-op since the global myclass now refers to the one imported from the other class (and besides the global call is unneeded when not changing the global variable from a local scope as explained here).
To work around this, I would strongly suggest encapsulating the desired module-switching behavior in a class that is independent of either module you would like to switch. You can then charge that class with holding a reference to both modules and providing the rest of you client code with the correct one. E.g.
module_a_wrapper.py
import module_x
import module_a
class ModuleAWrapper(object):
_target_module = module_x # the default
#classmethod
def get_module(cls):
return cls._target_module
def set_module(enable_a):
if enable_a:
ModuleAWrapper._target_module = module_a
else:
ModuleAWrapper._target_module = module_x
def get_module():
return ModuleAWrapper.get_module()
main.py:
from module_a_wrapper import set_module, get_module
set_module(True)
get_module().myclass()
set_module(False)
get_module().myclass()
Running:
python main.py
# Outputs:
i'm myclass in module_a
i'm myclass in module_x
You can read more about the guts of the python import system here
The answer by lemonhead properly explains why this effect happens and gives a valid solution.
The general rule seems to be: wherever and however you import a module, it will always replace any variables of the same name from the global scope.
Funnily, when I use the import foo as bar construct, then there must neither be a global variable named foo nor one named bar!
So while lemonhead's solution worked it adds lots of complexity and will lead to my code being much longer because every time I want to get something from either module I have to prefix that call with the getter function.
This solution allows me to solve the problem with a minimal amount of changed code:
module_x.py:
class myclass_fast():
def __init__(self):
print("i'm myclass in module_x")
def test(enable_a):
if enable_a:
try:
from module_a import myclass
except ImportError:
enable_a = False
myclass = myclass_fast
else:
myclass = myclass_fast
i = myclass()
So the only thing I changed was to rename the class I had in global scope from myclass to myclass_fast. This way it will not be overwritten anymore by the import of myclass from module_a. Then, on demand, I change the local variable myclass to either be the imported module or myclass_fast.
Is it possible to set a global variable in the main module from an imported module (using a string) in python?
(Yes I know this isn't best practice.)
Ultimately I want it to look something like this:
main.py
import mod
def func():
print('failure')
mod.run(func)
func()
mod.py
def func2():
print('success')
def run(f):
globals()[f.__name__] = func2
The result is 'failure' because global is relative to the module.
I'm wanting to overwrite the variable func with func2, from the module.
Another caveat: the variable func changes, so I need to refer to it be the f.__name__ string
I'm aware that this approach wouldn't work if the name of func were changed before it's changed via mod.run(func).
My question: Is it possible to change the function in the main module from an imported module, without changing the code in the above example main.py? If so, how?
Rather than change what the name func is bound to, you can change the code that function func is bound to actually executes.
def func2():
print('success')
def run(f):
f.__code__ = func2.__code__
This modifies the actual function object referenced by func, and only partially using the code above, so further surgery may be needed (e.g. to update func.__name__ to be func2 instead of func), and you may want to make a real copy of func before monkey patching it like this.
CODE
class MyClass(object):
def MyMethod(self):
print(self)
MyObject = MyClass()
print(MyObject.MyMethod())
OUTPUT
<__main__.MyClass object at 0x0000000002B70E10 >
What does this __main__ mean? And what is being passed in the self parameter?
What does this __main__ mean?
The script invoked directly is considered the __main__ module. It can be imported and accessed the same way as any other module.
And what is being passed in the self parameter?
The reference contained in MyObject.
__main__ is the name of the current module if you run directly from the commandline. If you would import the module from another module import my_module, it will known by this name. Accordingly, the print would say:
< my_module.MyClass object at 0x0000000002B70E10 >
__main__ is the name of the module that your script is running in. You didn't define a library, so it's not a module with a name.
As for self, it's the equivalent of this in C++ or Java, but Python requires you to name it explicitly. See a tutorial.
First:
__main__ indicates that the class running the method is the primary file being run - the file that you clicked on or that you typed into terminal is the one where the class presides. That's why it's good practice to write
if __name__ == "__main__":
#do stuff
on your test code - this guarantees that your test code will only run if the code is being run from the originally called file. It's also why you should never write top level code, especially if you want to multi-thread later on!
Self is a key word that identifies the class. Every method needs to have the first parameter"self" - watch out, if you don't, an error will NOT be thrown, your code will just fail. Calling self.variable indicates a look-up to a class variable as opposed to a local variable (just within that method) or a global variable (avaliable to everyone). Similarly, calling self.methodName() calls a method belonging to that class.
So:
class Foo: #a new class, foo
def __init__( self ):
#set an object variable to be accessed anywhere in this object
self.myVariable = 'happy'
#this one's a local variable, it will dissapear at the end of the method's execution
myVaraible = sad
#this calls the newMethod method of the class, defined below. Note that we do NOT write self when calling.
self.newMethod( "this is my variable!" )
def newMethod (self, myVariable):
#prints the variable you passed in as a parameter
print myVariable
#prints 'happy', because that's what the object variable is, as defined above
print self.myVariable
I resolved it.You can use list.
For example →
print(list(your object))