Python Module Update - python

I have a main file that imports a class from another file as such:
from pybrain.rl.environments.HoldemTask import HoldemTask.
When I change HoldemTask.py, the changes are not reflected in the main file. The only workaround I have found is to run Pybrain's
python setup.py install
Can I reload the module or something? Reload() doesn't seem to work.

First off: python setup.py install generally makes a copy of the code it is installing, so if you're finding that you need to run that before changes take effect, chances are that for development you should be adjusting your PYTHONPATH or sys.path so that your relevant imports come directly from the source tree rather than from the Python site-packages library. You can quickly check which file your code is importing by putting this on the top of the main file when you run it:
from pybrain.rl.environments import HoldemTask # module object, not class
print(HoldemTask.__file__)
Secondly, in general it is far better to restart a Python process when making code changes to ensure that they come into effect. If you really need to get changes to show up without a restart, read on.
Reloading a module in Python only affects future imports. For a reload to work in-process, you have to replace the imported class object after the reload. For example, in the context of the "main file" performing the import you listed (inside a class method or function is fine):
# we need a module object to reload(), not the class inside it
from import pybrain.rl.environments import HoldemTask as HoldemTask_module
reload(HoldemTask_module)
# we then need to replace the old class object with the reloaded one
# in the main file's module-wide (aka "global") namespace
global HoldemTask
HoldemTask = HoldemTask_module.HoldemTask
One final caveat here is that any existing HoldemTask objects will continue to use the old code, because they embed in themselves a reference to the pre-reload class object. The only way for an in-process reload to be complete is if the code is specifically written to throw away every instance of anything it made based on the original module.

Related

Does a package 'see' itself in __init__.py?

I have a flask app with the root folder called project_folder.
A code snippet from the __init__.py file of this project_folder package:
#jwt.token_in_blacklist_loader
def check_if_token_in_blacklist(decrypted_token):
jti = decrypted_token['jti']
return project_folder.Model.RevokedTokenModel.is_jti_blacklisted(jti)
from project_folder.Controller.root import root
from project_folder.Controller import auth_controller
from project_folder.Controller import item_controller
Now the interesting thing is, that the project_folder package naturally has other smaller packages itself, which I'm importing to use them (for REST resources in this example). These are the last 3 lines, nothing throws an error so far.
But, if you take a look at the annotated function (in this example it always runs before some kind of JWT Token is being used), I am returning some inner package's function. Now when the logic truly runs this part the code breaks:
PROJECT_ROUTE\project_folder\__init__.py", line 38, in check_if_token_in_blacklist
return project_folder.Model.RevokedTokenModel.is_jti_blacklisted(jti)
NameError: name 'project_folder' is not defined
After thinking about it, it seems understandable. Importing from project_folder does import from the __init__.py file of the package, which is the actual file the interpreter currently is. So removing the package name prefix form the
return project_folder.Model.RevokedTokenModel.is_jti_blacklisted(jti)
to
return Model.RevokedTokenModel.is_jti_blacklisted(jti)
does not throw an error anymore.
The question is: Why is it only a problem inside the callback function and not with the last 3 imports?
This has to do with circular imports in python. Circular import is a form of circular dependency, created at the module import level.
How it works:
When you launch your application, python keeps a register (a kind of table) in which it records all the imported modules. When you call somewhere in your code a module, python will see in its registry if it has already been registered and loads it from there. You can access this registry via sys.module, which is actually a dictionary containing all the modules that have been imported since Python was started.
Example of use:
>>> import sys
>>> print('\n'.join(sys.modules.keys()))
So, since Python is an interpreted language, reading and execution of code is done line by line from top to bottom.
In your code, you put your imports at the bottom of your __init__.py file.
While browsing it, when python arrives at the line return project_folder.Model.RevokedTokenModel.is_jti_blacklisted(jti), it will look if the module exists in its register. Which is clearly not yet the case. That's why he raises an NameError: name 'project_folder' is not defined exception.

Add a module when running a python program from terminal

I wrote a module that, if it is imported, automatically changes the error output of my program. It is quite handy to have it in almost any python code I write.
Thus I don't want to add the line import my_errorhook to every code I write but want to have this line added automatically.
I found this answer, stating that it should be avoided to change the behavior of python directly. So I thought about changing the command line, something like
python --importModule my_errorhook main.py
and defining an alias in the bashrc to overwrite the python command to automatically add the parameter. Is there any way I could achieve such a behavior?
There is no such thing like --importModule in python command line. The only way you can incept the code without explicitly importing is by putting your functions in builtins module. However, this is a practice that is discouraged because it makes your code hard to maintain without proper design.
Let's assume that your python file main.py is the entry point of the whole program. Now you can create another file bootstrap.py, and put below codes into the new file.
import main
__builtins__.func = lambda x: x>=0
main.main()
Then the function func() can be called from all modules without being imported. For example in main.py
def main():
...
print(func(1))
...

Python: force every import to reload

Is there a way to force import x to always reload x in Python (i.e., as if I had called reload(x), or imp.reload(x) for Python 3)? Or in general, is there some way to force some code to be run every time I run import x? I'm OK with monkey patching or hackery.
I've tried moving the code into a separate module and deleting x from sys.modules in that separate file. I dabbled a bit with import hooks, but I didn't try too hard because according to the documentation, they are only called after the sys.modules cache is checked. I also tried monkeypatching sys.modules with a custom dict subclass, but whenever I do that, from module import submodule raises KeyError (I'm guessing sys.modules is not a real dictionary).
Basically, I'm trying to write a debugging tool (which is why some hackery is OK here). My goal is simply that import x is shorter to type than import x;x.y.
If you really want to change the semantics of the import statement, you will have to patch the interpreter. import checks whether the named module already is loaded and if so it does nothing more. You would have to change exactly that, and that is hard-wired in the interpreter.
Maybe you can live with patching the Python sources to use myImport('modulename') instead of import modulename? That would make it possible within Python itself.
Taking a lead from Alfe's answer, I got it to work like this. This goes at the module level.
def custom_logic():
# Put whatever you want to run when the module is imported here
# This version is run on the first import
custom_logic()
def __myimport__(name, *args, **kwargs):
if name == 'x': # Replace with the name of this module
# This version is run on all subsequent imports
custom_logic()
return __origimport__(name, *args, **kwargs)
# Will only be run on first import
__builtins__['__origimport__'] = __import__
__builtins__['__import__'] = __myimport__
We are monkeypatching __builtins__, which is why __origimport__ is defined when __myimport__ is run.

reload/reimport a file/class that is imported using from * import *

well, as the title say, i got a group of import, all import a class, all in the same folder as the script running it:
from lvl import lvl
from get import get
from image import image
from video import vid
from video import MLStripper
from system import system
from setting import setting
from listsearch import lists
python3 doesn't have reload iirc but there is imp.reload() but it doesn't seem to work,
it just throw a error saying it not a module (it a class so it doesn't work)
after every little edit in those class that are imported, i would need to restart the script
isn't there a way to reload/reimport the class to show the effect of the edit without needing to start the script or rewriting most of the script so that imp.reload() works?
python3, linux (but prefer if it also work on window)
edit1:
example: if i use:
import system
system.system.temp()
it return:
65°C
if i change it to show °F and reload it using imp.reload
imp.reload(system)
system.system.temp()
it return:
149°F
so, it works BUT if i use
import system as _system
from system import system
system.temp()
it return:
65°C
then i change it to show °F and reload it using imp.reload
imp.reload(_system)
from system import system
system.temp()
it still return
65°C
BUT again, if i call it this way:
_system.system.temp()
it return
149°F
idk why it that but it is cause it happen in a while loop?
edit2:
file name: system.py:
before changing for test:
class system:
def temp():
temperature = open("/sys/class/thermal/thermal_zone0/temp","r").read()
temperature = temperature[:2]
return(temperature+"°C")
after changing for test:
class system:
def temp():
temperature = open("/sys/class/thermal/thermal_zone0/temp","r").read()
temperature = temperature[:2]
temperature = str(9.0 / 5.0 * int(temperature) + 32).split(".")[0]
return(temperature+"°C")
You can only reload a module:
The argument must be a module object, so it must have been successfully imported before.
In your case, you don't have any reference to the module object. You will have to import it, even if you don't want to use it for anything else, just for the sake of calling reload later.
Also, after the reload, you have re-import the names:
Other references to the old objects (such as names external to the module) are not rebound to refer to the new objects and must be updated in each namespace where they occur if that is desired.
When you do from foo import bar, that bar is a "name external to the module", so you have to rebind it explicitly.
If you think about it, it has to work this way. There's no way reload can enumerate all objects whose definition is dependent on the previous version of the module to update them. Even if it could, there could be infinite cycles. And what would happen if the new version didn't even have a definition for a class that was in the old one? Or if the class were defined dynamically?
Looking at it another way, from foo import bar is very similar to import foo; bar = foo.bar. bar is a name in your namespace, not foo's namespace, so reload(foo) will not touch it; you need to copy the new foo.bar over again.
The easy way to solve all of these problems is to just repeat all of your from foo import bar lines after the reload.
For simple cases:
import video
from video import MLStripper
# ... later
imp.reload(video)
from video import MLStripper
However, most of your examples have an obvious naming conflict: once you from video import video, you can no longer reload(video). So, you need another reference to the video module object.
Python keeps one around for you, which you can use:
imp.reload(sys.modules['video'])
from video import MLStripper
Or, alternatively, you can use an as clause, or just an = assignment, to give it whatever name you want.
import video as _video
from video import video
# ... later
imp.reload(_video)
from video import video
From your comments and your edited question, it sounds like you have a further problem. Let's use one of the simple non-colliding cases to discuss it.
I believe you're actually doing something like this:
import video
from video import MLStripper
stripper = MLStripper('foo")
# ... later
imp.reload(video)
from video import MLStripper
The first line will successfully reload the video module, and the second will copy its MLStripper class into your globals, so any new MLStripper instances you created will be of the new type.
But that doesn't affect any existing MLStripper instances, like stripper.
Just like MLStripper was, stripper is one of those "names external to the module". But it's actually even worse. In order to adjust it, reload would have to figure out what its state would have been, had the new version of the code been in effect from the time it was created. It should be obvious that this is an unsolvable problem.
If you know the instances you want to patch up, you can deal with them in effectively the same way you dealt with the classes: just create them again:
imp.reload(video)
from video import MLStripper
stripper = MLStripper('foo")
If that's not good enough, there are three hacky possibilities that may be what you want:
Monkeypatch the methods, attributes, etc. into the instance(s) and their __class__(es).
Patch the instances' __class__ attribute directly, so anything that was inherited from the class will now be inherited from the new class.
Serialize the instances with pickle before the reload, then deserialize after.
For very simple cases, all three of these will work. For more complex cases, you will have to understand what you're doing.
Note that you can wrap a lot of this stuff up in a function, but you have to understand how locals and globals work (and how import and reload work) or you're going to end up confusing yourself.
A simpler solution is to just create "dump all state" and "load all state" functions. Then you can dump everything, quit, relaunch, and restore. The Python tutorial and the ipython docs both describe a few different ways to do this in place of using reload; it's probably worth going back and rereading those.
Access the module through sys.modules, reload that, then reassign the imported names:
imp.reload(sys.modules['lvl'])
from lvl import lvl
imp.reload(sys.modules['get'])
from get import get
etc.
All the from something import name syntax does is import something then bind name to the same object something.name refers to.
By using sys.modules you don't have to explicitly import the module again, and can reach the new definitions of the objects for rebinding after reloading.

Linking Python Files Assistance

I understand how to actually link python files, however, i don't understand how to get variable's from these linked files. I've tried to grab them and I keep getting NameError.
How do I go about doing this? The reason i want to link files is to simply neaten up my script and not make it 10000000000 lines long. Also, in the imported python script, do i have to import everything again? Another question, do i use the self function when using another scripts functions?
ie;
Main Script:
import sys, os
import importedpyfile
Imported Py File
import sys, os
I understand how to actually link python files, however, i don't
understand how to get variable's from these linked files. I've tried
to grab them and I keep getting NameError.
How are you doing that? Post more code. For instance, the following works:
file1.py
#!/usr/bin/env python
from file2 import file2_func, file2_variable
file2_func()
print file2_variable
file2.py:
#!/usr/bin/env python
file2_variable = "I'm a variable!"
def file2_func():
print "Hello World!"
Also, in the imported python script, do i have to import everything
again?
Nope, modules should be imported when the python interpreter reads that file.
Another question, do i use the self function when using another
scripts functions?
Nope, that's usually to access class members. See python self explained.
There is also more than one way to import files. See the other answer for some explanations.
I think what you are trying to ask is how to get access to global vars from on .py file without having to deal with namespaces.
In your main script, replace the call to import importedpyfile to say this instead
from importedpyfile import *
Ideally, you keep the code the way you have it. But instead, just reference those global vars with the importedpyfile namespace.
e.g.
import importedpyfile
importedpyfile.MyFunction() # calls "MyFunction" that is defined in importedpyfile.py
Python modules are not "linked" in the sense of C/C++ linking libraries into an executable. A Python import operation creates a name that refers to the imported module; without this name there is no (direct) way to access another module.

Categories