Let's face it, the whole business of reloading python code after changing it is a mess. I figured out awhile back that calling import <module> at the interpreter is better than from <module> import <class/function>, because then I can call reload(module) to get updated code.
But I have more complicated issues now. So I have this file, module1.py, and at the top it says:
from module2 import <class1>, <function1>, etc.
And then I go and change code inside module2. Turns out that calling reload(module1) will not reload the code changed in module2, even though code from module2 is imported at the top of module1. Is there any way to reload everything without restarting the interpreter?
Before anyone gets on my case about style, I'll just say that:
I only call reload from the interpreter, never in active code. This question concerns when I'm testing new code.
I never call from <module> import *, I know that destroys readability
Have a look into IPython. It has the autoreload extension that automatically reloads modules during the interpreter session before calling functions within. I cite the example from the landing page:
In [1]: %load_ext autoreload
In [2]: %autoreload 2
In [3]: from foo import some_function
In [4]: some_function()
Out[4]: 42
In [5]: # open foo.py in an editor and change some_function to return 43
In [6]: some_function()
Out[6]: 43
To reload a module, you have to use reload, and you have to use it on the module you want to reload. Reloading a module doesn't recursively reload all modules imported by that module. It just reloads that one module.
When a module is imported, a reference to it is stored, and later imports of that module re-use the already-imported, already-stored version. When you reload module1, it re-runs the from module2 import ... statement, but that just reuses the already-imported version of module2 without reloading it.
The only way to fix this is to change your code so it does import module2 instead of (or in addition to) from module2 import .... You cannot reload a module unless the module itself has been imported and bound to a name (i.e., with an import module statement, not just a from module import stuff statement).
Note that you can use both forms of the import, and reloading the imported module will affect subsequent from imports. That is, you can do this:
>>> import module
>>> from module import x
>>> x
2
# Change module code here, changing x to 3
>>> reload(module)
>>> from module import x
>>> x
3
This can be handy for interactive work, since it lets you use short, unprefixed names to refer to what you need, while still being able to reload the module.
Rather than getting better at reloading modules, you could get better at restarting the interpreter. For example, you can put your setup code into its own file, and then run it like this:
$ python -i setup.py
>>>
This will run setup.py, then leave you at the interactive prompt. Or, rather than doing a lot of work in the interactive prompt, write automated tests that do your work for you.
You are right, reloading modules in Python is a mess. The semantics of the language make it difficult to change code while the process is running. Learn not to need reloading modules, you'll be happier.
Ok, I'm not sure that qualifies as an answer without a change to the code, but... at least, that doesn't involve a change to module1.
You can use some module wrapper class, that saves loaded modules before and after loading module1 and that provides a reload method, something like that:
import sys
class Reloader(object):
def __init__(self, modulename):
before = sys.modules.keys()
__import__(modulename)
after = sys.modules.keys()
names = list(set(after) - set(before))
self._toreload = [sys.modules[name] for name in names]
def do_reload(self):
for i in self._toreload:
reload(i)
Then load module1 with:
reloader = Reloader('module1')
Atfer that you can modify module2 and reload it in interpreter with:
reloader.do_reload()
Don't forget that an import is really just assigning a name in a namespace. So, you could reassign that name after reloading:
>>> reload(module2)
>>> module1.class1 = module2.class1
Now the class1 object inside module1 refers to the reloaded version from module2.
Here is a recursive reload function that you could use (credit to #Matthew):
https://stackoverflow.com/a/17194836/1020625
Related
I have a Jupyter Notebook - we'll call it main, and I'm running it piecemeal in Jupyter Lab 3.0.14 using a Python 3 environment.
Within this notebook, I'm trying to import custom functions from a module, we'll call this one module.py, which is in the same directory as main.
It looks something like this:
//main
import sys
import os
from module import foo, bar
//module.py
def foo():
return
def bar():
return
When I run the import statements in main, it throws an error
ImportError: cannot import name 'bar' from 'module' (/module.py)
If I run
from module import foo
or
from module import *
It's fine (although calling the functions will throw an error that the function is not defined). In fact, it will import foo even if there's no function named foo in module.py. It won't import any of my own functions by name. I feel like I must be missing something fundamental, here...
Any help would be greatly appreciated!!
Okay, this is my first time using Jupyter Lab, and this problem came from a fundamental misunderstanding of how Jupyter (and maybe other IDEs?) manages packages.
In short, as I understand it, re-running the code which imported functions from module.py did not actually re-import the contents. The only things it imported were what I asked for the first time, and any subsequent calls to import functions only referenced that first static image I pulled.
Therefore, if module.py had one function, foo, and I ran:
from module import foo
...then made some changes, such as editing the contents of foo and adding bar, running:
from module import foo, bar
...didn't actually do anything but reference that first copy of module.py I pulled. Because bar didn't exist the first time I ran the import function, it "couldn't see it", no matter how many times I re-ran the import function. I had to restart the kernel to fix it, basically.
I noticed this because when I changed the name of the script, it would import the functions that weren't previously working, but then edits to the functions, even once imported, weren't recognized.
This link provided the clue I needed to figure this out.
Hopefully this helps somebody!
Restart the kernel in the Jupiter notebook. I solved a similar import name issue by restarting the kernel.
Is there a way to force import x to always reload x in Python (i.e., as if I had called reload(x), or imp.reload(x) for Python 3)? Or in general, is there some way to force some code to be run every time I run import x? I'm OK with monkey patching or hackery.
I've tried moving the code into a separate module and deleting x from sys.modules in that separate file. I dabbled a bit with import hooks, but I didn't try too hard because according to the documentation, they are only called after the sys.modules cache is checked. I also tried monkeypatching sys.modules with a custom dict subclass, but whenever I do that, from module import submodule raises KeyError (I'm guessing sys.modules is not a real dictionary).
Basically, I'm trying to write a debugging tool (which is why some hackery is OK here). My goal is simply that import x is shorter to type than import x;x.y.
If you really want to change the semantics of the import statement, you will have to patch the interpreter. import checks whether the named module already is loaded and if so it does nothing more. You would have to change exactly that, and that is hard-wired in the interpreter.
Maybe you can live with patching the Python sources to use myImport('modulename') instead of import modulename? That would make it possible within Python itself.
Taking a lead from Alfe's answer, I got it to work like this. This goes at the module level.
def custom_logic():
# Put whatever you want to run when the module is imported here
# This version is run on the first import
custom_logic()
def __myimport__(name, *args, **kwargs):
if name == 'x': # Replace with the name of this module
# This version is run on all subsequent imports
custom_logic()
return __origimport__(name, *args, **kwargs)
# Will only be run on first import
__builtins__['__origimport__'] = __import__
__builtins__['__import__'] = __myimport__
We are monkeypatching __builtins__, which is why __origimport__ is defined when __myimport__ is run.
Demo:
from module1 import func1
from module1 import func2
Ques:
Is the module1 executed twice?
Is this the good and correct way to import?
Thx!
In SO2.py:
print 'SO2 is running'
a = 'cabbage'
b = 'python-is-awesome'
In SO.py:
from SO2 import a
from SO2 import b
When running the latter:
SO2 is running
Only once ;).
Per PEP 8, it's okay to do:
from SO2 import a, b
Modules are cached in the import system, and thus are only loaded once unless they are explicitly reloaded. When python tries to import a module it first tries to locate it in sys.modules, which contains the references to all previously loaded Modules. This is the same for packages as well. For a more in-depth explanation, I'd recommend reading the excellent documentation of the import machinery.
More generally, Python only loads a module once, the first time that it is needed--whether it's an "import form" or plain import statement that does the "needing". Any subsequent imports will just note that the module has already been loaded.
The module initialization is only done once, on load.
If you need to refresh the contents of a module, you can reload it. On Python 2, there's a reload(m) built-in function to reload module m. In Python 3 this has been moved to the imp module, and imp.reload(m). In Python 3.4, imp is being deprecated in favor of a new module (as of 3.1) named "importlib". The initialization will get run again when a module is reloaded.
See:
http://docs.python.org/2.7/library/functions.html#reload
http://docs.python.org/3.3/library/imp.html
http://docs.python.org/3.3/library/importlib.html#module-importlib
As far as "best practices" go, I'm not always "pythonic". I agree with the PEP 8 position on one module per import line, but would prefer that modules be also listed alphabetically.
I don't especially like "import from" statements. If module.func1 is going to be used a lot in some particular function in the current source, I prefer adding:
def myfunc(*a, **k):
func1 = module.func1
... etc.
...so it's clear that in that function only that func1 means module.func1.
When writing a Python module, is there a way to tell if the module is being imported or reloaded?
I know I can create a class, and the __init__() will only be called on the first import, but I hadn't planning on creating a class. Though, I will if there isn't an easy way to tell if we are being imported or reloaded.
The documentation for reload() actually gives a code snippet that I think should work for your purposes, at least in the usual case. You'd do something like this:
try:
reloading
except NameError:
reloading = False # means the module is being imported
else:
reloading = True # means the module is being reloaded
What this really does is detect whether the module is being imported "cleanly" (e.g. for the first time) or is overwriting a previous instance of the same module. In the normal case, a "clean" import corresponds to the import statement, and a "dirty" import corresponds to reload(), because import only really imports the module once, the first time it's executed (for each given module).
If you somehow manage to force a subsequent execution of the import statement into doing something nontrivial, or if you somehow manage to import your module for the first time using reload(), or if you mess around with the importing mechanism (through the imp module or the like), all bets are off. In other words, don't count on this always working in every possible situation.
P.S. The fact that you're asking this question makes me wonder if you're doing something you probably shouldn't be doing, but I won't ask.
>>> import os
>>> os.foo = 5
>>> os.foo
5
>>> import os
>>> os.foo
5
I saw in this useful Q&A that one can use reload(whatever_module) or, in Python 3, imp.reload(whatever_module).
My question is, what if I had said from whatever_module import * to import? Then I have no whatever_module to refer to when I use reload(). Are you guys gonna yell at me for throwing a whole module into the global namespace? :)
I agree with the "don't do this generally" consensus, but...
The correct answer is:
from importlib import reload
import X
reload(X)
from X import Y # or * for that matter
Never use import *; it destroys readability.
Also, be aware that reloading modules is almost never useful. You can't predict what state your program will end up in after reloading a module, so it's a great way to get incomprehensible, unreproduceable bugs.
A cleaner answer is a mix of Catskul's good answer and Ohad Cohen's use of sys.modules and direct redefinition:
import sys
Y = reload(sys.modules["X"]).Y # reload() returns the new module
In fact, doing import X creates a new symbol (X) that might be redefined in the code that follows, which is unnecessary (whereas sys is a common module, so this should not happen).
The interesting point here is that from X import Y does not add X to the namespace, but adds module X to the list of known modules (sys.modules), which allows the module to be reloaded (and its new contents accessed).
More generally, if multiple imported symbols need to be updated, it is then more convenient to import them like this:
import sys
reload(sys.modules["X"]) # No X symbol created!
from X import Y, Z, T
A
from module import *
takes all “exported” objects from module and binds them to module-level (or whatever-your-scope-was-level) names. You can reload the module as:
reload(sys.modules['module'])
but that won't do you any good: the whatever-your-scope-was-level names still point at the old objects.
I've found another way to deal with reloading a module when importing like:
from directory.module import my_func
It's nice to know how do modules are being imported generally.
The module is searched in sys.modules dictionary. If it already exists in sys.modules - the module will not be imported again.
So if we would like to reload our module, we can just remove it from sys.modules and import again:
import sys
from directory.module import my_func
my_func('spam')
# output: 'spam'
# here I have edited my_func in module.py
my_func('spam') # same result as above
#output: 'spam'
del sys.modules[my_func.__module__]
from directory.module import my_func
my_func('spam') # new result
#output: 'spam spam spam spam spam'
If You would like to get reloaded module when running whole script, you could use exception handler:
try:
del sys.modules[my_func.__module__]
except NameError as e:
print("""Can't remove module that haven't been imported.
Error: {}""".format(e))
from utils.module import my_func
..........
# code of the script here
When importing using from whatever_module import whatever, whatever is counted as part of the importing module, so to reload it - you should reload your module. But just reloading your module you will still get the old whatever - from the already-imported whatever_module, so you need to reload(whatever_module), and than reload your module:
# reload(whatever_module), if you imported it
reload(sys.modules['whatever_module'])
reload(sys.modules[__name__])
if you used from whatever_module import whatever you can also consider
whatever=reload(sys.modules['whatever_module']).whatever
or
whatever=reload(whatever_module).whatever
import re
for mod in sys.modules.values():
if re.search('name', str(mod)):
reload(mod)
for python 3.7 :
from importlib import reload #import function "reload"
import YourModule #import your any modules
reload(YourModule) #reload your module
Reload function can be called from your own function
def yourFunc():
reload(YourModule)