I saw in this useful Q&A that one can use reload(whatever_module) or, in Python 3, imp.reload(whatever_module).
My question is, what if I had said from whatever_module import * to import? Then I have no whatever_module to refer to when I use reload(). Are you guys gonna yell at me for throwing a whole module into the global namespace? :)
I agree with the "don't do this generally" consensus, but...
The correct answer is:
from importlib import reload
import X
reload(X)
from X import Y # or * for that matter
Never use import *; it destroys readability.
Also, be aware that reloading modules is almost never useful. You can't predict what state your program will end up in after reloading a module, so it's a great way to get incomprehensible, unreproduceable bugs.
A cleaner answer is a mix of Catskul's good answer and Ohad Cohen's use of sys.modules and direct redefinition:
import sys
Y = reload(sys.modules["X"]).Y # reload() returns the new module
In fact, doing import X creates a new symbol (X) that might be redefined in the code that follows, which is unnecessary (whereas sys is a common module, so this should not happen).
The interesting point here is that from X import Y does not add X to the namespace, but adds module X to the list of known modules (sys.modules), which allows the module to be reloaded (and its new contents accessed).
More generally, if multiple imported symbols need to be updated, it is then more convenient to import them like this:
import sys
reload(sys.modules["X"]) # No X symbol created!
from X import Y, Z, T
A
from module import *
takes all “exported” objects from module and binds them to module-level (or whatever-your-scope-was-level) names. You can reload the module as:
reload(sys.modules['module'])
but that won't do you any good: the whatever-your-scope-was-level names still point at the old objects.
I've found another way to deal with reloading a module when importing like:
from directory.module import my_func
It's nice to know how do modules are being imported generally.
The module is searched in sys.modules dictionary. If it already exists in sys.modules - the module will not be imported again.
So if we would like to reload our module, we can just remove it from sys.modules and import again:
import sys
from directory.module import my_func
my_func('spam')
# output: 'spam'
# here I have edited my_func in module.py
my_func('spam') # same result as above
#output: 'spam'
del sys.modules[my_func.__module__]
from directory.module import my_func
my_func('spam') # new result
#output: 'spam spam spam spam spam'
If You would like to get reloaded module when running whole script, you could use exception handler:
try:
del sys.modules[my_func.__module__]
except NameError as e:
print("""Can't remove module that haven't been imported.
Error: {}""".format(e))
from utils.module import my_func
..........
# code of the script here
When importing using from whatever_module import whatever, whatever is counted as part of the importing module, so to reload it - you should reload your module. But just reloading your module you will still get the old whatever - from the already-imported whatever_module, so you need to reload(whatever_module), and than reload your module:
# reload(whatever_module), if you imported it
reload(sys.modules['whatever_module'])
reload(sys.modules[__name__])
if you used from whatever_module import whatever you can also consider
whatever=reload(sys.modules['whatever_module']).whatever
or
whatever=reload(whatever_module).whatever
import re
for mod in sys.modules.values():
if re.search('name', str(mod)):
reload(mod)
for python 3.7 :
from importlib import reload #import function "reload"
import YourModule #import your any modules
reload(YourModule) #reload your module
Reload function can be called from your own function
def yourFunc():
reload(YourModule)
Related
I am importing a module in python as this:
from myutils.user_data import *
How do I find out what are the list of methods that I have imported?
I know one workaround from here:
How to list all functions in a Python module?
being:
from inspect import getmembers, isfunction
import myutils.user_data as module_name
functions_list = getmembers(module_name, isfunction)
print(functions_list)
but this would oblige me to use the nomenclature:
module_name.mehtodA() whereas I would like to be able to use the methods as such methodA()
of course I can do:
from myutils.user_data import *
import myutils.user_data as module_name
but this is actually importing two times.
Any idea?
EDIT: Why do I need this? I am creating documentation for a module in a JupyterHub environment (in premises). I create this documentation using notebooks, i.e. anyone interested in finding out the use of a particular .py file (including utility methods) can open the notebook and play around AND the jupyter notebook can be rendered as a web site with voila. In that case I would like to print all the methods included in the particular .py file.
This is also a question that just made me curious. Someone commented you would never import with * a module. Well, why not if you know what you are importing being a few very small methods.
Generally you are rarely recommended to use the from ... import * style, because it could override local symbols or symbols imported first by other modules.
That beeing said, you could do
symbols_before = dir()
from myutils.user_data import *
symbols_after = dir()
imported_symbols = [s for s in symbols_after if not s in symbols_before]
which stores the freshly imported symbols in the imported_symbols list.
Or you could use the fact, that the module is still loaded into sys.modules and do
import sys
from inspect import getmembers, isfunction
from myutils.user_data import *
functions_list = getmembers(sys.modules['myutils.user_data'], isfunction)
print(functions_list)
Let's say I have a file where I'm importing some packages:
# myfile.py
import os
import re
import pathlib
def func(x, y):
print(x, y)
If I go into another file and enter
from myfile import *
Not only does it import func, but it also imports os, re, and pathlib,
but I DO NOT want those modules to be imported when I do import *.
Why is it importing the other packages I'm importing and how do you avoid this?
The reason
Because import imports every name in the namespace. If something has a name inside the module, then it's valid to be exported.
How to avoid
First of all, you should almost never be using import *. It's almost always clearer code to either import the specific methods/variables you're trying to use (from module import func), or to import the whole module and access methods/variables via dot notation (import module; ...; module.func()).
That said, if you must use import * from module, there are a few ways to prevent certain names from being exported from module:
Names starting with _ will not be imported by import * from .... They can still be imported directly (i.e. from module import _name), but not automatically. This means you can rename your imports so that they don't get exported, e.g. import os as _os. However, this also means that your entire code in that module has to refer to the _os instead of os, so you may have to modify lots of code.
If a module contains the name __all__: List[str], then import * will export only the names contained in that list. In your example, add the line __all__ = ['func'] to your myfile.py, and then import * will only import func. See also this answer.
from myfile import func
Here is the fix :)
When you import *, you import everything from. Which includes what yu imported in the file your source.
It has actually been discussed on Medium, but for simplification, I will answer it myself.
from <module/package> import * is a way to import all the names we can get in that specific module/package. Usually, everyone doesn't actually use import * for this reason, and rather sticked with import <module>.
Python's import essentially just runs the file you point it to import (it's not quite that but close enough). So if you import a module it will also import all the things the module imports. If you want to import only specific functions within the module, try:
from myfile import func
...which would import only myfile.func() instead of the other things as well.
Let's face it, the whole business of reloading python code after changing it is a mess. I figured out awhile back that calling import <module> at the interpreter is better than from <module> import <class/function>, because then I can call reload(module) to get updated code.
But I have more complicated issues now. So I have this file, module1.py, and at the top it says:
from module2 import <class1>, <function1>, etc.
And then I go and change code inside module2. Turns out that calling reload(module1) will not reload the code changed in module2, even though code from module2 is imported at the top of module1. Is there any way to reload everything without restarting the interpreter?
Before anyone gets on my case about style, I'll just say that:
I only call reload from the interpreter, never in active code. This question concerns when I'm testing new code.
I never call from <module> import *, I know that destroys readability
Have a look into IPython. It has the autoreload extension that automatically reloads modules during the interpreter session before calling functions within. I cite the example from the landing page:
In [1]: %load_ext autoreload
In [2]: %autoreload 2
In [3]: from foo import some_function
In [4]: some_function()
Out[4]: 42
In [5]: # open foo.py in an editor and change some_function to return 43
In [6]: some_function()
Out[6]: 43
To reload a module, you have to use reload, and you have to use it on the module you want to reload. Reloading a module doesn't recursively reload all modules imported by that module. It just reloads that one module.
When a module is imported, a reference to it is stored, and later imports of that module re-use the already-imported, already-stored version. When you reload module1, it re-runs the from module2 import ... statement, but that just reuses the already-imported version of module2 without reloading it.
The only way to fix this is to change your code so it does import module2 instead of (or in addition to) from module2 import .... You cannot reload a module unless the module itself has been imported and bound to a name (i.e., with an import module statement, not just a from module import stuff statement).
Note that you can use both forms of the import, and reloading the imported module will affect subsequent from imports. That is, you can do this:
>>> import module
>>> from module import x
>>> x
2
# Change module code here, changing x to 3
>>> reload(module)
>>> from module import x
>>> x
3
This can be handy for interactive work, since it lets you use short, unprefixed names to refer to what you need, while still being able to reload the module.
Rather than getting better at reloading modules, you could get better at restarting the interpreter. For example, you can put your setup code into its own file, and then run it like this:
$ python -i setup.py
>>>
This will run setup.py, then leave you at the interactive prompt. Or, rather than doing a lot of work in the interactive prompt, write automated tests that do your work for you.
You are right, reloading modules in Python is a mess. The semantics of the language make it difficult to change code while the process is running. Learn not to need reloading modules, you'll be happier.
Ok, I'm not sure that qualifies as an answer without a change to the code, but... at least, that doesn't involve a change to module1.
You can use some module wrapper class, that saves loaded modules before and after loading module1 and that provides a reload method, something like that:
import sys
class Reloader(object):
def __init__(self, modulename):
before = sys.modules.keys()
__import__(modulename)
after = sys.modules.keys()
names = list(set(after) - set(before))
self._toreload = [sys.modules[name] for name in names]
def do_reload(self):
for i in self._toreload:
reload(i)
Then load module1 with:
reloader = Reloader('module1')
Atfer that you can modify module2 and reload it in interpreter with:
reloader.do_reload()
Don't forget that an import is really just assigning a name in a namespace. So, you could reassign that name after reloading:
>>> reload(module2)
>>> module1.class1 = module2.class1
Now the class1 object inside module1 refers to the reloaded version from module2.
Here is a recursive reload function that you could use (credit to #Matthew):
https://stackoverflow.com/a/17194836/1020625
I've got some python code in a library that attempts to load a simple value from a module that will exist for the applications that use this library
from somemodule import simplevalue
Normally, the application that uses the library will have the module file and everything works fine. However, in the unit tests for this library the module does not exist. I know that I can create a temporary file and add that file to my path at runtime, but I was curious if there is a way in python to load something in to memory that would allow the above import to work.
This is more of a curiosity, saying "add the module to your test path" is not helpful :P
It is. Use types.ModuleType to create a new module object, then add it to sys.modules:
sys.modules["somename"] = types.ModuleType("somename")
You can then do import somename. If you need to add classes or functions to it, import it before calling your test script, and just add functions to it:
def myfunc(x, y, z):
...
somename.myfunc = myfunc
It should go without saying, but just in case: this is largely an academic curiosity. It has some uses for testing, but other than that, stick to importing things the usual way.
Incidentally, how I know about this: I've come across the technique used in testing, to make a "stub" for the _winreg module on non-Windows systems. Here it is in use.
It isn't necessary to create a module. No Python code cares whether somemodule.simplevalue is actually a reference to an attribute of a module. To do so, a program would need to check the type of somemodule. Why bother?
Since you just want the single value from the module and are importing it into your own namespace, just define it:
simplevalue = 42
If you like, use try/except to try to import the real module first.
try:
from somemodule import simplevalue
except ImportError:
simplevalue = 42
If you are importing the entire module but only using one value, you can use a class to define a namespace.
try:
import somemodule
except ImportError:
class somemodule(object):
simplevalue = 42
Now somemodule.simplevalue refers to the value regardless of whether the module is available.
If you want other modules that also import somemodule to see your faked-up class-as-module, as you would in your unit test, just do this afterward:
import sys
sys.modules["somemodule"] = somemodule
Your system under test (sut in my example) needs to be able to cope with the fact that somemodule may not exist, so you can trap the ImportError:
#!/usr/bin/env python
try:
from somemodule import simplevalue
except ImportError, e:
if 'somemodule' in e:
'''We expect that to happen in the unittest but you should log something for when
this happens in production'''
def fn():
return simplevalue
Then you can inject a value in your unittest:
#!/usr/bin/env python
import unittest
import sut
class T(unittest.TestCase):
def test_fn(self):
sut.simplevalue = 42
self.assertEquals(42, sut.fn())
if __name__ == '__main__':
unittest.main()
Specifically, I need to get at some objects and globals from the main module in an imported module. I know how to find those things when the parent module wants some particular thing from a child module, but I can't figure out how to go in the other direction.
import __main__
But don't do this.
The answer you're looking for is:
import __main__
main_global1= __main__.global1
However, whenever a module module1 needs stuff from the __main__ module, then:
either the __main__ module should provide all necessary data as parameters to a module1 function/class,
or you should put everything that needs to be shared in another module, and import it as import module2 in both __main__ and module1.
I think this would work:
import sys
main_mod = sys.modules['__main__']
Not sure if it is a good practice but maybe you could pass the objects and variables you need as parameters to the methods or classes you call in the imported module.