I am importing a module in python as this:
from myutils.user_data import *
How do I find out what are the list of methods that I have imported?
I know one workaround from here:
How to list all functions in a Python module?
being:
from inspect import getmembers, isfunction
import myutils.user_data as module_name
functions_list = getmembers(module_name, isfunction)
print(functions_list)
but this would oblige me to use the nomenclature:
module_name.mehtodA() whereas I would like to be able to use the methods as such methodA()
of course I can do:
from myutils.user_data import *
import myutils.user_data as module_name
but this is actually importing two times.
Any idea?
EDIT: Why do I need this? I am creating documentation for a module in a JupyterHub environment (in premises). I create this documentation using notebooks, i.e. anyone interested in finding out the use of a particular .py file (including utility methods) can open the notebook and play around AND the jupyter notebook can be rendered as a web site with voila. In that case I would like to print all the methods included in the particular .py file.
This is also a question that just made me curious. Someone commented you would never import with * a module. Well, why not if you know what you are importing being a few very small methods.
Generally you are rarely recommended to use the from ... import * style, because it could override local symbols or symbols imported first by other modules.
That beeing said, you could do
symbols_before = dir()
from myutils.user_data import *
symbols_after = dir()
imported_symbols = [s for s in symbols_after if not s in symbols_before]
which stores the freshly imported symbols in the imported_symbols list.
Or you could use the fact, that the module is still loaded into sys.modules and do
import sys
from inspect import getmembers, isfunction
from myutils.user_data import *
functions_list = getmembers(sys.modules['myutils.user_data'], isfunction)
print(functions_list)
Let's say I have a file where I'm importing some packages:
# myfile.py
import os
import re
import pathlib
def func(x, y):
print(x, y)
If I go into another file and enter
from myfile import *
Not only does it import func, but it also imports os, re, and pathlib,
but I DO NOT want those modules to be imported when I do import *.
Why is it importing the other packages I'm importing and how do you avoid this?
The reason
Because import imports every name in the namespace. If something has a name inside the module, then it's valid to be exported.
How to avoid
First of all, you should almost never be using import *. It's almost always clearer code to either import the specific methods/variables you're trying to use (from module import func), or to import the whole module and access methods/variables via dot notation (import module; ...; module.func()).
That said, if you must use import * from module, there are a few ways to prevent certain names from being exported from module:
Names starting with _ will not be imported by import * from .... They can still be imported directly (i.e. from module import _name), but not automatically. This means you can rename your imports so that they don't get exported, e.g. import os as _os. However, this also means that your entire code in that module has to refer to the _os instead of os, so you may have to modify lots of code.
If a module contains the name __all__: List[str], then import * will export only the names contained in that list. In your example, add the line __all__ = ['func'] to your myfile.py, and then import * will only import func. See also this answer.
from myfile import func
Here is the fix :)
When you import *, you import everything from. Which includes what yu imported in the file your source.
It has actually been discussed on Medium, but for simplification, I will answer it myself.
from <module/package> import * is a way to import all the names we can get in that specific module/package. Usually, everyone doesn't actually use import * for this reason, and rather sticked with import <module>.
Python's import essentially just runs the file you point it to import (it's not quite that but close enough). So if you import a module it will also import all the things the module imports. If you want to import only specific functions within the module, try:
from myfile import func
...which would import only myfile.func() instead of the other things as well.
I have to create a Python file on the fly, using Python code (functions, classes, imports). This is the code that I currently have
def write_code_to_file(code_list, file_path):
with open(file_path, 'w') as ofh:
for c in code_list:
fh = io.StringIO(inspect.getsource(c))
ofh.writelines(fh.readlines())
ofh.write('\n\n')
write_code_to_file([myFunc, myClass],'/some/path')
1) Is there a better way of doing this?
2) How do I do import statements? I can pass a string directly i.e. "from lib import util" but is there a better way of doing it? I can't assign an import statement to a variable but how could I do something like.
import_stmt1 = ...something which represents "from lib import util" ....
write_code_to_file([import_stmt1, myFunc], '/tmp/myfile.py')
So, when I open /tmp/myfile.py, it should be:
from lib import util
def myFunc(arg1, arg2..)
...
Actually, I am starting to understand what you're trying to do. I thought you were trying to import a module you built on the fly, you are trying to see how to import.
Thing is that importing does somethings to the module in memory. If you do something like:
import myMod
myMod.__dict__
You will see how import loads all the functions of the imported modules into the myMod dictionary. This is probably how you would do it in code. You may want to look at creating your own module dictionaries, aka using metaprogramming (http://python-3-patterns-idioms-test.readthedocs.io/en/latest/Metaprogramming.html).
I saw in this useful Q&A that one can use reload(whatever_module) or, in Python 3, imp.reload(whatever_module).
My question is, what if I had said from whatever_module import * to import? Then I have no whatever_module to refer to when I use reload(). Are you guys gonna yell at me for throwing a whole module into the global namespace? :)
I agree with the "don't do this generally" consensus, but...
The correct answer is:
from importlib import reload
import X
reload(X)
from X import Y # or * for that matter
Never use import *; it destroys readability.
Also, be aware that reloading modules is almost never useful. You can't predict what state your program will end up in after reloading a module, so it's a great way to get incomprehensible, unreproduceable bugs.
A cleaner answer is a mix of Catskul's good answer and Ohad Cohen's use of sys.modules and direct redefinition:
import sys
Y = reload(sys.modules["X"]).Y # reload() returns the new module
In fact, doing import X creates a new symbol (X) that might be redefined in the code that follows, which is unnecessary (whereas sys is a common module, so this should not happen).
The interesting point here is that from X import Y does not add X to the namespace, but adds module X to the list of known modules (sys.modules), which allows the module to be reloaded (and its new contents accessed).
More generally, if multiple imported symbols need to be updated, it is then more convenient to import them like this:
import sys
reload(sys.modules["X"]) # No X symbol created!
from X import Y, Z, T
A
from module import *
takes all “exported” objects from module and binds them to module-level (or whatever-your-scope-was-level) names. You can reload the module as:
reload(sys.modules['module'])
but that won't do you any good: the whatever-your-scope-was-level names still point at the old objects.
I've found another way to deal with reloading a module when importing like:
from directory.module import my_func
It's nice to know how do modules are being imported generally.
The module is searched in sys.modules dictionary. If it already exists in sys.modules - the module will not be imported again.
So if we would like to reload our module, we can just remove it from sys.modules and import again:
import sys
from directory.module import my_func
my_func('spam')
# output: 'spam'
# here I have edited my_func in module.py
my_func('spam') # same result as above
#output: 'spam'
del sys.modules[my_func.__module__]
from directory.module import my_func
my_func('spam') # new result
#output: 'spam spam spam spam spam'
If You would like to get reloaded module when running whole script, you could use exception handler:
try:
del sys.modules[my_func.__module__]
except NameError as e:
print("""Can't remove module that haven't been imported.
Error: {}""".format(e))
from utils.module import my_func
..........
# code of the script here
When importing using from whatever_module import whatever, whatever is counted as part of the importing module, so to reload it - you should reload your module. But just reloading your module you will still get the old whatever - from the already-imported whatever_module, so you need to reload(whatever_module), and than reload your module:
# reload(whatever_module), if you imported it
reload(sys.modules['whatever_module'])
reload(sys.modules[__name__])
if you used from whatever_module import whatever you can also consider
whatever=reload(sys.modules['whatever_module']).whatever
or
whatever=reload(whatever_module).whatever
import re
for mod in sys.modules.values():
if re.search('name', str(mod)):
reload(mod)
for python 3.7 :
from importlib import reload #import function "reload"
import YourModule #import your any modules
reload(YourModule) #reload your module
Reload function can be called from your own function
def yourFunc():
reload(YourModule)
I am a little confused by the multitude of ways in which you can import modules in Python.
import X
import X as Y
from A import B
I have been reading up about scoping and namespaces, but I would like some practical advice on what is the best strategy, under which circumstances and why. Should imports happen at a module level or a method/function level? In the __init__.py or in the module code itself?
My question is not really answered by "Python packages - import by class, not file" although it is obviously related.
In production code in our company, we try to follow the following rules.
We place imports at the beginning of the file, right after the main file's docstring, e.g.:
"""
Registry related functionality.
"""
import wx
# ...
Now, if we import a class that is one of few in the imported module, we import the name directly, so that in the code we only have to use the last part, e.g.:
from RegistryController import RegistryController
from ui.windows.lists import ListCtrl, DynamicListCtrl
There are modules, however, that contain dozens of classes, e.g. list of all possible exceptions. Then we import the module itself and reference to it in the code:
from main.core import Exceptions
# ...
raise Exceptions.FileNotFound()
We use the import X as Y as rarely as possible, because it makes searching for usage of a particular module or class difficult. Sometimes, however, you have to use it if you wish to import two classes that have the same name, but exist in different modules, e.g.:
from Queue import Queue
from main.core.MessageQueue import Queue as MessageQueue
As a general rule, we don't do imports inside methods -- they simply make code slower and less readable. Some may find this a good way to easily resolve cyclic imports problem, but a better solution is code reorganization.
Let me just paste a part of conversation on django-dev mailing list started by Guido van Rossum:
[...]
For example, it's part of the Google Python style guides[1] that all
imports must import a module, not a class or function from that
module. There are way more classes and functions than there are
modules, so recalling where a particular thing comes from is much
easier if it is prefixed with a module name. Often multiple modules
happen to define things with the same name -- so a reader of the code
doesn't have to go back to the top of the file to see from which
module a given name is imported.
Source: http://groups.google.com/group/django-developers/browse_thread/thread/78975372cdfb7d1a
1: http://code.google.com/p/soc/wiki/PythonStyleGuide#Module_and_package_imports
I would normally use import X on module level. If you only need a single object from a module, use from X import Y.
Only use import X as Y in case you're otherwise confronted with a name clash.
I only use imports on function level to import stuff I need when the module is used as the main module, like:
def main():
import sys
if len(sys.argv) > 1:
pass
HTH
Someone above said that
from X import A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P
is equivalent to
import X
import X allows direct modifications to A-P, while from X import ... creates copies of A-P. For from X import A..P you do not get updates to variables if they are modified. If you modify them, you only modify your copy, but X does know about your modifications.
If A-P are functions, you won't know the difference.
Others have covered most of the ground here but I just wanted to add one case where I will use import X as Y (temporarily), when I'm trying out a new version of a class or module.
So if we were migrating to a new implementation of a module, but didn't want to cut the code base over all at one time, we might write a xyz_new module and do this in the source files that we had migrated:
import xyz_new as xyz
Then, once we cut over the entire code base, we'd just replace the xyz module with xyz_new and change all of the imports back to
import xyz
DON'T do this:
from X import *
unless you are absolutely sure that you will use each and every thing in that module. And even then, you should probably reconsider using a different approach.
Other than that, it's just a matter of style.
from X import Y
is good and saves you lots of typing. I tend to use that when I'm using something in it fairly frequently But if you're importing a lot from that module, you could end up with an import statement that looks like this:
from X import A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P
You get the idea. That's when imports like
import X
become useful. Either that or if I'm not really using anything in X very frequently.
I generally try to use the regular import modulename, unless the module name is long, or used often..
For example, I would do..
from BeautifulSoup import BeautifulStoneSoup as BSS
..so I can do soup = BSS(html) instead of BeautifulSoup.BeautifulStoneSoup(html)
Or..
from xmpp import XmppClientBase
..instead of importing the entire of xmpp when I only use the XmppClientBase
Using import x as y is handy if you want to import either very long method names , or to prevent clobbering an existing import/variable/class/method (something you should try to avoid completely, but it's not always possible)
Say I want to run a main() function from another script, but I already have a main() function..
from my_other_module import main as other_module_main
..wouldn't replace my main function with my_other_module's main
Oh, one thing - don't do from x import * - it makes your code very hard to understand, as you cannot easily see where a method came from (from x import *; from y import *; my_func() - where is my_func defined?)
In all cases, you could just do import modulename and then do modulename.subthing1.subthing2.method("test")...
The from x import y as z stuff is purely for convenience - use it whenever it'll make your code easier to read or write!
When you have a well-written library, which is sometimes case in python, you ought just import it and use it as it. Well-written library tends to take life and language of its own, resulting in pleasant-to-read -code, where you rarely reference the library. When a library is well-written, you ought not need renaming or anything else too often.
import gat
node = gat.Node()
child = node.children()
Sometimes it's not possible to write it this way, or then you want to lift down things from library you imported.
from gat import Node, SubNode
node = Node()
child = SubNode(node)
Sometimes you do this for lot of things, if your import string overflows 80 columns, It's good idea to do this:
from gat import (
Node, SubNode, TopNode, SuperNode, CoolNode,
PowerNode, UpNode
)
The best strategy is to keep all of these imports on the top of the file. Preferrably ordered alphabetically, import -statements first, then from import -statements.
Now I tell you why this is the best convention.
Python could perfectly have had an automatic import, which'd look from the main imports for the value when it can't be found from global namespace. But this is not a good idea. I explain shortly why. Aside it being more complicated to implement than simple import, programmers wouldn't be so much thinking about the depedencies and finding out from where you imported things ought be done some other way than just looking into imports.
Need to find out depedencies is one reason why people hate "from ... import *". Some bad examples where you need to do this exist though, for example opengl -wrappings.
So the import definitions are actually valuable as defining the depedencies of the program. It is the way how you should exploit them. From them you can quickly just check where some weird function is imported from.
The import X as Y is useful if you have different implementations of the same module/class.
With some nested try..import..except ImportError..imports you can hide the implementation from your code. See lxml etree import example:
try:
from lxml import etree
print("running with lxml.etree")
except ImportError:
try:
# Python 2.5
import xml.etree.cElementTree as etree
print("running with cElementTree on Python 2.5+")
except ImportError:
try:
# Python 2.5
import xml.etree.ElementTree as etree
print("running with ElementTree on Python 2.5+")
except ImportError:
try:
# normal cElementTree install
import cElementTree as etree
print("running with cElementTree")
except ImportError:
try:
# normal ElementTree install
import elementtree.ElementTree as etree
print("running with ElementTree")
except ImportError:
print("Failed to import ElementTree from any known place")
I'm with Jason in the fact of not using
from X import *
But in my case (i'm not an expert programmer, so my code does not meet the coding style too well) I usually do in my programs a file with all the constants like program version, authors, error messages and all that stuff, so the file are just definitions, then I make the import
from const import *
That saves me a lot of time. But it's the only file that has that import, and it's because all inside that file are just variable declarations.
Doing that kind of import in a file with classes and definitions might be useful, but when you have to read that code you spend lots of time locating functions and classes.