EDIT: Turns out the problem has to do with the path.
If I cd into the directory containing the library and run python __init__.py the imports all work fine. It's if I'm in a different directory and try to import the library itself (i.e. in the parent directory, and trying to import) that the failure occurs.
I don't see any way to literally specify a path for an import statement.
So, I'm wondering if the best way is just to add the directory in scriptDir to the sys.path? Is this the best way to do it? I feel like there should be a more elegant method, but...
I want to write a library that I will be able to extend very easily.
Here's some skeleton/pseudo code for what I want to do. In reality this code is a lot more complex, but it follows the basic premise - import each file, check it, and determine if we should use it; then allocate it into a list of module references. All of this would be contained in a single library folder.
I want the library, when imported, to dynamically import any file found in its directory starting with "plugin_". See the code:
init.py:
import os.path
scriptDir = os.path.dirname(__file__)
mods = []
thisMod = 0
for file in os.listdir(scriptDir):
if (file[0:7] == "plugin_" and file[-3:] == ".py"):
thisMod = __import__(".".join(file.split(".")[0:-1]))
print "debug: imported %s" % thisMod.modName
if (thisMod.enable == True):
mods.append(thisMod)
else:
print "debug: not loading %s because it's disabled." % thisMod.modName
def listMods():
"This function should print the names of all loaded modules."
for m in mods:
print "debug: module %s" % m.modName
def runMods():
"This function should execute the run method for ALL modules."
for m in mods:
c = m.ModuleClass()
c.run()
def selectMod(modNum):
"This function should let us select a single module for the runSelectedMod function."
thisMod = mods[modNum]
def runSelectedMod():
"This function should run the 'run' method for ONLY the previously selected module."
if (thisMod == 0):
raise ArgumentError("you didn't assign a module yet.")
c = thisMod.ModuleClass()
c.run()
plugin_test1.py
modName = "test module 1"
enable = True
class ModuleClass:
def run(self):
print "test module 1 is running"
plugin_math.py
modName = "math module"
enable = True
class ModuleClass:
def run(self):
print "mathematical result: %d" % (1+1)
plugin_bad.py
modName = "bad module"
enable = False
class ModuleClass:
def __init__(self):
print "x"[4] # throws IndexError, this code should not run.
def run(self):
print "divide by zero: %f" % (5 / 0) # this code should not run.
The problem I've already found is import won't work since I'm not importing whole libraries, but rather individual files. I'm guessing there is either an alternate syntax to import for this purpose? For example, import plugin_test and from plugin_math import ModuleClass work but my use of import isn't. I get an error:
thisMod = __import__(".".join(file.split(".")[0:-1]))
ImportError: No module named plugin_test1
Now, another question is: How will this end up working if I use py2exe, py2app, etc. to compile this into a compact library? If I recall, don't these apps compress all the local libraries into a site_packages.zip file?...
I'm still learning how to do this type of advanced coding in Python, so any advice is appreciated.
Thanks!
I was able to run it in Python 3. The only change not regarding syntax is the scriptDir:
import os.path
scriptDir = os.path.dirname(os.path.abspath(__file__))
mods = []
thisMod = 0
for file in os.listdir(scriptDir):
if (file[0:7] == "plugin_" and file[-3:] == ".py"):
thisMod = __import__(".".join(file.split(".")[0:-1]))
print ("debug: imported %s" % thisMod.modName)
if (thisMod.enable == True):
mods.append(thisMod)
else:
print ("debug: not loading %s because it's disabled." % thisMod.modName)
def listMods():
"This function should print the names of all loaded modules."
for m in mods:
print ("debug: module %s" % m.modName)
def runMods():
"This function should execute the run method for ALL modules."
for m in mods:
c = m.ModuleClass()
c.run()
def selectMod(modNum):
"This function should let us select a single module for the runSelectedMod function."
thisMod = mods[modNum]
def runSelectedMod():
"This function should run the 'run' method for ONLY the previously selected module."
if (thisMod == 0):
raise ArgumentError("you didn't assign a module yet.")
c = thisMod.ModuleClass()
c.run()
gives:
C:\Users\mm\Desktop\ING.SW\python>python so\dina.py
debug: imported bad module
debug: not loading bad module because it's disabled.
debug: imported math module
debug: imported test module 1
I guess you have a different problem, since the file name is actually read (as it's printed in the error message), but still...
Related
I am trying to dynamically import modules in python.
I need something like a plugin import system.
I use the following code for import of a module, and it works fine, as long as the entire code of the module is in the same file.
caller.py code:
PluginFolder = "./plugins"
MainModule = "__init__"
possibleplugins = os.listdir(PluginFolder)
for possible_plugin in possibleplugins:
location = os.path.abspath(os.path.join(PluginFolder, possible_plugin))
if not os.path.isdir(location) or not MainModule + ".py" in os.listdir(location):
continue
info = imp.find_module(MainModule, [location])
plugin = {"name": possible_plugin, "info": info}
After that, I use the load_module method to load the module:
module = imp.load_module(MainModule, *plugin["info"])
The structure of the directories is as follows:
Project/
--plugins/
----plugin_1/
-------__init__.py
-------X.py
caller.py
Everything works great when all the methods are in the same file (__init__.py).
When the method I use in __init__.py calls another method from another file (in the same package) than an error saying "No module named X"
the __init__.py code fails at the line:
import X
I also tried
import plugin_1.X
and other variations.
Just to make sure you guys understand- importing and using the module in a normal way (not dynamically) works fine.
What am I missing?
Is there another way to do this? Maybe use the __import__ method, or something else?
I usually use importlib. It's load_module method does the job. You can put the importing code into plugins/__init__.py to do the following:
import importlib
import os
import logging
skip = set(("__init__.py",))
plugins = []
cwd = os.getcwd()
os.chdir(os.path.dirname(__file__))
for mod in glob.glob("*.py"):
if mod in skip:
continue
try:
mod = mod.replace(".py", "")
plugin = importlib.import_module("." + mod, __name__)
plugin.append(plugin)
except Exception as e:
logging.warn("Failed to load %s: %s. Skipping", mod, e)
os.chdir(cwd)
How can I know if a Python module exists, without importing it?
Importing something that might not exist (not what I want) results in:
try:
import eggs
except ImportError:
pass
TL;DR) Use importlib.util.find_spec(module_name) (Python 3.4+).
Python2: imp.find_module
To check if import can find something in Python 2, using imp:
import imp
try:
imp.find_module('eggs')
found = True
except ImportError:
found = False
To find dotted imports, you need to do more:
import imp
try:
spam_info = imp.find_module('spam')
spam = imp.load_module('spam', *spam_info)
imp.find_module('eggs', spam.__path__) # __path__ is already a list
found = True
except ImportError:
found = False
You can also use pkgutil.find_loader (more or less the same as the Python 3 part:
import pkgutil
eggs_loader = pkgutil.find_loader('eggs')
found = eggs_loader is not None
Python 3
Python 3 ≤ 3.3: importlib.find_loader
You should use importlib. I went about doing this like:
import importlib
spam_loader = importlib.find_loader('spam')
found = spam_loader is not None
My expectation being, if you can find a loader for it, then it exists. You can also be a bit more smart about it, like filtering out what loaders you will accept. For example:
import importlib
spam_loader = importlib.find_loader('spam')
# only accept it as valid if there is a source file for the module - no bytecode only.
found = issubclass(type(spam_loader), importlib.machinery.SourceFileLoader)
Python 3 ≥ 3.4: importlib.util.find_spec
In Python 3.4 importlib.find_loader Python documentation was deprecated in favour of importlib.util.find_spec. The recommended method is the importlib.util.find_spec. There are others like importlib.machinery.FileFinder, which is useful if you're after a specific file to load. Figuring out how to use them is beyond the scope of this.
import importlib
spam_spec = importlib.util.find_spec("spam")
found = spam_spec is not None
This also works with relative imports, but you must supply the starting package, so you could also do:
import importlib
spam_spec = importlib.util.find_spec("..spam", package="eggs.bar")
found = spam_spec is not None
spam_spec.name == "eggs.spam"
While I'm sure there exists a reason for doing this - I'm not sure what it would be.
Warning
When trying to find a submodule, it will import the parent module (for ALL of the above methods)!
food/
|- __init__.py
|- eggs.py
## __init__.py
print("module food loaded")
## eggs.py
print("module eggs")
were you then to run
>>> import importlib
>>> spam_spec = importlib.util.find_spec("food.eggs")
module food loaded
ModuleSpec(name='food.eggs', loader=<_frozen_importlib.SourceFileLoader object at 0x10221df28>, origin='/home/user/food/eggs.py')
Comments are welcome on getting around this
Acknowledgements
#rvighne for importlib
#lucas-guido for Python 3.3+ deprecating find_loader
#enpenax for pkgutils.find_loader behaviour in Python 2.7
Python 3 >= 3.6: ModuleNotFoundError
The ModuleNotFoundError has been introduced in Python 3.6 and can be used for this purpose:
try:
import eggs
except ModuleNotFoundError:
# Error handling
pass
The error is raised when a module or one of its parents cannot be found. So
try:
import eggs.sub
except ModuleNotFoundError as err:
# Error handling
print(err)
would print a message that looks like No module named 'eggs' if the eggs module cannot be found; but it would print something like No module named 'eggs.sub' if only the sub module couldn't be found, but the eggs package could be found.
See the documentation of the import system for more information on the ModuleNotFoundError.
After using yarbelk's response, I've made this so I don't have to import ìmp.
try:
__import__('imp').find_module('eggs')
# Make things with a supposed existing module
except ImportError:
pass
It is useful in Django's settings.py file, for example.
Python 2, without relying on ImportError
Until the current answer is updated, here is the way for Python 2
import pkgutil
import importlib
if pkgutil.find_loader(mod) is not None:
return importlib.import_module(mod)
return None
Why another answer?
A lot of answers make use of catching an ImportError. The problem with that is, that we cannot know what throws the ImportError.
If you import your existent module and there happens to be an ImportError in your module (e.g., typo on line 1), the result will be that your module does not exist.
It will take you quite the amount of backtracking to figure out that your module exists and the ImportError is caught and makes things fail silently.
go_as's answer as a one-liner:
python -c "help('modules');" | grep module
Use one of the functions from pkgutil, for example:
from pkgutil import iter_modules
def module_exists(module_name):
return module_name in (name for loader, name, ispkg in iter_modules())
I wrote this helper function:
def is_module_available(module_name):
if sys.version_info < (3, 0):
# python 2
import importlib
torch_loader = importlib.find_loader(module_name)
elif sys.version_info <= (3, 3):
# python 3.0 to 3.3
import pkgutil
torch_loader = pkgutil.find_loader(module_name)
elif sys.version_info >= (3, 4):
# python 3.4 and above
import importlib
torch_loader = importlib.util.find_spec(module_name)
return torch_loader is not None
Here is a way to check if a module is loaded from the command line:
Linux/UNIX script file method: make a file module_help.py:
#!/usr/bin/env python
help('modules')
Then make sure it's executable: chmod u+x module_help.py
And call it with a pipe to grep:
./module_help.py | grep module_name
Invoke the built-in help system. (This function is intended for interactive use.) If no argument is given, the interactive help system starts on the interpreter console. If the argument is a string, then the string is looked up as the name of a module, function, class, method, keyword, or documentation topic, and a help page is printed on the console. If the argument is any other kind of object, a help page on the object is generated.
Interactive method: in the console, load python
>>> help('module_name')
If found, quit reading by typing q.
To exit the Python interpreter interactive session, press Ctrl + D
Windows script file method, also Linux/UNIX compatible, and better overall:
#!/usr/bin/env python
import sys
help(sys.argv[1])
Calling it from the command like:
python module_help.py site
Would output:
Help on module site:
NAME
site - Append module search paths for third-party packages to sys.path.
FILE
/usr/lib/python2.7/site.py
MODULE DOCS
http://docs.python.org/library/site
DESCRIPTION
...
:
And you'd have to press q to exit interactive mode.
Using it for an unknown module, e.g.,
python module_help.py lkajshdflkahsodf
Would output:
no Python documentation found for 'lkajshdflkahsodf'
and exit.
You could just write a little script that would try to import all the modules and tell you which ones are failing and which ones are working:
import pip
if __name__ == '__main__':
for package in pip.get_installed_distributions():
pack_string = str(package).split(" ")[0]
try:
if __import__(pack_string.lower()):
print(pack_string + " loaded successfully")
except Exception as e:
print(pack_string + " failed with error code: {}".format(e))
Output:
zope.interface loaded successfully
zope.deprecation loaded successfully
yarg loaded successfully
xlrd loaded successfully
WMI loaded successfully
Werkzeug loaded successfully
WebOb loaded successfully
virtualenv loaded successfully
...
A word of warning: this will try to import everything, so you'll see things like PyYAML failed with error code: No module named pyyaml, because the actual import name is just yaml. So as long as you know your imports, this should do the trick for you.
There isn't any way to reliably check if "dotted module" is importable without importing its parent package. Saying this, there are many solutions to problem "how to check if a Python module exists".
The below solution addresses the problem that an imported module can raise an ImportError even if it exists. We want to distinguish that situation from such in which the module does not exist.
Python 2:
import importlib
import pkgutil
import sys
def find_module(full_module_name):
"""
Returns module object if module `full_module_name` can be imported.
Returns None if module does not exist.
Exception is raised if (existing) module raises exception during its import.
"""
module = sys.modules.get(full_module_name)
if module is None:
module_path_tail = full_module_name.split('.')
module_path_head = []
loader = True
while module_path_tail and loader:
module_path_head.append(module_path_tail.pop(0))
module_name = ".".join(module_path_head)
loader = bool(pkgutil.find_loader(module_name))
if not loader:
# Double check if module realy does not exist
# (case: full_module_name == 'paste.deploy')
try:
importlib.import_module(module_name)
except ImportError:
pass
else:
loader = True
if loader:
module = importlib.import_module(full_module_name)
return module
Python 3:
import importlib
def find_module(full_module_name):
"""
Returns module object if module `full_module_name` can be imported.
Returns None if module does not exist.
Exception is raised if (existing) module raises exception during its import.
"""
try:
return importlib.import_module(full_module_name)
except ImportError as exc:
if not (full_module_name + '.').startswith(exc.name + '.'):
raise
In django.utils.module_loading.module_has_submodule:
import sys
import os
import imp
def module_has_submodule(package, module_name):
"""
check module in package
django.utils.module_loading.module_has_submodule
"""
name = ".".join([package.__name__, module_name])
try:
# None indicates a cached miss; see mark_miss() in Python/import.c.
return sys.modules[name] is not None
except KeyError:
pass
try:
package_path = package.__path__ # No __path__, then not a package.
except AttributeError:
# Since the remainder of this function assumes that we're dealing with
# a package (module with a __path__), so if it's not, then bail here.
return False
for finder in sys.meta_path:
if finder.find_module(name, package_path):
return True
for entry in package_path:
try:
# Try the cached finder.
finder = sys.path_importer_cache[entry]
if finder is None:
# Implicit import machinery should be used.
try:
file_, _, _ = imp.find_module(module_name, [entry])
if file_:
file_.close()
return True
except ImportError:
continue
# Else see if the finder knows of a loader.
elif finder.find_module(name):
return True
else:
continue
except KeyError:
# No cached finder, so try and make one.
for hook in sys.path_hooks:
try:
finder = hook(entry)
# XXX Could cache in sys.path_importer_cache
if finder.find_module(name):
return True
else:
# Once a finder is found, stop the search.
break
except ImportError:
# Continue the search for a finder.
continue
else:
# No finder found.
# Try the implicit import machinery if searching a directory.
if os.path.isdir(entry):
try:
file_, _, _ = imp.find_module(module_name, [entry])
if file_:
file_.close()
return True
except ImportError:
pass
# XXX Could insert None or NullImporter
else:
# Exhausted the search, so the module cannot be found.
return False
In case you know the location of file and want to check that the respective Python code file has that module or not, you can simply check via the astor package in Python. Here is a quick example:
"""
Check if a module function exists or not without importing a Python package file
"""
import ast
import astor
tree = astor.parse_file('handler.py')
method_to_check = 'handle'
for item in tree.body:
if isinstance(item, ast.FunctionDef):
if item.name == method_to_check:
print('method exists')
break
A simpler if statement from Ask Ubuntu, How do I check whether a module is installed in Python?:
import sys
print('eggs' in sys.modules)
You can also use importlib.util directly
import importlib.util
def module_exists_without_import(module_name):
spec = importlib.util.find_spec(module_name)
return spec is not None
I would like to provide users with a python script which would import some modules for them and then drop to the python interpreter with the imported modules available at that level. I have some code which I thought might work, but it does not seem to:
module_list = ['module_a','module_b']
# Import our common modules
for module in module_list:
try:
print "Importing: {0}".format(module)
exec("import {0}".format(module))
except:
print "FYI we failed importing {0}. It will not be available for you to use".format(module)
So when the script is done it will drop to the python module where the user can do:
>>> module_a.run()
You can use code.InteractiveConsole() and pass a locals dictionary containing the local context in which the console will run. By storing the modules in there you can easily make them available to the interactive shell.
I wouldn't do this, if I were you.. but
import imp
module_list = ['module_a','module_b']
for name in module_list:
try:
module_info = imp.find_module(name)
globals()[name] = imp.load_module(name, *module_info)
except:
"Import error: %s" %name
save above code file, for example imptest.py and load it with -i switch, as
python -i imptest.py
Against the advise of many comments I have read on touching __main__, I did it.
So here is the code that ended up working for me:
module_list = ['module_a','module_b']
# Import our modules
for name in module_list:
try:
__import__(name)
if name in sys.modules:
setattr(__main__, name, sys.modules[name])
except:
print "FYI we failed importing {0}. It will not be available for you to use".format(name)
I have an application that reads test scripts in python and sends them across the network for execution on a remote python instance. As the controlling program does not need to run these scripts I do not want to have all the modules the test scripts use installed on the controller's python environment. However the controller does need information from the test script to tell it how to run the test.
Currently what I do to read and import test script data is something like
with open( 'test.py', 'r' ) as f:
source = f.read()
m = types.ModuleType( "imported-temp", "Test module" )
co = compile( source, 'test.py', 'exec' )
exec co in m.__dict__
which yields a new module that contains the test. Unfortunately exec will raise ImportErrors if the test tries to import something the controller does not have. And worse, the module will not be fully imported.
If I can guarantee that the controller will not use the missing modules, is there someway I can ignore these exceptions? Or some other way to find out the names and classes defined in the test?
Examples test:
from controller import testUnit
import somethingThatTheControllerDoesNotHave
_testAttr = ['fast','foo','function']
class PartOne( testUnit ):
def run( self ):
pass
What the controller needs to know is the data in _testAttr and the name of all class definitions inheriting from testUnit.
Write an import hook that catches the exception and returns a dummy module if the module doesn't exist.
import __builtin__
from types import ModuleType
class DummyModule(ModuleType):
def __getattr__(self, key):
return None
__all__ = [] # support wildcard imports
def tryimport(name, globals={}, locals={}, fromlist=[], level=-1):
try:
return realimport(name, globals, locals, fromlist, level)
except ImportError:
return DummyModule(name)
realimport, __builtin__.__import__ = __builtin__.__import__, tryimport
import sys # works as usual
import foo # no error
from bar import baz # also no error
from quux import * # ditto
You could also write it to always return a dummy module, or to return a dummy module if the specified module hasn't already been loaded (hint: if it's in sys.modules, it has already been loaded).
I think, based on what you're saying, that you can just say:
try:
exec co in m.__dict__
except ImportError: pass
Does that help?
You could use python ast module, and parse the script to an AST tree and then scan through the tree looking for elements of interest. That way you don't have to execute the script at all.
File "G:\Python25\Lib\site-packages\PyAMF-0.6b2-py2.5-win32.egg\pyamf\util\__init__.py", line 15, in <module>
ImportError: cannot import name python
How do I fix it?
If you need any info to know how to fix this problem, I can explain, just ask.
Thanks
Code:
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext import webapp
from TottysGateway import TottysGateway
import logging
def main():
services_root = 'services'
#services = ['users.login']
#gateway = TottysGateway(services, services_root, logger=logging, debug=True)
#app = webapp.WSGIApplication([('/', gateway)], debug=True)
#run_wsgi_app(app)
if __name__ == "__main__":
main()
Code:
from pyamf.remoting.gateway.google import WebAppGateway
import logging
class TottysGateway(WebAppGateway):
def __init__(self, services_available, root_path, not_found_service, logger, debug):
# override the contructor and then call the super
self.services_available = services_available
self.root_path = root_path
self.not_found_service = not_found_service
WebAppGateway.__init__(self, {}, logger=logging, debug=True)
def getServiceRequest(self, request, target):
# override the original getServiceRequest method
try:
# try looking for the service in the services list
return WebAppGateway.getServiceRequest(self, request, target)
except:
pass
try:
# don't know what it does but is an error for now
service_func = self.router(target)
except:
if(target in self.services_available):
# only if is an available service import it's module
# so it doesn't access services that should be hidden
try:
module_path = self.root_path + '.' + target
paths = target.rsplit('.')
func_name = paths[len(paths) - 1]
import_as = '_'.join(paths) + '_' + func_name
import_string = "from "+module_path+" import "+func_name+' as service_func'
exec import_string
except:
service_func = False
if(not service_func):
# if is not found load the default not found service
module_path = self.rootPath + '.' + self.not_found_service
import_string = "from "+module_path+" import "+func_name+' as service_func'
# add the service loaded above
assign_string = "self.addService(service_func, target)"
exec assign_string
return WebAppGateway.getServiceRequest(self, request, target)
You need to post your full traceback. What you show here isn't all that useful. I ended up digging up line 15 of pyamf/util/init.py. The code you should have posted is
from pyamf import python
This should not fail unless your local environment is messed up.
Can you 'import pyamf.util' and 'import pyamf.python' in a interactive Python shell? What about if you start Python while in /tmp (on the assumption that you might have a file named 'pyamf.py' in the current directory. Which is a bad thing.)
= (older comment below) =
Fix your question. I can't even tell where line 15 of util/__init__.py is supposed to be. Since I can't figure that out, I can't answer your question. Instead, I'll point out ways to improve your question and code.
First, use the markup language correctly, so that all the code is in a code block. Make sure you've titled the code, so we know it's from util/__init__.py and not some random file.
In your error message, include the full traceback, and not the last two lines.
Stop using parens in things like "if(not service_func):" and use a space instead, so its " if not service_func:". This is discussed in PEP 8.
Read the Python documentation and learn how to use the language. Something like "func_name = paths[len(paths) - 1]" should be "func_name = paths[-1]"
Learn about the import function and don't use "exec" for this case. Nor do you need the "exec assign_string" -- just do the "self.addService(service_func, target)"