We Keep Coding

utility to check imports in python without running the code (python2.7)

Is there a way to make sure all imports listed in all of my python files are in PYTHONPATH. Basically a validator for all my imports in files.
My approach: Change all file path from / to "." e.g. foo/bar to foo.bar and then run:
pkgutil.find_loader("foo.bar")
Problem: It does not work if I have foo.bar.zoo.data as a module
export PYTHONPATH=$(pwd):"my_lib_path"
for root,dir,files in os.walk(work_dir):
for filename in files:
if not "__init__" in filename and filename.endswith(".py"):
print "Testing file {}".format(os.path.join(root,filename))
filepath = os.path.join(root,filename)
filepath = filepath.replace(work_dir,'',)
filepath = filepath.replace('/', '.')
filepath = filepath.lstrip(".").rstrip(".py")
print " testing filepath" , filepath
# tried this also
'''for imp, name, _ in pkgutil.iter_modules(root):
full_name = "{}.{}".format(root,name)
''' module = imp.find_module(full_name)
mod = pkgutil.find_loader(filepath)

The best way to do this is to use the importlib (Python 3.1+) or imp (Python 2.x) module to do all the steps an import would do up to, but not including, running the code.
The key functions for each Python version are:
3.4-3.8: importlib.util.find_spec
3.3-3.3: importlib.find_loader.
3.1-3.2: importlib.find_module
3.0-3.0: imp.find_module
1.5-2.7: imp.find_module
0.9-1.4: No idea. (There were no packages yet; everything was different…)
Advantages of doing it this way:
It works whether the module is a normal Python module, a .pyc-only module, a C extension module, a builtin, or some funky special type of module that you've installed a custom import hook for.
It works even if the module is inside a .egg, or in the frozen bootstrap collection, or if the whole library is wrapped up in a .zip or even buried inside a .exe.
It automatically takes care of the funky rules, like figuring out what is or isn't a valid namespace package extension directory (including dealing with things like site.py and old-style setuptools path-injection) that would be a huge pain to get right.
In 3.4+, this is literally the same code that import uses, because the entire import system is written in Python. For older versions, that's not true—but for 2.3+, it's guaranteed to get you the same thing you would get in an import hook, which is almost surely close enough.
Quoting the 3.7 docs:
importlib.util.find_spec(name, package=None)
Find the spec for a module, optionally relative to the specified package name. If the module is in sys.modules, then sys.modules[name].__spec__ is returned (unless the spec would be None or is not set, in which case ValueError is raised). Otherwise a search using sys.meta_path is done. None is returned if no spec is found.
If name is for a submodule (contains a dot), the parent module is automatically imported.
name and package work the same as for import_module().
(You don't really have to care what a spec is here; if Python can find the spec for a module, the module is present; if it returns None, the module is not present.)
So, this will just magically handle foo.bar.zoo.data.
If you look at the Examples, there's one that does exactly what you want, "Checking if a module can be imported".
def test_module(name):
if not importlib.util.find_spec(name):
raise ImportError(name)
From the 2.7 docs:
imp.find_module(name[, path])
Try to find the module name. If path is omitted or None, the list of directory names given by sys.path is searched, but first a few special places are searched: the function tries to find a built-in module with the given name (C_BUILTIN), then a frozen module (PY_FROZEN), and on some systems some other places are looked in as well (on Windows, it looks in the registry which may point to a specific file).
Otherwise, path must be a list of directory names; each directory is searched for files with any of the suffixes returned by get_suffixes() above. Invalid names in the list are silently ignored (but all list items must be strings).
If search is successful, the return value is a 3-element tuple (file, pathname, description):
file is an open file object positioned at the beginning, pathname is the pathname of the file found, and description is a 3-element tuple as contained in the list returned by get_suffixes() describing the kind of module found.
If the module does not live in a file, the returned file is None, pathname is the empty string, and the description tuple contains empty strings for its suffix and mode; the module type is indicated as given in parentheses above. If the search is unsuccessful, ImportError is raised. Other exceptions indicate problems with the arguments or environment.
If the module is a package, file is None, pathname is the package path and the last item in the description tuple is PKG_DIRECTORY.
This function does not handle hierarchical module names (names containing dots). In order to find P.M, that is, submodule M of package P, use find_module() and load_module() to find and load package P, and then use find_module() with the path argument set to P.__path__. When P itself has a dotted name, apply this recipe recursively.
As you can see, it's a little more complicated—it doesn't magically handle foo.bar.zoo.data; you will need to find foo, verify that it's a package, load it, find bar, etc., and finally find (but not load) data.
Something like this (untested):
def test_module(name):
parts = name.split('.')
path = None
for part in parts[:-1]:
file, pathname, description = imp.find_module(part, path)
if description[-1] != imp.PKG_DIRECTORY:
raise ImportError(name)
pkg = imp.load_module(part, file, pathname, description)
path = pkg.__path__
file, pathname, description = imp.find_module(part[-1], path)

Get path for every file python loads when importing modules

I'm trying to make a directory containing all the modules a python script depends on. Rather than manually tracking down all those files, I would like to automatically find these files as python imports them. To do this, I've added a module finder to sys.meta_path:
import sys, imp
class ImportPrint(object):
def find_module(self, name, path=None):
toks = name.split(".")
pre, loc = ".".join(toks[:-1]), toks[-1]
try:
module_info = imp.find_module(loc, path)
except ImportError:
module_info = imp.find_module(loc)
if module_info[0]: module_info[0].close
print "A", name, module_info[1]
return None
sys.meta_path = [ImportPrint()]
import mymod1, mymod2, etc..
This almost works, but the __init__.py files are not found this way. Is there a better way to find them, or should I just hackily add them whenever the file found is a directory? Will this method miss any other files.

According to the documentation for sys.meta_path, your find_module method will be called with the path argument set to the path of a package if it is one. Why not use os.path.join(path, '__init__.py') for when path exists?

Importing Python files that are in a folder

I've been working on a project that creates its own .py files that store handlers for the method, I've been trying to figure out how to store the Python files in folder and open them. Here is the code I'm using to create the files if they don't already exist, then importing the file:
if os.path.isfile("Btn"+ str(self.ButtonSet[self.IntBtnID].IntPID) +".py") == False:
TestPy = open("Btn"+ str(self.ButtonSet[self.IntBtnID].IntPID) +".py","w+")
try:
TestPy.write(StrHandler)
except Exception as Error:
print(Error)
TestPy.close()
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Btn" + str(self.IntBtnID))
self.IntBtnID += 1
when I change this line:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Btn" + str(self.IntBtnID))
to this:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Buttons\\Btn" + str(self.IntBtnID))
the fill can't be found and ends up throwing an error because it can't find the file in the folder.
Do know why it doesn't work I just don't know how to get around the issue:/
My question is how do I open the .py when its stored in a folder?

There are a couple of unidiomatic things in your code that may be the cause of your issue. First of all, it is generally better to use the functions in os.path to manipulate paths to files. From your backslash usage, it appears you're working on Windows, but the os.path module ensures consistent behaviour across all platforms.
Also there is importlib.import_module, which is usually recommended over __import__. Furthermore, in case you want to load the generated module more than once during the lifetime of your program, you have to do that explicitly using imp.reload.
One last tip: I'd factor out the module path to avoid having to change it in more than one place.

You can't reference a path directory when you are importing files. Instead, you want to add the directory to your path and then import the name of the module.
import sys
sys.path.append( "Buttons" )
__import__("Btn"+str(self.IntBtnId))
See this so question for more information.

The first argument to the __import__() function is the name of the module, not a path to it. Therefore I think you need to use:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Buttons.Btn" + str(self.IntBtnID))
You may also need to put an empty __init__.py file in the Buttons folder to indicate it's a package of modules.

How to load modules dynamically on package import?

Given the following example layout:
test/
test.py
formats/
__init__.py
format_a.py
format_b.py
What I try to archive is, that whenever I import formats, the __init__.py looks for all available modules in the formats subdir, loads them and makes them available (right now simply through a variable, supported_formats). If theres a better, more pythonic or otherwise approach to dynamically loading stuff on runtime, based on physical available files, please tell.
My Approach
I tried something like this (in __init__.py):
supported_formats = [__import__(f[:f.index('.py')]) for f in glob.glob('*.py')]
So far I just get it to work, when I run __init__.py from the command line (from the formats subdir or from other directories) . But when I import it from test.py, it bails on me like this:
ImportError: No module named format_a.py
Same when I import it from the python interpreter, when I started the interpreter in an other directory but the formats subdir.
Here's the whole code. It also looks for a specific class and stores one instance of each class in an dict, but loading the modules dynamically is the main part I don't get:
def dload(get_cls=True, get_mod=True, key=None, fstring_mod='*.py', fstring_class=''):
if p.dirname(__file__):
path = p.split(p.abspath(__file__))[0]
fstring_mod = p.join(path, fstring_mod)
print >> sys.stderr, 'Path-Glob:', fstring_mod
modules = [p.split(fn)[1][:fn.index('.py')] for fn in glob.glob(fstring_mod)]
print >> sys.stderr, 'Modules:', ', '.join(modules)
modules = [__import__(m) for m in modules]
if get_cls:
classes = {} if key else []
for m in modules:
print >> sys.stderr, "-", m
for c in [m.__dict__[c]() for c in m.__dict__ if c.startswith(fstring_class)]:
print >> sys.stderr, " ", c
if key:
classes[getattr(c, key)] = c
else:
classes.append(c)
if get_mod:
return (modules, classes)
else:
return classes
elif get_mod:
return modules
_supported_formats = dload(get_mod=False, key='fid', fstring_mod='format_*.py', fstring_class='Format')
My Idea
The whole messing with filesystem-paths and the like is probably messy anyway. I would like to handle this with module namespaces or something similar, but I'm kinda lost right now on how start and how to address the modules, so they reachable from anywhere.

There are two fixes you need to make to your code:
You should call __import__(m, globals(), locals()) instead of __import__(m). This is needed for Python to locate the modules within the package.
Your code doesn't remove the .py extension properly since you call index() on the wrong string. If it will always be a .py extension, you can simply use p.split(fn)[1][:-3] instead.

First you must make it so that your code works regardless of the current working directory. For that you use the __file__ variable. You should also use absolute imports.
So something like (untested):
supported_formats = {}
for fn in os.listdir(os.path.dirname(__file__)):
if fn.endswith('.py'):
exec ("from formats import %s" % fn[:-3]) in supported_formats

A module is searched in sys.path. You should be able to extend sys.path with the path to your module. I'm also not really sure whether you can load a module on sys.path with a 'module.py' convention, I would think without '.py' is preferred.
This is obviously not a solution, but may be handy nonetheless.

I thought if you did something that, 'formats' would be your package, so when you tell it import formats you should be able to access the rest of the modules inside that package, so, you would have something like formats.format_a.your_method
Not sure though, I'm just a n00b.

Here's the code I came up with after the corrections from interjay. Still not sure if this is good style.
def load_modules(filemask='*.py', ignore_list=('__init__.py', )):
modules = {}
dirname = os.path.dirname(__file__)
if dirname:
filemask = os.path.join(dirname, filemask)
for fn in glob.glob(filemask):
fn = os.path.split(fn)[1]
if fn in ignore_list:
continue
fn = os.path.splitext(fn)[0]
modules[fn] = __import__(fn, globals(), locals())
return modules

Python Module by Path

I am writing a minimal replacement for mod_python's publisher.py
The basic premise is that it is loading modules based on a URL scheme:
/foo/bar/a/b/c/d
Whereby /foo/ might be a directory and 'bar' is a method ExposedBar in a publishable class in /foo/index.py. Likewise /foo might map to /foo.py and bar is a method in the exposed class. The semantics of this aren't really important. I have a line:
sys.path.insert(0, path_to_file) # /var/www/html/{bar|foo}
mod_obj = __import__(module_name)
mod_obj.__name__ = req.filename
Then the module is inspected for the appropriate class/functions/methods. When the process gets as far as it can the remaining URI data, /a/b/c is passed to that method or function.
This was working fine until I had /var/www/html/foo/index.py and /var/www/html/bar/index.py
When viewing in the browser, it is fairly random which 'index.py' gets selected, even though I set the first search path to '/var/www/html/foo' or '/var/www/html/bar' and then loaded __import__('index'). I have no idea why it is finding either by seemingly random choice. This is shown by:
__name__ is "/var/www/html/foo/index.py"
req.filename is "/var/www/html/foo/index.py"
__file__ is "/var/www/html/bar/index.py"
This question then is, why would the __import__ be randomly selecting either index. I would understand this if the path was '/var/www/html' but it isn't. Secondly:
Can I load a module by it's absolute path into a module object? Without modification of sys.path. I can't find any docs on __import__ or new.module() for this.

Can I load a module by it's absolute
path into a module object? Without
modification of sys.path. I can't find
any docs on __import__ or new.module()
for this.
import imp
import os
def module_from_path(path):
filename = os.path.basename(path)
modulename = os.path.splitext(filename)[0]
with open(path) as f:
return imp.load_module(modulename, f, path, ('py', 'U', imp.PY_SOURCE))