We Keep Coding

How to access a docstring from a separate script?

Building a GUI for users to select Python scripts they want to run. Each script has its own docstring explaining inputs and outputs for the script. I want to display that information in the UI once they've highlighted the script, but not selected to run it, and I can't seem to get access to the docstrings from the base program.
ex.
test.py
"""this is a docstring"""
print('hello world')
program.py
index is test.py for this example, but is normally not known because it's whatever the user has selected in the GUI.
# index is test.py
def on_selected(self, index):
script_path = self.tree_view_model.filePath(index)
fparse = ast.parse(''.join(open(script_path)))
self.textBrowser_description.setPlainText(ast.get_docstring(fparse))

Let's the docstring you want to access belongs to the file, file.py.
You can get the docstring by doing the following:
import file
print(file.__doc__)
If you want to get the docstring before you import it then the you could read the file and extract the docstring. Here is an example:
import re
def get_docstring(file)
with open(file, "r") as f:
content = f.read() # read file
quote = content[0] # get type of quote
pattern = re.compile(rf"^{quote}{quote}{quote}[^{quote}]*{quote}{quote}{quote}") # create docstring pattern
return re.findall(pattern, content)[0][3:-3] # return docstring without quotes
print(get_docstring("file.py"))
Note: For this regex to work the docstring will need to be at the very top.

Here's how to get it via importlib. Most of the logic has been put in a function. Note that using importlib does import the script (which causes all its top-level statements to be executed), but the module itself is discarded when the function returns.
If this was the script docstring_test.py in the current directory that I wanted to get the docstring from:
""" this is a multiline
docstring.
"""
print('hello world')
Here's how to do it:
import importlib.util
def get_docstring(script_name, script_path):
spec = importlib.util.spec_from_file_location(script_name, script_path)
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
return foo.__doc__
if __name__ == '__main__':
print(get_docstring('docstring_test', "./docstring_test.py"))
Output:
hello world
this is a multiline
docstring.
Update:
Here's how to do it by letting the ast module in the standard library do the parsing which avoids both importing/executing the script as well as trying to parse it yourself with a regex.
This looks more-or-less equivalent to what's in your question, so it's unclear why what you have isn't working for you.
import ast
def get_docstring(script_path):
with open(script_path, 'r') as file:
tree = ast.parse(file.read())
return ast.get_docstring(tree, clean=False)
if __name__ == '__main__':
print(repr(get_docstring('./docstring_test.py')))
Output:
' this is a multiline\n docstring.\n'

Load dynamically a file, and run a function inside [duplicate]

How do I load a Python module given its full path?
Note that the file can be anywhere in the filesystem where the user has access rights.
See also: How to import a module given its name as string?

For Python 3.5+ use (docs):
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/path/to/file.py")
foo = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = foo
spec.loader.exec_module(foo)
foo.MyClass()
For Python 3.3 and 3.4 use:
from importlib.machinery import SourceFileLoader
foo = SourceFileLoader("module.name", "/path/to/file.py").load_module()
foo.MyClass()
(Although this has been deprecated in Python 3.4.)
For Python 2 use:
import imp
foo = imp.load_source('module.name', '/path/to/file.py')
foo.MyClass()
There are equivalent convenience functions for compiled Python files and DLLs.
See also http://bugs.python.org/issue21436.

The advantage of adding a path to sys.path (over using imp) is that it simplifies things when importing more than one module from a single package. For example:
import sys
# the mock-0.3.1 dir contains testcase.py, testutils.py & mock.py
sys.path.append('/foo/bar/mock-0.3.1')
from testcase import TestCase
from testutils import RunTests
from mock import Mock, sentinel, patch

To import your module, you need to add its directory to the environment variable, either temporarily or permanently.
Temporarily
import sys
sys.path.append("/path/to/my/modules/")
import my_module
Permanently
Adding the following line to your .bashrc (or alternative) file in Linux
and excecute source ~/.bashrc (or alternative) in the terminal:
export PYTHONPATH="${PYTHONPATH}:/path/to/my/modules/"
Credit/Source: saarrrr, another Stack Exchange question

If your top-level module is not a file but is packaged as a directory with __init__.py, then the accepted solution almost works, but not quite. In Python 3.5+ the following code is needed (note the added line that begins with 'sys.modules'):
MODULE_PATH = "/path/to/your/module/__init__.py"
MODULE_NAME = "mymodule"
import importlib
import sys
spec = importlib.util.spec_from_file_location(MODULE_NAME, MODULE_PATH)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
Without this line, when exec_module is executed, it tries to bind relative imports in your top level __init__.py to the top level module name -- in this case "mymodule". But "mymodule" isn't loaded yet so you'll get the error "SystemError: Parent module 'mymodule' not loaded, cannot perform relative import". So you need to bind the name before you load it. The reason for this is the fundamental invariant of the relative import system: "The invariant holding is that if you have sys.modules['spam'] and sys.modules['spam.foo'] (as you would after the above import), the latter must appear as the foo attribute of the former" as discussed here.

It sounds like you don't want to specifically import the configuration file (which has a whole lot of side effects and additional complications involved). You just want to run it, and be able to access the resulting namespace. The standard library provides an API specifically for that in the form of runpy.run_path:
from runpy import run_path
settings = run_path("/path/to/file.py")
That interface is available in Python 2.7 and Python 3.2+.

You can also do something like this and add the directory that the configuration file is sitting in to the Python load path, and then just do a normal import, assuming you know the name of the file in advance, in this case "config".
Messy, but it works.
configfile = '~/config.py'
import os
import sys
sys.path.append(os.path.dirname(os.path.expanduser(configfile)))
import config

I have come up with a slightly modified version of #SebastianRittau's wonderful answer (for Python > 3.4 I think), which will allow you to load a file with any extension as a module using spec_from_loader instead of spec_from_file_location:
from importlib.util import spec_from_loader, module_from_spec
from importlib.machinery import SourceFileLoader
spec = spec_from_loader("module.name", SourceFileLoader("module.name", "/path/to/file.py"))
mod = module_from_spec(spec)
spec.loader.exec_module(mod)
The advantage of encoding the path in an explicit SourceFileLoader is that the machinery will not try to figure out the type of the file from the extension. This means that you can load something like a .txt file using this method, but you could not do it with spec_from_file_location without specifying the loader because .txt is not in importlib.machinery.SOURCE_SUFFIXES.
I've placed an implementation based on this, and #SamGrondahl's useful modification into my utility library, haggis. The function is called haggis.load.load_module. It adds a couple of neat tricks, like the ability to inject variables into the module namespace as it is loaded.

You can use the
load_source(module_name, path_to_file)
method from the imp module.

Do you mean load or import?
You can manipulate the sys.path list specify the path to your module, and then import your module. For example, given a module at:
/foo/bar.py
You could do:
import sys
sys.path[0:0] = ['/foo'] # Puts the /foo directory at the start of your path
import bar

Here is some code that works in all Python versions, from 2.7-3.5 and probably even others.
config_file = "/tmp/config.py"
with open(config_file) as f:
code = compile(f.read(), config_file, 'exec')
exec(code, globals(), locals())
I tested it. It may be ugly, but so far it is the only one that works in all versions.

You can do this using __import__ and chdir:
def import_file(full_path_to_module):
try:
import os
module_dir, module_file = os.path.split(full_path_to_module)
module_name, module_ext = os.path.splitext(module_file)
save_cwd = os.getcwd()
os.chdir(module_dir)
module_obj = __import__(module_name)
module_obj.__file__ = full_path_to_module
globals()[module_name] = module_obj
os.chdir(save_cwd)
except Exception as e:
raise ImportError(e)
return module_obj
import_file('/home/somebody/somemodule.py')

If we have scripts in the same project but in different directory means, we can solve this problem by the following method.
In this situation utils.py is in src/main/util/
import sys
sys.path.append('./')
import src.main.util.utils
#or
from src.main.util.utils import json_converter # json_converter is example method

To add to Sebastian Rittau's answer:
At least for CPython, there's pydoc, and, while not officially declared, importing files is what it does:
from pydoc import importfile
module = importfile('/path/to/module.py')
PS. For the sake of completeness, there's a reference to the current implementation at the moment of writing: pydoc.py, and I'm pleased to say that in the vein of xkcd 1987 it uses neither of the implementations mentioned in issue 21436 -- at least, not verbatim.

I believe you can use imp.find_module() and imp.load_module() to load the specified module. You'll need to split the module name off of the path, i.e. if you wanted to load /home/mypath/mymodule.py you'd need to do:
imp.find_module('mymodule', '/home/mypath/')
...but that should get the job done.

You can use the pkgutil module (specifically the walk_packages method) to get a list of the packages in the current directory. From there it's trivial to use the importlib machinery to import the modules you want:
import pkgutil
import importlib
packages = pkgutil.walk_packages(path='.')
for importer, name, is_package in packages:
mod = importlib.import_module(name)
# do whatever you want with module now, it's been imported!

There's a package that's dedicated to this specifically:
from thesmuggler import smuggle
# À la `import weapons`
weapons = smuggle('weapons.py')
# À la `from contraband import drugs, alcohol`
drugs, alcohol = smuggle('drugs', 'alcohol', source='contraband.py')
# À la `from contraband import drugs as dope, alcohol as booze`
dope, booze = smuggle('drugs', 'alcohol', source='contraband.py')
It's tested across Python versions (Jython and PyPy too), but it might be overkill depending on the size of your project.

Create Python module test.py:
import sys
sys.path.append("<project-path>/lib/")
from tes1 import Client1
from tes2 import Client2
import tes3
Create Python module test_check.py:
from test import Client1
from test import Client2
from test import test3
We can import the imported module from module.

This area of Python 3.4 seems to be extremely tortuous to understand! However with a bit of hacking using the code from Chris Calloway as a start I managed to get something working. Here's the basic function.
def import_module_from_file(full_path_to_module):
"""
Import a module given the full path/filename of the .py file
Python 3.4
"""
module = None
try:
# Get module name and path from full path
module_dir, module_file = os.path.split(full_path_to_module)
module_name, module_ext = os.path.splitext(module_file)
# Get module "spec" from filename
spec = importlib.util.spec_from_file_location(module_name,full_path_to_module)
module = spec.loader.load_module()
except Exception as ec:
# Simple error printing
# Insert "sophisticated" stuff here
print(ec)
finally:
return module
This appears to use non-deprecated modules from Python 3.4. I don't pretend to understand why, but it seems to work from within a program. I found Chris' solution worked on the command line but not from inside a program.

I made a package that uses imp for you. I call it import_file and this is how it's used:
>>>from import_file import import_file
>>>mylib = import_file('c:\\mylib.py')
>>>another = import_file('relative_subdir/another.py')
You can get it at:
http://pypi.python.org/pypi/import_file
or at
http://code.google.com/p/import-file/

To import a module from a given filename, you can temporarily extend the path, and restore the system path in the finally block reference:
filename = "directory/module.py"
directory, module_name = os.path.split(filename)
module_name = os.path.splitext(module_name)[0]
path = list(sys.path)
sys.path.insert(0, directory)
try:
module = __import__(module_name)
finally:
sys.path[:] = path # restore

A simple solution using importlib instead of the imp package (tested for Python 2.7, although it should work for Python 3 too):
import importlib
dirname, basename = os.path.split(pyfilepath) # pyfilepath: '/my/path/mymodule.py'
sys.path.append(dirname) # only directories should be added to PYTHONPATH
module_name = os.path.splitext(basename)[0] # '/my/path/mymodule.py' --> 'mymodule'
module = importlib.import_module(module_name) # name space of defined module (otherwise we would literally look for "module_name")
Now you can directly use the namespace of the imported module, like this:
a = module.myvar
b = module.myfunc(a)
The advantage of this solution is that we don't even need to know the actual name of the module we would like to import, in order to use it in our code. This is useful, e.g. in case the path of the module is a configurable argument.

I have written my own global and portable import function, based on importlib module, for:
Be able to import both modules as submodules and to import the content of a module to a parent module (or into a globals if has no parent module).
Be able to import modules with a period characters in a file name.
Be able to import modules with any extension.
Be able to use a standalone name for a submodule instead of a file name without extension which is by default.
Be able to define the import order based on previously imported module instead of dependent on sys.path or on a what ever search path storage.
The examples directory structure:
<root>
|
+- test.py
|
+- testlib.py
|
+- /std1
| |
| +- testlib.std1.py
|
+- /std2
| |
| +- testlib.std2.py
|
+- /std3
|
+- testlib.std3.py
Inclusion dependency and order:
test.py
-> testlib.py
-> testlib.std1.py
-> testlib.std2.py
-> testlib.std3.py
Implementation:
Latest changes store: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/python/tacklelib/tacklelib.py
test.py:
import os, sys, inspect, copy
SOURCE_FILE = os.path.abspath(inspect.getsourcefile(lambda:0)).replace('\\','/')
SOURCE_DIR = os.path.dirname(SOURCE_FILE)
print("test::SOURCE_FILE: ", SOURCE_FILE)
# portable import to the global space
sys.path.append(TACKLELIB_ROOT) # TACKLELIB_ROOT - path to the library directory
import tacklelib as tkl
tkl.tkl_init(tkl)
# cleanup
del tkl # must be instead of `tkl = None`, otherwise the variable would be still persist
sys.path.pop()
tkl_import_module(SOURCE_DIR, 'testlib.py')
print(globals().keys())
testlib.base_test()
testlib.testlib_std1.std1_test()
testlib.testlib_std1.testlib_std2.std2_test()
#testlib.testlib.std3.std3_test() # does not reachable directly ...
getattr(globals()['testlib'], 'testlib.std3').std3_test() # ... but reachable through the `globals` + `getattr`
tkl_import_module(SOURCE_DIR, 'testlib.py', '.')
print(globals().keys())
base_test()
testlib_std1.std1_test()
testlib_std1.testlib_std2.std2_test()
#testlib.std3.std3_test() # does not reachable directly ...
globals()['testlib.std3'].std3_test() # ... but reachable through the `globals` + `getattr`
testlib.py:
# optional for 3.4.x and higher
#import os, inspect
#
#SOURCE_FILE = os.path.abspath(inspect.getsourcefile(lambda:0)).replace('\\','/')
#SOURCE_DIR = os.path.dirname(SOURCE_FILE)
print("1 testlib::SOURCE_FILE: ", SOURCE_FILE)
tkl_import_module(SOURCE_DIR + '/std1', 'testlib.std1.py', 'testlib_std1')
# SOURCE_DIR is restored here
print("2 testlib::SOURCE_FILE: ", SOURCE_FILE)
tkl_import_module(SOURCE_DIR + '/std3', 'testlib.std3.py')
print("3 testlib::SOURCE_FILE: ", SOURCE_FILE)
def base_test():
print('base_test')
testlib.std1.py:
# optional for 3.4.x and higher
#import os, inspect
#
#SOURCE_FILE = os.path.abspath(inspect.getsourcefile(lambda:0)).replace('\\','/')
#SOURCE_DIR = os.path.dirname(SOURCE_FILE)
print("testlib.std1::SOURCE_FILE: ", SOURCE_FILE)
tkl_import_module(SOURCE_DIR + '/../std2', 'testlib.std2.py', 'testlib_std2')
def std1_test():
print('std1_test')
testlib.std2.py:
# optional for 3.4.x and higher
#import os, inspect
#
#SOURCE_FILE = os.path.abspath(inspect.getsourcefile(lambda:0)).replace('\\','/')
#SOURCE_DIR = os.path.dirname(SOURCE_FILE)
print("testlib.std2::SOURCE_FILE: ", SOURCE_FILE)
def std2_test():
print('std2_test')
testlib.std3.py:
# optional for 3.4.x and higher
#import os, inspect
#
#SOURCE_FILE = os.path.abspath(inspect.getsourcefile(lambda:0)).replace('\\','/')
#SOURCE_DIR = os.path.dirname(SOURCE_FILE)
print("testlib.std3::SOURCE_FILE: ", SOURCE_FILE)
def std3_test():
print('std3_test')
Output (3.7.4):
test::SOURCE_FILE: <root>/test01/test.py
import : <root>/test01/testlib.py as testlib -> []
1 testlib::SOURCE_FILE: <root>/test01/testlib.py
import : <root>/test01/std1/testlib.std1.py as testlib_std1 -> ['testlib']
import : <root>/test01/std1/../std2/testlib.std2.py as testlib_std2 -> ['testlib', 'testlib_std1']
testlib.std2::SOURCE_FILE: <root>/test01/std1/../std2/testlib.std2.py
2 testlib::SOURCE_FILE: <root>/test01/testlib.py
import : <root>/test01/std3/testlib.std3.py as testlib.std3 -> ['testlib']
testlib.std3::SOURCE_FILE: <root>/test01/std3/testlib.std3.py
3 testlib::SOURCE_FILE: <root>/test01/testlib.py
dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'os', 'sys', 'inspect', 'copy', 'SOURCE_FILE', 'SOURCE_DIR', 'TackleGlobalImportModuleState', 'tkl_membercopy', 'tkl_merge_module', 'tkl_get_parent_imported_module_state', 'tkl_declare_global', 'tkl_import_module', 'TackleSourceModuleState', 'tkl_source_module', 'TackleLocalImportModuleState', 'testlib'])
base_test
std1_test
std2_test
std3_test
import : <root>/test01/testlib.py as . -> []
1 testlib::SOURCE_FILE: <root>/test01/testlib.py
import : <root>/test01/std1/testlib.std1.py as testlib_std1 -> ['testlib']
import : <root>/test01/std1/../std2/testlib.std2.py as testlib_std2 -> ['testlib', 'testlib_std1']
testlib.std2::SOURCE_FILE: <root>/test01/std1/../std2/testlib.std2.py
2 testlib::SOURCE_FILE: <root>/test01/testlib.py
import : <root>/test01/std3/testlib.std3.py as testlib.std3 -> ['testlib']
testlib.std3::SOURCE_FILE: <root>/test01/std3/testlib.std3.py
3 testlib::SOURCE_FILE: <root>/test01/testlib.py
dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'os', 'sys', 'inspect', 'copy', 'SOURCE_FILE', 'SOURCE_DIR', 'TackleGlobalImportModuleState', 'tkl_membercopy', 'tkl_merge_module', 'tkl_get_parent_imported_module_state', 'tkl_declare_global', 'tkl_import_module', 'TackleSourceModuleState', 'tkl_source_module', 'TackleLocalImportModuleState', 'testlib', 'testlib_std1', 'testlib.std3', 'base_test'])
base_test
std1_test
std2_test
std3_test
Tested in Python 3.7.4, 3.2.5, 2.7.16
Pros:
Can import both module as a submodule and can import content of a module to a parent module (or into a globals if has no parent module).
Can import modules with periods in a file name.
Can import any extension module from any extension module.
Can use a standalone name for a submodule instead of a file name without extension which is by default (for example, testlib.std.py as testlib, testlib.blabla.py as testlib_blabla and so on).
Does not depend on a sys.path or on a what ever search path storage.
Does not require to save/restore global variables like SOURCE_FILE and SOURCE_DIR between calls to tkl_import_module.
[for 3.4.x and higher] Can mix the module namespaces in nested tkl_import_module calls (ex: named->local->named or local->named->local and so on).
[for 3.4.x and higher] Can auto export global variables/functions/classes from where being declared to all children modules imported through the tkl_import_module (through the tkl_declare_global function).
Cons:
Does not support complete import:
Ignores enumerations and subclasses.
Ignores builtins because each what type has to be copied exclusively.
Ignore not trivially copiable classes.
Avoids copying builtin modules including all packaged modules.
[for 3.3.x and lower] Require to declare tkl_import_module in all modules which calls to tkl_import_module (code duplication)
Update 1,2 (for 3.4.x and higher only):
In Python 3.4 and higher you can bypass the requirement to declare tkl_import_module in each module by declare tkl_import_module in a top level module and the function would inject itself to all children modules in a single call (it's a kind of self deploy import).
Update 3:
Added function tkl_source_module as analog to bash source with support execution guard upon import (implemented through the module merge instead of import).
Update 4:
Added function tkl_declare_global to auto export a module global variable to all children modules where a module global variable is not visible because is not a part of a child module.
Update 5:
All functions has moved into the tacklelib library, see the link above.

This should work
path = os.path.join('./path/to/folder/with/py/files', '*.py')
for infile in glob.glob(path):
basename = os.path.basename(infile)
basename_without_extension = basename[:-3]
# http://docs.python.org/library/imp.html?highlight=imp#module-imp
imp.load_source(basename_without_extension, infile)

Import package modules at runtime (Python recipe)
http://code.activestate.com/recipes/223972/
###################
## #
## classloader.py #
## #
###################
import sys, types
def _get_mod(modulePath):
try:
aMod = sys.modules[modulePath]
if not isinstance(aMod, types.ModuleType):
raise KeyError
except KeyError:
# The last [''] is very important!
aMod = __import__(modulePath, globals(), locals(), [''])
sys.modules[modulePath] = aMod
return aMod
def _get_func(fullFuncName):
"""Retrieve a function object from a full dotted-package name."""
# Parse out the path, module, and function
lastDot = fullFuncName.rfind(u".")
funcName = fullFuncName[lastDot + 1:]
modPath = fullFuncName[:lastDot]
aMod = _get_mod(modPath)
aFunc = getattr(aMod, funcName)
# Assert that the function is a *callable* attribute.
assert callable(aFunc), u"%s is not callable." % fullFuncName
# Return a reference to the function itself,
# not the results of the function.
return aFunc
def _get_class(fullClassName, parentClass=None):
"""Load a module and retrieve a class (NOT an instance).
If the parentClass is supplied, className must be of parentClass
or a subclass of parentClass (or None is returned).
"""
aClass = _get_func(fullClassName)
# Assert that the class is a subclass of parentClass.
if parentClass is not None:
if not issubclass(aClass, parentClass):
raise TypeError(u"%s is not a subclass of %s" %
(fullClassName, parentClass))
# Return a reference to the class itself, not an instantiated object.
return aClass
######################
## Usage ##
######################
class StorageManager: pass
class StorageManagerMySQL(StorageManager): pass
def storage_object(aFullClassName, allOptions={}):
aStoreClass = _get_class(aFullClassName, StorageManager)
return aStoreClass(allOptions)

I'm not saying that it is better, but for the sake of completeness, I wanted to suggest the exec function, available in both Python 2 and Python 3.
exec allows you to execute arbitrary code in either the global scope, or in an internal scope, provided as a dictionary.
For example, if you have a module stored in "/path/to/module" with the function foo(), you could run it by doing the following:
module = dict()
with open("/path/to/module") as f:
exec(f.read(), module)
module['foo']()
This makes it a bit more explicit that you're loading code dynamically, and grants you some additional power, such as the ability to provide custom builtins.
And if having access through attributes, instead of keys is important to you, you can design a custom dict class for the globals, that provides such access, e.g.:
class MyModuleClass(dict):
def __getattr__(self, name):
return self.__getitem__(name)

In Linux, adding a symbolic link in the directory your Python script is located works.
I.e.:
ln -s /absolute/path/to/module/module.py /absolute/path/to/script/module.py
The Python interpreter will create /absolute/path/to/script/module.pyc and will update it if you change the contents of /absolute/path/to/module/module.py.
Then include the following in file mypythonscript.py:
from module import *

This will allow imports of compiled (pyd) Python modules in 3.4:
import sys
import importlib.machinery
def load_module(name, filename):
# If the Loader finds the module name in this list it will use
# module_name.__file__ instead so we need to delete it here
if name in sys.modules:
del sys.modules[name]
loader = importlib.machinery.ExtensionFileLoader(name, filename)
module = loader.load_module()
locals()[name] = module
globals()[name] = module
load_module('something', r'C:\Path\To\something.pyd')
something.do_something()

A quite simple way: suppose you want import file with relative path ../../MyLibs/pyfunc.py
libPath = '../../MyLibs'
import sys
if not libPath in sys.path: sys.path.append(libPath)
import pyfunc as pf
But if you make it without a guard you can finally get a very long path.

These are my two utility functions using only pathlib. It infers the module name from the path.
By default, it recursively loads all Python files from folders and replaces init.py by the parent folder name. But you can also give a Path and/or a glob to select some specific files.
from pathlib import Path
from importlib.util import spec_from_file_location, module_from_spec
from typing import Optional
def get_module_from_path(path: Path, relative_to: Optional[Path] = None):
if not relative_to:
relative_to = Path.cwd()
abs_path = path.absolute()
relative_path = abs_path.relative_to(relative_to.absolute())
if relative_path.name == "__init__.py":
relative_path = relative_path.parent
module_name = ".".join(relative_path.with_suffix("").parts)
mod = module_from_spec(spec_from_file_location(module_name, path))
return mod
def get_modules_from_folder(folder: Optional[Path] = None, glob_str: str = "*/**/*.py"):
if not folder:
folder = Path(".")
mod_list = []
for file_path in sorted(folder.glob(glob_str)):
mod_list.append(get_module_from_path(file_path))
return mod_list

This answer is a supplement to Sebastian Rittau's answer responding to the comment: "but what if you don't have the module name?" This is a quick and dirty way of getting the likely Python module name given a filename -- it just goes up the tree until it finds a directory without an __init__.py file and then turns it back into a filename. For Python 3.4+ (uses pathlib), which makes sense since Python 2 people can use "imp" or other ways of doing relative imports:
import pathlib
def likely_python_module(filename):
'''
Given a filename or Path, return the "likely" python module name. That is, iterate
the parent directories until it doesn't contain an __init__.py file.
:rtype: str
'''
p = pathlib.Path(filename).resolve()
paths = []
if p.name != '__init__.py':
paths.append(p.stem)
while True:
p = p.parent
if not p:
break
if not p.is_dir():
break
inits = [f for f in p.iterdir() if f.name == '__init__.py']
if not inits:
break
paths.append(p.stem)
return '.'.join(reversed(paths))
There are certainly possibilities for improvement, and the optional __init__.py files might necessitate other changes, but if you have __init__.py in general, this does the trick.

how to import function in pytest

I am trying to create a pytest for a Python 2.x script executable with dashes included in its name. I tried to import it the usual way but I can't figure out how to make it work with the dashes.
My project structure is as follows:
package
-- tests
-- bin
-- subpackage
-- ...py
Specifically, I need to test a function called master_disaster() which exists inside bin/let-me-out (yes with -). let-me-out is an executable .py file and my folder has no setup.py file or anything similar.
How can I import this function inside my test? My test is going to be a simple fixture that checks the time with:
#pytest.fixture
def now():
return timezone.now()
It then uses the now() function to create a new file which let-me-out will delete after a specific amount of time.

First of all, dashes make let-me-out word to an invalid identifier in Python. To work around it, you have to invoke the imp (Python 2.7)
or importlib (Python 3.5+) machinery.
Python 3.5+
Here is an example of importing a new module having a qualified name let_me_out, but using bin/let-me-out as source file:
import importlib
def test_master_disaster():
loader = importlib.machinery.SourceFileLoader('let_me_out', 'bin/let-me-out')
spec = importlib.util.spec_from_loader(loader.name, loader)
let_me_out = importlib.util.module_from_spec(spec)
loader.exec_module(let_me_out)
# this is only a stub, to show an example of calling the master_disaster function
assert let_me_out.master_disaster() == 'spam'
You can extract this code into a fixture to make it reusable:
import importlib
import pytest
#pytest.fixture(scope='session')
def let_me_out():
loader = importlib.machinery.SourceFileLoader('let_me_out', 'bin/let-me-out')
spec = importlib.util.spec_from_loader(loader.name, loader)
let_me_out = importlib.util.module_from_spec(spec)
loader.exec_module(let_me_out)
return let_me_out
def test_master_disaster(let_me_out):
assert let_me_out.master_disaster() == 'spam'
Python 2.7
Things are even easier with Python 2.7:
import imp
import pytest
#pytest.fixture(scope='session')
def let_me_out():
return imp.load_source('let_me_out', 'bin/let-me-out')
def test_master_disaster(let_me_out):
assert let_me_out.master_disaster() == 'spam'

boilerplateless tests [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I have testing files wirtten in Python : test1.py test2.py...
Before executing any of them I need to initialize them with a file called initialize.py that takes arguments.
The testing files must stay as light and easy to write as they can be.
I want to create a script that:
Takes input arguments
Start the initialize.py file with those arguments
Start the test file using the variables created by initialize.py
I thought about a few ways:
Import the two files : it doesn't work because with import you can use the return argument on your main script but you can't give input argument
Transform both files into functions : it's not an issue with initialize.py but as I said I want to keep the test file to be as easy and light as possible so if I can avoid that it's better.
The perfect solution would be to simply "copy" the code from initialize and put it in the beggining of the test file (or the opposite). May be create a temporary file containing both code but i don't think it very clean.
To sum up : it's as if I had 100 files that starts with the same 25 lines and I want to put those 25 lines in one file and import them each time.
An other way to see things is 3 files:
#File1.py
var1 = sys.argv(1)
#File2.py
var2 = var1+"second"
#File3.py
var3 = var1+var2+"third"
print var3
I want to start ./File1.py first
And get "first second thrid"
I succeed with
#! /usr/bin/python
import sys
import subprocess
source_content = "#! /usr/bin/python\n"+"import sys\n"
sourcefile = "file2.py"
txt_file = open(sourcefile)
source_content += txt_file.read()
sourcefile = "file3.py"
txt_file = open(sourcefile)
source_content += txt_file.read()
destinationfile = "Copyfile2.py"
target = open (destinationfile, 'w')
target.write(source_content)
target.close()
chmodFile = "chmod 777 "+destinationfile
chmod = subprocess.Popen(chmodFile, shell=True)
chmod.wait()
arguments = str("./"+destinationfile+" ")
arguments += " ".join(map(str,sys.argv[1:len(sys.argv)]))
startTest = subprocess.Popen(arguments, shell=True)
startTest.wait()
But I had to delete the "#! /usr/bin/python" from test2 and test and rename var1 to sys.arg[1] on thoses same files.
And I don't think it's a nice solution...

How about you use the unittest module?
import unittest
class BaseTest(unittest.TestCase):
def initialisation_script_stuff(blah,etc):
foo
def setUp(self):
common_setup_stuff()
def tearDown(self):
whatever
Now you can just inherit from BaseTest in each of your test files.
from moo import BaseTest
class CoolTest(BaseTest):
def setUp(self):
BaseTest.setUp(self)
args = BaseTest.initialisation_script_stuff()
do_stuff(args)
def testNumberOne(self):
self.assertEqual(1,1)
Alternatively if you want to stay away from standard unit testing methods...
Assuming the directory structure:
all_tests\
__init__.py
some_tests\
__init__.py
test1.py
test2.py
other _tests\
__init__.py
etc
Some naming conventions:
each test py file has a function named run that runs the test
each test py file has a name starting with 'test'
each of the test grouping folders has a name ending in '_tests'
Make a single script called run_tests.py (or something like that...)
def run_tests():
import os
import importlib
import re
dTests = {}
lFolders = [s for s in os.listdir('all_tests') if re.match('.*_tests$',s)]
for sFolder in lFolders:
sFolderPath = os.path.join('all_tests',sFolder)
lTestFileNames = [s for s in os.listdir(sFolderPath) if re.match('^test.*py$',s)]
for sFileName in lTestFileNames:
sSubPath = '{0}.{1}'.format(sFolder,sFileName.split('.')[0])
dTests[sSubPath] = importlib.import_module('all_tests.{0}'.format(sSubPath))
#now you have all the tests...
for k in dTests:
stuff = initialisation_stuff()
test_result = dTests[k].run(stuff)
do_whatever_you_want(test_result)
if __name__ == "__main__":
run_tests()
Now you need absolutely no boiler plate code in your test files. Just so long as you follow the formula

Importing classes from different files in a subdirectory

Here's the structure I'm working with:
directory/
script.py
subdir/
__init__.py
myclass01.py
myclass02.py
What I want to do is import in script.py the classes defined in myclass01.py and myclass02.py. If I do:
from subdir.myclass01 import *
It works fine for the class defined in myclass01.py. But with this solution if there are many classes defined in different files in subdir and I want to import all of them, I'd have to type one line for each file. There must be a shortcut for this. I tried:
from subdir.* import *
But it didn't work out.
EDIT: here are the contents of the files:
This is __init__.py (using __all__ as Apalala suggested):
__all__ = ['MyClass01','MyClass02']
This is myclass01.py:
class MyClass01:
def printsomething():
print 'hey'
This is myclass02.py:
class MyClass02:
def printsomething():
print 'sup'
This is script.py:
from subdir import *
MyClass01().printsomething()
MyClass02().printsomething()
This is the traceback that I get when I try to run script.py:
File "script.py", line 1, in <module>
from subdir import *
AttributeError: 'module' object has no attribute 'MyClass01'

Although the names used there are different from what's shown in your question's directory structure, you could use my answer to the question titled Namespacing and classes. The __init__.py shown there would have also allowed the usepackage.py script to have been written this way (package maps to subdir in your question, and Class1 to myclass01, etc):
from package import *
print Class1
print Class2
print Class3
Revision (updated):
Oops, sorry, the code in my other answer doesn't quite do what you want — it only automatically imports the names of any package submodules. To make it also import the named attributes from each submodule requires a few more lines of code. Here's a modified version of the package's __init__.py file (which also works in Python 3.4.1):
def _import_package_files():
""" Dynamically import all the public attributes of the python modules in this
file's directory (the package directory) and return a list of their names.
"""
import os
exports = []
globals_, locals_ = globals(), locals()
package_path = os.path.dirname(__file__)
package_name = os.path.basename(package_path)
for filename in os.listdir(package_path):
modulename, ext = os.path.splitext(filename)
if modulename[0] != '_' and ext in ('.py', '.pyw'):
subpackage = '{}.{}'.format(package_name, modulename) # pkg relative
module = __import__(subpackage, globals_, locals_, [modulename])
modict = module.__dict__
names = (modict['__all__'] if '__all__' in modict else
[name for name in modict if name[0] != '_']) # all public
exports.extend(names)
globals_.update((name, modict[name]) for name in names)
return exports
if __name__ != '__main__':
__all__ = ['__all__'] + _import_package_files() # '__all__' in __all__
Alternatively you can put the above into a separate .py module file of its own in the package directory—such as _import_package_files.py—and use it from the package's __init__.py like this:
if __name__ != '__main__':
from ._import_package_files import * # defines __all__
__all__.remove('__all__') # prevent export (optional)
Whatever you name the file, it should be something that starts with an _ underscore character so it doesn't try to import itself recursively.

Your best option, though probably not the best style, is to import everything into the package's namespace:
# this is subdir/__init__.py
from myclass01 import *
from myclass02 import *
from myclass03 import *
Then, in other modules, you can import what you want directly from the package:
from subdir import Class1

I know it's been a couple months since this question was answered, but I was looking for the same thing and ran across this page. I wasn't very satisfied with the chosen answer, so I ended up writing my own solution and thought I'd share it. Here's what I came up with:
# NOTE: The function name starts with an underscore so it doesn't get deleted by iself
def _load_modules(attr_filter=None):
import os
curdir = os.path.dirname(__file__)
imports = [os.path.splitext(fname)[0] for fname in os.listdir(curdir) if fname.endswith(".py")]
pubattrs = {}
for mod_name in imports:
mod = __import__(mod_name, globals(), locals(), ['*'], -1)
for attr in mod.__dict__:
if not attr.startswith('_') and (not attr_filter or attr_filter(mod_name, attr)):
pubattrs[attr] = getattr(mod, attr)
# Restore the global namespace to it's initial state
for var in globals().copy():
if not var.startswith('_'):
del globals()[var]
# Update the global namespace with the specific items we want
globals().update(pubattrs)
# EXAMPLE: Only load classes that end with "Resource"
_load_modules(attr_filter=lambda mod, attr: True if attr.endswith("Resource") else False)
del _load_modules # Keep the namespace clean
This simply imports * from all .py files in the package directory and then only pulls the public ones into the global namespace. Additionally, it allows a filter if only certain public attributes are desired.

I use this simple way:
add the directory to the system path, then
import module or from module import function1, class1 in that directory.
notice that the module is nothing but the name of your *.py file, without the extension part.
Here is a general example:
import sys
sys.path.append("/path/to/folder/")
import module # in that folder
In your case it could be something like this:
import sys
sys.path.append("subdir/")
import myclass01
# or
from myclass01 import func1, class1, class2 # .. etc

from subdir.* import *
You can not use '*' this way directly after the 'from' statement.
You need explict imports. Please check with the Python documentation on imports and packages.