In a little project I have following path-structure:
main.py
Data---
|
__init__.py
Actions_2016_01.py
Actions_2016_02.py
Actions_2016_03.py
... and so on...
Every 'Action_date.py' file includes a list called
data = [something_1, something_2, ...]
Now I am trying get all the data-lists of the 'Action_date.py' files to a single list in the 'main.py' file.
I tried something like
files = os.listdir(os.path.join(os.path.dirname(__name__), 'Data'))
all_data = []
for name in files:
if name.startswith('Actions'):
import Data.name
all_data.extend(name.data)
But this doesn't work at all... I'm getting
ImportError: No module named 'Data.name'
as output.
I found a solution. Simply have to use the importlib module.
You could make your Data package's __init_.py do the work:
def _import_modules():
""" Dynamically import certain modules in the package, extract data in each
of them, and store it in a module global named all_data.
"""
from fnmatch import fnmatch
import traceback
import os
global __all__
__all__ = []
global all_data
all_data = []
globals_, locals_ = globals(), locals()
# dynamically import the desired package modules
for filename in os.listdir(os.path.join(os.path.dirname(__name__), 'Data')):
# process desired python files in directory
if fnmatch(filename, 'Actions*.py'):
modulename = filename.split('.')[0] # filename without extension
package_module = '.'.join([__name__, modulename])
try:
module = __import__(package_module, globals_, locals_, [modulename])
except:
traceback.print_exc()
raise
all_data.extend(module.data)
__all__.append('all_data')
_import_modules()
Which would allow you to do this in main.py:
import Data
print(Data.all_data)
This is an adaption of my answer to the question How to import members of modules within a package?
Related
Using VSCode v1.61.0
I have a python package, a single folder named "tfmodules" with a whole lotta modules, and in the init.py I'm doing this:
from os.path import dirname, basename, isfile, join
import glob
modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')]
In my code I import it like so:
from tfmodules import *
Then I call the modules with something like this:
def write(aci_topology_model, provider_source, provider_version):
with open('fabric-bringup.tf', 'w') as f:
x = fabricbringup.tf_write_fabric_bringup(aci_topology_model, provider_source, provider_version)
f.write(x)
fabricbringup is a module in the package.
The code works just fine, but Pylance is throwing 36 reportUndefinedVariable instances, one for each module called.
The specific error (from one of the modules) is:
"fabricbringup" is not defined Pylance(reportUndefinedVariable)
I have selected the proper interpreter for the venv in vs code, and I tried adding this to the settings.json file:
"python.analysis.extraPaths": ["tfmodules", "/home/aj/projects/python/dcnet-aci-lld-convert/tfmodules"]
But I still get the errors from Pylance. What am I doing wrong? Thanks!
Considering you will not be adding any extensions at runtime how about creating a file that will generate all references directly to "__init__.py"?
if __name__ == "__main__":
from os.path import dirname, basename, isfile, join
import glob
modules = glob.glob(join(dirname(__file__), "*.py"))
print(join(dirname(__file__), "*.py"))
__all__ = ["\t\"" +basename(f)[:-3]+"\",\n" for f in modules if isfile(f) and not f.endswith('__init__.py') and not f.endswith('__add__manager.py')]
f = open(join(dirname(__file__), "__init__.py"), "a")
f.write("__all__ = [\n")
f.writelines(__all__)
f.write("]\n")
f.close()
Result will look something like this:
#(previous contents of __init__.py)
__all__ = [
"file1",
"file2",
]
I have a function that takes in an external path that holds some python files and discovers any classes that exist within those files. It returns a map of the class name and the class itself so that I can create an object from it, if I want. The following code below works how I intended it to, but I am trying to do it without appending the sys.path.
def find_plugins(path):
sys.path.append(path)
plugin_map = {}
current_module_name = os.path.splitext(os.path.basename(path))[0]
for file in glob.glob(path + '/*.py'):
name = os.path.splitext(os.path.basename(file))[0]
if name.startswith('__'):
continue
module = importlib.import_module(name, package=current_module_name)
for member in dir(module):
plugin_class = getattr(module, member)
if plugin_class and inspect.isclass(plugin_class):
plugin_map[member] = plugin_class
return plugin_map
If I remove the sys.path line, I get the following stack trace:
ModuleNotFoundError: No module named 'plugin'
where 'plugin' is the plugin.py file in the path provided to the function. Is it even possible to do this without appending the sys.path?
You could do it using the technique shown in the accepted answer to the question How to import a module given the full path which doesn't require manipulating sys.path.
import importlib.util
import inspect
from pathlib import Path
def find_plugins(path):
plugins = {}
for module_path in Path(path).glob('*.py'):
module_name = module_path.stem
if not module_name.startswith('__'):
# Load module.
spec = importlib.util.spec_from_file_location(module_name, module_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module) # Execute module in its own namespace.
# Extract classes defined in module.
for member in dir(module):
plugin_class = getattr(module, member)
if plugin_class and inspect.isclass(plugin_class):
plugins[member] = plugin_class
return plugins
if __name__ == '__main__':
from pprint import pprint
plugins_folder = '/path/to/plugins'
plugins = find_plugins(plugins_folder)
pprint(plugins)
I need to dynamically import modules into my project from another package.
The structure is like:
project_folder/
project/
__init__.py
__main__.py
plugins/
__init__.py
plugin1/
__init__.py
...
plugin2/
__init__.py
...
I made this function to load a module:
import os
from importlib.util import spec_from_file_location, module_from_spec
def load_module(path, name=""):
""" loads a module by path """
try:
name = name if name != "" else path.split(os.sep)[-1] # take the module name by default
spec = spec_from_file_location(name, os.path.join(path, "__init__.py"))
plugin_module = module_from_spec(spec)
spec.loader.exec_module(plugin_module)
return plugin_module
except Exception as e:
print("failed to load module", path, "-->", e)
It works, unless the module uses relative imports:
failed to load module /path/to/plugins/plugin1 --> Parent module 'plugin1' not loaded, cannot perform relative import
What am I doing wrong?
I managed to solve my own issue after a LOT of googling. Turns out I needed to import using relative paths:
>>> from importlib import import_module
>>> config = import_module("plugins.config")
>>> config
<module 'plugins.config' from '/path/to/plugins/config/__init__.py'>
>>>
I had a similar problem not long ago. I added the path of the project folder to the sys.path using the module's absolute path like this:
import sys
import os
sys.path.append(os.path.dirname(os.path.realpath(__file__))+'/..')
This adds the project_folder to the sys.path thus allowing the import statement to find the plugin modules.
I want to make a function that loads all the .py files in a directory, and imports them using
__import__(), but I keep getting an ImportError: No module named toolboxtool1.
This is file structure:
project/dirreader.py
project/tools/toolboxtool1.py
project/tools/toolboxtool2.py
project/tools/toolboxtool3.py
What am I doing wrong?
import os
os.chdir(os.getcwd()+"/tools/")
stuff = os.listdir(os.getcwd())
for i in range(0,len(stuff)):
if stuff[i][-3:] == ".py":
stuff[i] = stuff[i][:-3]
else:
pass
modules = map(__import__, stuff)
Try prefixing the module names with "tools."
stuff[i] = 'tools.' + stuff[i][:-3]
because the modules you are trying to import are inside tools module package.
I need to reload all the python modules within a specified directory.
I've tried something like this:
import sys, os
import_folder = "C:\\myFolder"
sys.path.insert( 0 , import_folder )
for dir in os.listdir(import_folder):
name = os.path.splitext(dir)[0]
ext = os.path.splitext(dir)[1]
if ext == ".py":
import( eval(name) )
reload( eval(name) )
Anyone know how to do this correctly?
import os # we use os.path.join, os.path.basename
import sys # we use sys.path
import glob # we use glob.glob
import importlib # we use importlib.import_module
import_folder = 'C:\\myFolder'
sys.path.append(import_folder) # this tells python to look in `import_folder` for imports
for src_file in glob.glob(os.path.join(import_folder, '*.py')):
name = os.path.basename(src_file)[:-3]
importlib.import_module(name)
reload(sys.modules[name])
importlib.import_module(name)
There is the code. Now to the semantics of the whole thing: using importlib makes this a little bit more normal, but it still promotes some bugs. You can see that this breaks for source files in a subdirectory. What you should probably do is: import the package, (import the whole folder), and use the . operator like so:
import sys # we use sys.path
sys.path.append('C:\\')
import myFolder
...
myFolder.func1(foo)
myFolder.val
bar = myFolder.Class1()
Perhaps you should take a look at the documentation for modules, but don't forget to update the path to include the parent of the folder you want to import.