I need to dynamically import modules into my project from another package.
The structure is like:
project_folder/
project/
__init__.py
__main__.py
plugins/
__init__.py
plugin1/
__init__.py
...
plugin2/
__init__.py
...
I made this function to load a module:
import os
from importlib.util import spec_from_file_location, module_from_spec
def load_module(path, name=""):
""" loads a module by path """
try:
name = name if name != "" else path.split(os.sep)[-1] # take the module name by default
spec = spec_from_file_location(name, os.path.join(path, "__init__.py"))
plugin_module = module_from_spec(spec)
spec.loader.exec_module(plugin_module)
return plugin_module
except Exception as e:
print("failed to load module", path, "-->", e)
It works, unless the module uses relative imports:
failed to load module /path/to/plugins/plugin1 --> Parent module 'plugin1' not loaded, cannot perform relative import
What am I doing wrong?
I managed to solve my own issue after a LOT of googling. Turns out I needed to import using relative paths:
>>> from importlib import import_module
>>> config = import_module("plugins.config")
>>> config
<module 'plugins.config' from '/path/to/plugins/config/__init__.py'>
>>>
I had a similar problem not long ago. I added the path of the project folder to the sys.path using the module's absolute path like this:
import sys
import os
sys.path.append(os.path.dirname(os.path.realpath(__file__))+'/..')
This adds the project_folder to the sys.path thus allowing the import statement to find the plugin modules.
Related
I am trying to import python module from one directory to another directory but not able to do it.
Here is my folder structure
Parent_Folder
---------Module1
----__init__.py (empty file)
----mymodule1.py
---------Module2
----__init__.py (empty file)
----mymodule2.py
I am trying to import
mymodule1.py
into
mymodule2.py
Here is the command that I have used to import the python module:
import Module1.mymodule1
but getting an error like
ModuleNotFoundError: No module named 'Module1'
I do see there are option like system-path at runtime, PYTHONPATH but I don't want to use those options.
Do you all have any other recommended solutions?
You can insert this at the top of module1.py, or in the init.py in Module1 if you're importing module1.py as well:
import sys
sys.path.append("../Module2") #assuming linuxOS
import module2
This adds the Module2 directory to the system path and allows you to import files as if they were local to your current directory.
Other answers can be found in this link: beyond top level package error in relative import
I'll give you an example of importing modules
I created the module1.py file
print("Module : 1")
def module01():#function
print("module : 1.2")
class Module02:# class
def module021(self):# function
print('MODULE 1')
to import you need to indicate the file name
module2.py
from module1 import module01 #import the function
from module1 import Module02 #importing the class from module2.py
module01()
mod1 = Module02() #
mod1.module021()# imported class function
I have a simple python project with the following directory structure:
sample-project/
main/
config/
config.yaml
docker/
Dockerfile
tests/
__init__.py
sometests.py
config.py
run.py
__init__.py
requirements.txt
README.rst
In the config.py file, I have:
import yaml
class Config:
def __init__(self):
with open("config/config.yaml", 'r') as ymlfile:
cfg = yaml.load(ymlfile)
self.host = cfg["mysql"]["host"]
self.database = cfg["mysql"]["database"]
self.user = cfg["mysql"]["user"]
self.password = cfg["mysql"]["password"]
my_config = Config()
In the run.py file, I have an import statement as follows:
from main.config import my_config
when I run using command line: python3.5 run.py, I get the error:
from main.config import my_config
ImportError: No module named 'main'
But when I add this in run.py imports it works:
import sys
sys.path.append('/home/tomas/sample-project/')
Is there any better way to make it work rather than give absolute path or any other way? Please give some example :)
Generally, never ever touch sys.path from within your program.
Since main/ (not the best name) contains an __init__.py, we'll consider it a package, so let's make it runnable as one. This way imports will be considered correctly.
Add main/__main__.py with e.g.
from main.run import run
if __name__ == "__main__":
run()
Then run your app with
python -m main
instead of
python main/main.py
when mentioning below
from main.config import my_config
keep dir path also
eg:
from sample-project/main.config import my_config
assuming you are executing from /home/tomas
Although I share AKX's sentiment that editing your path is something to avoid I would generally suggest the following method:
import sys
from os.path import dirname
abs_dir = dirname(__file__)
sys.path.append(abs_dir)
The __file__ variable provides access to the location of the file in which it is accessed. This allows you to get an absolute path to any file in your project.
Trying to import aFile.py from within the bSubFile.py but getting an error saying 'exceptions.ValueError, Attempted relative import in non-package'
My file structure is as follows:
app/
- __init__.py
FolderA/
- __init__.py
- aFile.py
FolderB/
- __init__.py
- bFile.py
SubfolderB/
- __init__.py
- bSubFile.py
I am trying to import aFile from bSubFile.py
tried:
from ..FolderA import aFile
class bSubFile():
...
and:
from ...FolderA import aFile
class bSubFile():
...
but I always get 'Attempted relative import in non-package', I must be missing something very obvious. Thanks!
You can add the other path to your system path. This may be not the most elegant way, but it works.
import sys
import os
# get the folder of the current file, go one directory back and add "FolderA"
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "FolderA"))
# now you can import everything that is in FolderA directly
import aFile
I have following file structure
./AppFolder
__init__.py
main.py
./plugin
__init__.py
./simple_plugin_1
__init__.py
simple_plugin_1.py
./simple_plugin_2
__init__.py
simple_plugin_2.py
and i want to load all plugin modules recursively into main.py. So can I use __init__.py in AppFolder as following?
import os
import glob
__all__ = [ os.path.basename(f)[:-3] for f in glob.glob(os.path.dirname(__file__)+"/plugin/*/*.py")]
Or is there are other techniques?
I have this bit of code that I use for this purpose, just use this as your top level __init__.py:
import pkgutil
__all__ = []
for loader, module_name, is_pkg in pkgutil.walk_packages(__path__):
module = loader.find_module(module_name).load_module(module_name)
exec('%s = module' % module_name)
__all__.append(module_name)
Basically same idea as what you have, but just using pkgutil.walk_packages to find all the modules instead of a file glob.
I need to reload all the python modules within a specified directory.
I've tried something like this:
import sys, os
import_folder = "C:\\myFolder"
sys.path.insert( 0 , import_folder )
for dir in os.listdir(import_folder):
name = os.path.splitext(dir)[0]
ext = os.path.splitext(dir)[1]
if ext == ".py":
import( eval(name) )
reload( eval(name) )
Anyone know how to do this correctly?
import os # we use os.path.join, os.path.basename
import sys # we use sys.path
import glob # we use glob.glob
import importlib # we use importlib.import_module
import_folder = 'C:\\myFolder'
sys.path.append(import_folder) # this tells python to look in `import_folder` for imports
for src_file in glob.glob(os.path.join(import_folder, '*.py')):
name = os.path.basename(src_file)[:-3]
importlib.import_module(name)
reload(sys.modules[name])
importlib.import_module(name)
There is the code. Now to the semantics of the whole thing: using importlib makes this a little bit more normal, but it still promotes some bugs. You can see that this breaks for source files in a subdirectory. What you should probably do is: import the package, (import the whole folder), and use the . operator like so:
import sys # we use sys.path
sys.path.append('C:\\')
import myFolder
...
myFolder.func1(foo)
myFolder.val
bar = myFolder.Class1()
Perhaps you should take a look at the documentation for modules, but don't forget to update the path to include the parent of the folder you want to import.