I always have the same problem and I finally want to get rid of it. My folder structure looks like this
project
├── scipts
│ └── folder
│ └── file.py
└── submodules
└── lab_devices
└── optical_devices
└── __init__.py
└── powermeter_driver.py
I now want to include the powermeter_driver.py in file.py. So what I do in the file.py is:
from submodules.lab_devices.optical_devices.powermeter_driver import PowermeterDriver
but this gives ModuleNotFoundError: No module named 'submodules'. I don't want to use
import sys
sys.path.insert(0, '../submodules')
Is there an easy workaround?
The imports will be resolved correctly if you run the script in the correct way, which is from the parent directory and using the -m switch. So you should cd in the parent folder, add __init__.py files as in:
project
├── scripts
└── __init__.py
│ └── folder
└── __init__.py
│ └── file.py
└── submodules
└── __init__.py
└── lab_devices
└── __init__.py
└── optical_devices
└── __init__.py
└── powermeter_driver.py
so that python knows these are packages then run
python -m scripts.folder.file # note no .py
In file.py you can then use the absolute import as you are cause submodules will be detected as a package. You should indeed avoid hacking the sys.path by all means.
You need to consider that if you write from submodules.... this is an absolute import. It means Python starts searching for the submodules in all directories in sys.path. Python usually adds your current working directory as first item to sys.path, so if you cd to your project directory and then run it as a module using python -m it could work.
Of course absolute imports suck if you have files in a relative location to each other. I've had similar issues and I've created an experimental, new import library ultraimport that allows to do file system based imports. It could solve your issue if you are willing to add a new library for this.
Instead of:
from submodules.lab_devices.optical_devices.powermeter_driver import PowermeterDriver
In file.py you would then write:
import ultraimport
PowermeterDriver = ultraimport('__dir__/../../submodules/lab_devices/optical_devices/powermeter_driver.py', 'PowermeterDriver')
The file path is relative to file.py and thus this will always work, no matter how you run your code or what is in sys.path.
One caveat when importing scripts like this is if they contain further relative imports. ultraimport has a builtin preprocessor to rewrite subsequent relative imports so they continue to work.
Related
I'm making a python library and i want to be able to run the code i'm developing, in another folder i have a python file, but I get the error: ModuleNotFoundError: No module named
this is my folder structure
.
└── project
└── library_directory
├── __init__.py
└── main.py
└── examples_directory
├── __init__.py
└── code_directory
├── __init__.py
└── test.py
init.py from library_directory
from library_directory.main import Class
test.py file
from library_directory import Class
when I run test.py file it says: ModuleNotFoundError: No module named 'fpdf_table'
if i put test.py file at project level this configuration of init and test works, but i want to run the test.py in the code_directory because i will have a lot of files and don't want 15+ single files at project level
.
└── project
└── library_directory
├── __init__.py
└── main.py
└── examples_directory
├── __init__.py
└── code_directory
├── __init__.py
└── test.py
i already tried absolute and relative imports but they don't work
Im not sure if this is the correct solution, but you can try this:
In your test module:
import sys
sys.path.insert(0, '/Your_project_root_path'
Now can access to the packages in the root directory.
I took this solution from here.
Your library_directory folder is a Python package. Within the package, each .py file is a module.
In your library_directory/init.py file, insert the line from .main import Class. Now you have a package called "library_directory", which has a module called "Main", and a class called "Class".
Now, in your environment variables, create a user variable called PYTHONPATH and add to this, the path to your project directory.
When you import in your project file, you should import using the structure: from package.module import class, or in your case: from library_directory.main import Class. The python interpreter will be able to find the package due to being in your PYTHONPATH and the init file directory means Python will recognise it as a package.
(You may wish to rename "library_directory" to be a bit more project specific)
I would like to integrate pytest into my workflow. I made a following folder structure:
myproject
├── venv
└── src
├── __init__.py
├── foo
│ ├── __init__.py
│ └── bar.py
└── tests
├── __init__.py
└── test_bar.py
I would like to be able to import the namespace from the foo package so that I can write test scripts in the tests folder. Whenever I try to run pytest, or pytest --import-mode append I always get the following error:
ModuleNotFoundError: No module named 'foo'
I found this similar question here but adding the __init__.py files to the tests and the src folder does not solve the issue.
Does this have to do with the PYTHONPATH system variable? This folder structure works perfectly if I run the __main__.py from the src folder, but fails when I want to use pytest. Is there a way to do this without having to mess with PYTHONPATH or are there automated ways to edit the system variable?
This question already has answers here:
How do you organise a python project that contains multiple packages so that each file in a package can still be run individually?
(3 answers)
Closed 2 years ago.
I am new to python. I have a project with the structure as described below and I just want to use functions from one module into another across the package1 and package2 directories. Can I use functions from module3.py in module1.py just by using the import statement (similar as in java or C#)
# in package1.module1.py
from package2 import module3.py
without any package installation (e.g. locally with 'pip install -e .') and setup.py configuration?
Here an example project structure
└── project
├── package1
│ ├── module1.py <- I want that this module uses functions from module3.py
│ └── module2.py
└── package2
├── __init__.py
├── module3.py
├── module4.py
└── subpackage1
└── module5.py
Yes, that should work, and you don't need an __init__.py unless you want to import the package itself, i.e.
import package2
The key thing though, is that 'project' needs to be in your PYTHONPATH. You will not notice that if you run python interactively, because I believe it automatically adds the current directory to the PYTHONPATH.
in practice you'd do
from package2 import module3 # notice, no .py
module3.somefn()
File tree:
/path/to/project/root/
├── .git/
├── Pipfile
├── Pipfile.lock
├── __init__.py
└── package1
├── __init__.py
├── src
│ ├── __init__.py
│ ├── external_lib
│ └── packageX
│ ├── packageY
│ └── packageZ
│ ├── foo.py
│ ├── bar.py
Constraints:
foo.py imports bar.py
both foo.py and bar.py import some files from external_lib/packageX/packageZ
files in external_lib/packageX/packageZ require packageX and packageY to be in the PYTHONPATH for their own includes.
I don't have a single entry point: sometimes I want to launch foo.py, sometimes bar.py.
external_lib is external, it's a git submodule. I don't want to change its content.
This is a shared project, I'd like a solution that's included in the git repository...
What I'm doing right now
I'm following the advice given here (in French though), that claims to be good practice:
Before running any script, I run cd package1/src/external_lib/packageX; export PYTHONPATH=$PYTHONPATH:pwd:pwd/packageY; cd ../../../..
All my imports are written like this: import package1.src.bar, import package1.src.external_lib.packageX.packageZ.some_module
I always call all scripts from the project root.
This gives something like this:
cd /path/to/project/root/
pipenv shell
cd package1/src/external_lib/packageX; export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/packageY; cd ../../../..
python package1/src/foo.py
python package1/src/bar.py
Question
What is the best practice, in terms of project architecture, Pipenv use, import writing, etc., to manage all the imports smoothly, with a minimum amount of manual actions and constraints (such as export PYTHONPATH=... commands before running scripts and launching the scripts from the root), in the most Pythonic and Pipenv-ic way ?
Ideally I'd also like Pycharm to be able to get all the imports correctly without further manual settings, so that what I do in Pycharm reflects what would happen from a terminal.
One possible solution I've seen is to use Pipenv's .env file, to add the export PYTHONPATH=... in it, and rely on its automatic loading. However, apparently this file is not meant for that and should not be commited.
I would like to include a third party library into my Python script folder to distribute it all togehter (I am awary of the distribution license, and this library is fine to distribute). This is in order to avoid installing the library on another machine.
Saying I have a script (my_script.py), which calls this external library. I tried to copy this library from the site-packages subdirectory of Python directory into the directory where I have my files, but it seems not to be enough (I think th reason is in the __init__.py of this library which probably needs the folder to be in the PYTHONPATH).
Would it be reasonable to insert some lines of code in my_script.py to temporary append its folder to sys.path in order to make the all things working?
For instance, if I have a structure similar to this:
Main_folder
my_script.py
/external_lib_folder
__init__.py
external_lib.py
and external_lib_folder is the external library I copied from site-packages and inserted in my Main_folder, would it be fine if I write these lines (e.g.) in my_script.py?
import os,sys
main_dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(main_dir)
EDIT
I ended up choosing the sys.path.append solution. I added these lines to my my_script.py:
import os, sys
# temporarily appends the folder containing this file into sys.path
main_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)),'functions')
sys.path.append(main_dir)
Anyway, I chose to insert this as an edit in my question and accept the answer of Torxed because of the time he spent in helping me (and of course because his solution works as well).
Python3
import importlib.machinery, imp
namespace = 'external_lib'
loader = importlib.machinery.SourceFileLoader(namespace, '/home/user/external_lib_folder/external_lib.py')
external_lib = loader.load_module(namespace)
# How to use it:
external_lib.function(data_or_something)
This would be an ideal way to load custom paths in Python 3.
Not entirely sure this is what you wanted but It's relevant enough to post an alternative to adding to sys.path.
Python2
In python 2 you could just do (if i'm not mistaken, been a while since i used an older version of Python):
external_lib = __import__('external_lib_folder')
This does however require you to keep the __init__.py and a proper declaration of functions in sad script, otherwise it will fail.
**It's also important that the folder you're trying to import from is of the same name that the __init__.py script in sad folder is trying to import it's sub-libraries from, for instance geopy would be:
./myscript.py
./geopy/
./geopy/__init__.py
./geopy/compat.py
...
And the code of myscript.py would look like this:
handle = __import__('geopy')
print(handle)
Which would produce the following output:
[user#machine project]$ python2 myscript.py
<module 'geopy' from '/home/user/project/geopy/__init__.pyc'>
[user#machine project]$ tree -L 2
.
├── geopy
│ ├── compat.py
│ ├── compat.pyc
│ ├── distance.py
│ ├── distance.pyc
│ ├── exc.py
│ ├── exc.pyc
│ ├── format.py
│ ├── format.pyc
│ ├── geocoders
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── location.py
│ ├── location.pyc
│ ├── point.py
│ ├── point.pyc
│ ├── units.py
│ ├── units.pyc
│ ├── util.py
│ ├── util.pyc
│ └── version.pyc
└── myscript.py
2 directories, 20 files
Because in __init__.py of geopy, it's defined imports such as from geopy.point import Point which requires a namespace or a folder of geopy to be present.
There for you can't rename the folder to functions and place a folder called geopy in there because that won't work, nor will placing the contents of geopy in a folder called functions because that's not what geopy will look for.
Adding the path to sys.path (Py2 + 3)
As discussed in the comments, you can also add the folder to your sys.path variable prior to imports.
import sys
sys.path.insert(0, './functions')
import geopy
print(geopy)
>>> <module 'geopy' from './functions/geopy/__init__.pyc'>
Why this is a bad idea: It will work, and is used by many. The problems that can occur is that you might replace system functions or other modules might get loaded from other folders if you're not careful where you import stuff from. There for use .insert(0, ...) for most and be sure you actually want to risk replacing system built-ins with "shady" path names.
What you suggest is bad practice, it is a weak arrangement. The best solution (which is also easy to do) is to package it properly and add an explicit dependency, like this:
from setuptools import setup
setup(name='funniest',
version='0.1',
description='The funniest joke in the world',
url='http://github.com/storborg/funniest',
author='Flying Circus',
author_email='flyingcircus#example.com',
license='MIT',
packages=['funniest'],
install_requires=[
'markdown',
],
zip_safe=False)
This will work if the third party library is on pipy. If it's not, use this:
setup(
...
dependency_links=['http://github.com/user/repo/tarball/master#egg=package-1.0']
...
)
(See this explanation for packaging).