My project structure is the following. Inside api.py i need some functions written in the upper level.
Project1
├── model.py
├── audio_utils.py
├── audio.py
└── backend
├── static
│ ├──js
│ ├──img
└── api.py
Why am I unable to import inside api.py the functions in the upper level?
When i try to do:
from audio_utils import *
I got the following:
No module named 'audio_utils'
Modules are imported from paths prefixes specified in sys.path. It usually contains '' that means that modules from current working directory are gonna be loaded.
(https://docs.python.org/3/tutorial/modules.html#packages)
I think you are starting your Python interpret while being in the backend directory. Then I think there is no way to access the modules in the upper directory -- not even with the .. (https://realpython.com/absolute-vs-relative-python-imports/#syntax-and-practical-examples_1) unless you change the sys.path which would be a really messy solution.
I suggest you create __init__.py files to indicate that the directories containing them are Python packages:
Project1
├── model.py
├── audio_utils.py
├── audio.py
└── backend
|-- __init__.py
├── static
│ ├──js
│ ├──img
└── api.py
And always start the interpret from the Project1 dir. Doing so, you should be able to import any module like this:
import model
from backed import api
import audio_utils
no matter in which module in the Project1 you are writing this in. The current directory of the interpret will be tried.
Note there is also the PYTHONPATH env variable and that you can use to your advantage.
Note that for publishing your project it is encouraged to put all the modules in a package (in other words: don't put the modues to the top level). This is to help prevent name collisions. I think this may help you to understand: https://realpython.com/pypi-publish-python-package/
You have __init__.py files in both directories right?
Try from ..audio_utils import *
If you create the dir structure this way:
$ tree
.
├── bar
│ ├── den.py
│ └── __init__.py # This indicates the bar is python package.
└── baz.py
1 directory, 3 files
$ cat bar/den.py
import baz
Then in the dir containing the bar/ and baz.py (the top level) you can start the Python interpret and use the absolute imports:
In [1]: import bar.den
In [2]: import baz
In [3]: bar.den.baz
Out[3]: <module 'baz' from '/tmp/Project1/baz.py'>
As you can see, we were able to import bar.den which also could import the baz from the top-level.
Related
I always have the same problem and I finally want to get rid of it. My folder structure looks like this
project
├── scipts
│ └── folder
│ └── file.py
└── submodules
└── lab_devices
└── optical_devices
└── __init__.py
└── powermeter_driver.py
I now want to include the powermeter_driver.py in file.py. So what I do in the file.py is:
from submodules.lab_devices.optical_devices.powermeter_driver import PowermeterDriver
but this gives ModuleNotFoundError: No module named 'submodules'. I don't want to use
import sys
sys.path.insert(0, '../submodules')
Is there an easy workaround?
The imports will be resolved correctly if you run the script in the correct way, which is from the parent directory and using the -m switch. So you should cd in the parent folder, add __init__.py files as in:
project
├── scripts
└── __init__.py
│ └── folder
└── __init__.py
│ └── file.py
└── submodules
└── __init__.py
└── lab_devices
└── __init__.py
└── optical_devices
└── __init__.py
└── powermeter_driver.py
so that python knows these are packages then run
python -m scripts.folder.file # note no .py
In file.py you can then use the absolute import as you are cause submodules will be detected as a package. You should indeed avoid hacking the sys.path by all means.
You need to consider that if you write from submodules.... this is an absolute import. It means Python starts searching for the submodules in all directories in sys.path. Python usually adds your current working directory as first item to sys.path, so if you cd to your project directory and then run it as a module using python -m it could work.
Of course absolute imports suck if you have files in a relative location to each other. I've had similar issues and I've created an experimental, new import library ultraimport that allows to do file system based imports. It could solve your issue if you are willing to add a new library for this.
Instead of:
from submodules.lab_devices.optical_devices.powermeter_driver import PowermeterDriver
In file.py you would then write:
import ultraimport
PowermeterDriver = ultraimport('__dir__/../../submodules/lab_devices/optical_devices/powermeter_driver.py', 'PowermeterDriver')
The file path is relative to file.py and thus this will always work, no matter how you run your code or what is in sys.path.
One caveat when importing scripts like this is if they contain further relative imports. ultraimport has a builtin preprocessor to rewrite subsequent relative imports so they continue to work.
I'm a beginner and I'm not understanding something in the project folder structure.
I have this project:
.
└── convertersProject/
├── conftest.py -----------------> best practice if you're not using src structure
├── converters -----------------> module I'm trying to import/
│ ├── __init__.py
│ ├── binconverter.py
│ └── importester1.py --------> import binconverter OR from converters import */
│ └── submodule/
│ ├── __init__.py -----> not needed if I don't want to use it as package
│ └── importester2.py -> from converter import binconverter
├── outsidemodule /
│ ├── importester3.py
│ └── __init__.py
└── test /
└── test_converters.py
I'm receiving always ModuleNotErrorFound when trying to execute importester1/2/3.py directly from project folder. I'm using a virtual enviroment, setting up with python -m venv 'name' from a pyenv python 3.8.5 set with pyenv shell 3.8.5
What I think I'm understanding:
I have to use absolute paths as from converters.binconverter import bin2dec being bin2dec fucntion in binconverter. If I'd want to use relative, I should be inside the folder tree as trying to execute importertest2.py, because submodule is inside converters. So, I coudn't use relatives for outsidemodule
PYTHONPATH is the current folder you're executing, so if I executing from project folder as python converters/submodule/importester2.py, I don't have to append any value to the PYTHONPATH of the virtual enviroment (indeed, I've read it's not a good practice)
__init__.py allows you to use module without appending values to PYTHONPATH virtualenv, so I could import the converter module into outsidemodule using absolute paths.
tests are working using this logic. In fact if I change something the VSCode debugger detect import problems.
What the hell am I missing?
I'm using pyright for type checking and I'm also using pytest for testing inside Visual Studio Code. The folder structure for my tests is to have a 'test' subfolder in the package root . For example
|
MyPackage
|-- __init__.py
|-- MyModule.py
|--test
|-- __init__.py
|--MyModule_test.py
I'm organizing things like this as there will be many packages and I want to keep things organized.
Inside pytest I have
import pytest
import MyPackage.MyModule
...
Pytest is able to discover the tests and run them OK because it has some special ability to adjust its sys.path (or something).
However, pyright will just complain that it cannot import the module,
Import 'MyPackage.MyModule' could not be resolvedpyright (reportMissingImports). This makes sense, but is there some way to deal with this, either in pyright or in the Visual Studio Code settings to stop this from complaining?
You can add the library path to the path variable.
import sys
sys.path.insert(1, str('..'))
import MyModule
To enable Pylance to use your library properly (for auto-complete ...), use the following steps:
Pylance, by default, includes the root path of your workspace. If you want to include other subdirectories as import resolution paths, you can add them using the python.analysis.extraPaths setting for the workspace.
In VS Code press +<,> to open Settings.
Type in python.analysis.extraPaths
Select "Add Item"
Type in the path to your library `..'
Ok, a relative import as illustrated here was able to solve this. So in my case I should have
# MyModule_test.py
import pytest
from .. import MyModule
You should create a pyrightconfig.json file or pyproject.toml file at the root of your project. For example, if it's a Django project, you should have one of those files where manage.py is placed. Then, set include parameter and add the subdirectories (or app folders in Django terms).
You can consult this sample config file. See this issue ticket.
For example, if this were my project structure:
├── manage.py
├── movie
│ ├── admin.py
│ ├── apps.py
│ ├── __init__.py
│ ├── models.py
│ ├── tests.py
│ └── views.py
├── moviereviews
│ ├── asgi.py
│ ├── __init__.py
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
└── pyproject.toml
my pyproject.toml would be:
[tool.pyright]
include = ["movie", "moviereviews"]
If you are working within a Python virtual environment, set venvPath and venv. Consult the documentation for an exhaustive list of options.
My current way of importing scripts I've written looks like this:
import sys
sys.path.append('../../')
from lib.helper import file_manager
because my folder structure looks like this:
|
├── lib
│ ├── upm
│ └── helper
| └── file_manager.py <- the imported file.
│
└── src
└── deployment
└── cli.py <- the executed file.
So when cli.py runs it has to append the root folder onto the sys.path before it can include anything from lib. If I move cli.py around, I have to change this line to reflect how nested it is: sys.path.append('../../')
There has got to be a better way to import modules that are in the same project. Because of this bad pattern, I find myself fighting with the current working directory of my project way too much.
What would you do if you were me?
I would like to include a third party library into my Python script folder to distribute it all togehter (I am awary of the distribution license, and this library is fine to distribute). This is in order to avoid installing the library on another machine.
Saying I have a script (my_script.py), which calls this external library. I tried to copy this library from the site-packages subdirectory of Python directory into the directory where I have my files, but it seems not to be enough (I think th reason is in the __init__.py of this library which probably needs the folder to be in the PYTHONPATH).
Would it be reasonable to insert some lines of code in my_script.py to temporary append its folder to sys.path in order to make the all things working?
For instance, if I have a structure similar to this:
Main_folder
my_script.py
/external_lib_folder
__init__.py
external_lib.py
and external_lib_folder is the external library I copied from site-packages and inserted in my Main_folder, would it be fine if I write these lines (e.g.) in my_script.py?
import os,sys
main_dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(main_dir)
EDIT
I ended up choosing the sys.path.append solution. I added these lines to my my_script.py:
import os, sys
# temporarily appends the folder containing this file into sys.path
main_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)),'functions')
sys.path.append(main_dir)
Anyway, I chose to insert this as an edit in my question and accept the answer of Torxed because of the time he spent in helping me (and of course because his solution works as well).
Python3
import importlib.machinery, imp
namespace = 'external_lib'
loader = importlib.machinery.SourceFileLoader(namespace, '/home/user/external_lib_folder/external_lib.py')
external_lib = loader.load_module(namespace)
# How to use it:
external_lib.function(data_or_something)
This would be an ideal way to load custom paths in Python 3.
Not entirely sure this is what you wanted but It's relevant enough to post an alternative to adding to sys.path.
Python2
In python 2 you could just do (if i'm not mistaken, been a while since i used an older version of Python):
external_lib = __import__('external_lib_folder')
This does however require you to keep the __init__.py and a proper declaration of functions in sad script, otherwise it will fail.
**It's also important that the folder you're trying to import from is of the same name that the __init__.py script in sad folder is trying to import it's sub-libraries from, for instance geopy would be:
./myscript.py
./geopy/
./geopy/__init__.py
./geopy/compat.py
...
And the code of myscript.py would look like this:
handle = __import__('geopy')
print(handle)
Which would produce the following output:
[user#machine project]$ python2 myscript.py
<module 'geopy' from '/home/user/project/geopy/__init__.pyc'>
[user#machine project]$ tree -L 2
.
├── geopy
│ ├── compat.py
│ ├── compat.pyc
│ ├── distance.py
│ ├── distance.pyc
│ ├── exc.py
│ ├── exc.pyc
│ ├── format.py
│ ├── format.pyc
│ ├── geocoders
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── location.py
│ ├── location.pyc
│ ├── point.py
│ ├── point.pyc
│ ├── units.py
│ ├── units.pyc
│ ├── util.py
│ ├── util.pyc
│ └── version.pyc
└── myscript.py
2 directories, 20 files
Because in __init__.py of geopy, it's defined imports such as from geopy.point import Point which requires a namespace or a folder of geopy to be present.
There for you can't rename the folder to functions and place a folder called geopy in there because that won't work, nor will placing the contents of geopy in a folder called functions because that's not what geopy will look for.
Adding the path to sys.path (Py2 + 3)
As discussed in the comments, you can also add the folder to your sys.path variable prior to imports.
import sys
sys.path.insert(0, './functions')
import geopy
print(geopy)
>>> <module 'geopy' from './functions/geopy/__init__.pyc'>
Why this is a bad idea: It will work, and is used by many. The problems that can occur is that you might replace system functions or other modules might get loaded from other folders if you're not careful where you import stuff from. There for use .insert(0, ...) for most and be sure you actually want to risk replacing system built-ins with "shady" path names.
What you suggest is bad practice, it is a weak arrangement. The best solution (which is also easy to do) is to package it properly and add an explicit dependency, like this:
from setuptools import setup
setup(name='funniest',
version='0.1',
description='The funniest joke in the world',
url='http://github.com/storborg/funniest',
author='Flying Circus',
author_email='flyingcircus#example.com',
license='MIT',
packages=['funniest'],
install_requires=[
'markdown',
],
zip_safe=False)
This will work if the third party library is on pipy. If it's not, use this:
setup(
...
dependency_links=['http://github.com/user/repo/tarball/master#egg=package-1.0']
...
)
(See this explanation for packaging).