We Keep Coding

How can I use relative imports in Python to import a function in another directory

I have a directory structure with 2 basic python files inside seperate directories:
├── package
│ ├── subpackage1
│ │ └── module1.py
└── subpackage2
└── module2.py
module1.py:
def module1():
print('hello world')
module2.py:
from ..subpackage1.module1 import module1
module1()
When running python3 module2.py I get the error: ImportError: attempted relative import with no known parent package
However when I run it with the imports changed to use sys.path.append() it runs successfully
import sys
sys.path.append('../subpackage1/')
from module1 import module1
module1()
Can anyone help me understand why this is and how to correct my code so that I can do this with relative imports?

To be considered a package, a Python directory has to include an __init__.py file. Since your module2.py file is not below a directory that contains an __init__.py file, it isn't considered to be part of a package. Relative imports only work inside packages.
UPDATE:
I only gave part of the answer you needed. Sorry about that. This business of running a file inside a package as a script is a bit of a can of worms. It's discussed pretty well in this SO question:
Relative imports in Python 3
The main take-away is that you're better off (and you're doing what Guido wants you to) if you don't do this at all, but rather move directly executable code outside of any module. You can usually do this by adding an extra file next to your package root dir that just imports the module you want to run.
Here's how to do that with your setup:
.
├── package
│   ├── __init__.py
│   ├── subpackage1
│   │   └── module1.py
│   └── subpackage2
│   └── module2.py
└── test.py
test.py:
import package.subpackage2.module2
You then run test.py directly. Because the directory containing the executed script is included in sys.path, this will work regardless of what the working directory is when you run the script.
You can also do basically this same thing without changing any code (you don't need test.py) by running the "script" as a module.
python3 -m package.subpackage2.module2
If you have to make what you're trying to do work, I think I'd take this approach:
import os, sys
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from subpackage1.module1 import module1
module1()
So you compute in a relative way where the root of the enclosing package is in the filesystem, you add that to the Python path, and then you use an absolute import rather than a relative import.
There are other solutions that involve extra tools and/or installation steps. I can't think why you could possibly prefer those solutions to the last solution I show.

By default, Python just considers a directory with code in it to be a directory with code in it, not a package/subpackage. In order to make it into a package, you'll need to add an __init__.py file to each one, as well as an __init__.py file to within the main package directory.

Even adding the __init__.py files won't be enough, but you should. You should also create a setup.py file next to your package directory. Your file tree would look like this:
├── setup.py
└── package
├── __init__.py
└── subpackage1
│ ├── __init__.py
│ └── module1.py
└── subpackage2
├── __init__.py
└── module2.py
This setup.py file could start off like this:
from setuptools import setup
setup(
name='package',
packages=['package'],
)
These configurations are enough to get you started. Then, on the root of your directory (parent folder to package and setup.py), you will execute next command in you terminal pip install -e . to install your package, named package, in development mode. Then you'll be able to navigate to package/subpackage2/ and execute python module2.py having your expected result. You could even execute python package/subpackage2/module2.py and it works.
The thing is, modules and packages don't work the same way they work in another programming languages. Without the creation of setup.py if you were to create a program in your root directory, named main.py for example, then you could import modules from inside package folder tree. But if you're looking to execute package\subpackage2\module2.py.

If you want relative imports without changing your directory structure and without adding a lot of boilerplate you could use my import library: ultraimport
It gives the programmer more control over their imports and lets you do file system based relative or absolute imports.
Your module2.py could then look like this:
import ultraimport
module1 = ultraimport('__dir__/../subpackage1/module1.py')
This will always work, no matter how you run your code or if you have any init files and independent of sys.path.

ImportError when using pytest in terminal [duplicate]

I used easy_install to install pytest on a Mac and started writing tests for a project with a file structure likes so:
repo/
|--app.py
|--settings.py
|--models.py
|--tests/
|--test_app.py
Run py.test while in the repo directory, and everything behaves as you would expect.
But when I try that same thing on either Linux or Windows (both have pytest 2.2.3 on them), it barks whenever it hits its first import of something from my application path. For instance, from app import some_def_in_app.
Do I need to be editing my PATH to run py.test on these systems?

I'm not sure why py.test does not add the current directory in the PYTHONPATH itself, but here's a workaround (to be executed from the root of your repository):
python -m pytest tests/
It works because Python adds the current directory in the PYTHONPATH for you.

Recommended approach for pytest>=7: use the pythonpath setting
Recently, pytest has added a new core plugin that supports sys.path modifications via the pythonpath configuration value. The solution is thus much simpler now and doesn't require any workarounds anymore:
pyproject.toml example:
[tool.pytest.ini_options]
pythonpath = [
"."
]
pytest.ini example:
[pytest]
pythonpath = .
The path entries are calculated relative to the rootdir, thus . adds repo directory to sys.path in this case.
Multiple path entries are also allowed: for a layout
repo/
├── src/
| └── lib.py
├── app.py
└── tests
├── test_app.py
└── test_lib.py
the configuration
[tool.pytest.ini_options]
pythonpath = [
".", "src",
]
or
[pytest]
pythonpath = . src
will add both app and lib modules to sys.path, so
import app
import lib
will both work.
Original answer (not recommended for recent pytest versions; use for pytest<7 only): conftest solution
The least invasive solution is adding an empty file named conftest.py in the repo/ directory:
$ touch repo/conftest.py
That's it. No need to write custom code for mangling the sys.path or remember to drag PYTHONPATH along, or placing __init__.py into dirs where it doesn't belong (using python -m pytest as suggested in Apteryx's answer is a good solution though!).
The project directory afterwards:
repo
├── conftest.py
├── app.py
├── settings.py
├── models.py
└── tests
└── test_app.py
Explanation
pytest looks for the conftest modules on test collection to gather custom hooks and fixtures, and in order to import the custom objects from them, pytest adds the parent directory of the conftest.py to the sys.path (in this case the repo directory).
Other project structures
If you have other project structure, place the conftest.py in the package root dir (the one that contains packages but is not a package itself, so does not contain an __init__.py), for example:
repo
├── conftest.py
├── spam
│ ├── __init__.py
│ ├── bacon.py
│ └── egg.py
├── eggs
│ ├── __init__.py
│ └── sausage.py
└── tests
├── test_bacon.py
└── test_egg.py
src layout
Although this approach can be used with the src layout (place conftest.py in the src dir):
repo
├── src
│ ├── conftest.py
│ ├── spam
│ │ ├── __init__.py
│ │ ├── bacon.py
│ │ └── egg.py
│ └── eggs
│ ├── __init__.py
│ └── sausage.py
└── tests
├── test_bacon.py
└── test_egg.py
beware that adding src to PYTHONPATH mitigates the meaning and benefits of the src layout! You will end up with testing the code from repository and not the installed package. If you need to do it, maybe you don't need the src dir at all.
Where to go from here
Of course, conftest modules are not just some files to help the source code discovery; it's where all the project-specific enhancements of the pytest framework and the customization of your test suite happen. pytest has a lot of information on conftest modules scattered throughout their docs; start with conftest.py: local per-directory plugins
Also, SO has an excellent question on conftest modules: In py.test, what is the use of conftest.py files?

I had the same problem. I fixed it by adding an empty __init__.py file to my tests directory.

Yes, the source folder is not in Python's path if you cd to the tests directory.
You have two choices:
Add the path manually to the test files. Something like this:
import sys, os
myPath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, myPath + '/../')
Run the tests with the env var PYTHONPATH=../.

Run pytest itself as a module with:
python -m pytest tests
This happens when the project hierarchy is, for example, package/src package/tests and in tests you import from src. Executing as a module will consider imports as absolute rather than relative to the execution location.

You can run with PYTHONPATH in project root
PYTHONPATH=. py.test
Or use pip install as editable import
pip install -e . # install package using setup.py in editable mode

I had the same problem in Flask.
When I added:
__init__.py
to the tests folder, the problem disappeared :)
Probably the application couldn't recognize folder tests as a module.

I created this as an answer to your question and my own confusion. I hope it helps. Pay attention to PYTHONPATH in both the py.test command line and in the tox.ini.
https://github.com/jeffmacdonald/pytest_test
Specifically: You have to tell py.test and tox where to find the modules you are including.
With py.test you can do this:
PYTHONPATH=. py.test
And with tox, add this to your tox.ini:
[testenv]
deps= -r{toxinidir}/requirements.txt
commands=py.test
setenv =
PYTHONPATH = {toxinidir}

I fixed it by removing the top-level __init__.py in the parent folder of my sources.

I started getting weird ConftestImportFailure: ImportError('No module named ... errors when I had accidentally added __init__.py file to my src directory (which was not supposed to be a Python package, just a container of all source).

It is a bit of a shame that this is an issue in Python... But just adding this environment variable is the most comfortable way, IMO:
export PYTHONPATH=$PYTHONPATH:.
You can put this line in you .zshrc or .bashrc file.

I was having the same problem when following the Flask tutorial and I found the answer on the official Pytest documentation.
It's a little shift from the way I (and I think many others) are used to do things.
You have to create a setup.py file in your project's root directory with at least the following two lines:
from setuptools import setup, find_packages
setup(name="PACKAGENAME", packages=find_packages())
where PACKAGENAME is your app's name. Then you have to install it with pip:
pip install -e .
The -e flag tells pip to install the package in editable or "develop" mode. So the next time you run pytest it should find your app in the standard PYTHONPATH.

I had a similar issue. pytest did not recognize a module installed in the environment I was working in.
I resolved it by also installing pytest into the same environment.

Also if you run pytest within your virtual environment make sure pytest module is installed within your virtual environment. Activate your virtual environment and run pip install pytest.

For me the problem was tests.py generated by Django along with tests directory. Removing tests.py solved the problem.

I got this error as I used relative imports incorrectly. In the OP example, test_app.py should import functions using e.g.
from repo.app import *
However liberally __init__.py files are scattered around the file structure, this does not work and creates the kind of ImportError seen unless the files and test files are in the same directory.
from app import *
Here's an example of what I had to do with one of my projects:
Here’s my project structure:
microbit/
microbit/activity_indicator/activity_indicator.py
microbit/tests/test_activity_indicator.py
To be able to access activity_indicator.py from test_activity_indicator.py I needed to:
start test_activity_indicatory.py with the correct relative import:
from microbit.activity_indicator.activity_indicator import *
put __init__.py files throughout the project structure:
microbit/
microbit/__init__.py
microbit/activity_indicator/__init__.py
microbit/activity_indicator/activity_indicator.py
microbit/tests/__init__.py
microbit/tests/test_activity_indicator.py

According to a post on Medium by Dirk Avery (and supported by my personal experience) if you're using a virtual environment for your project then you can't use a system-wide install of pytest; you have to install it in the virtual environment and use that install.
In particular, if you have it installed in both places then simply running the pytest command won't work because it will be using the system install. As the other answers have described, one simple solution is to run python -m pytest instead of pytest; this works because it uses the environment's version of pytest. Alternatively, you can just uninstall the system's version of pytest; after reactivating the virtual environment the pytest command should work.

I was getting this error due to something even simpler (you could even say trivial). I hadn't installed the pytest module. So a simple apt install python-pytest fixed it for me.
'pytest' would have been listed in setup.py as a test dependency. Make sure you install the test requirements as well.

Since no one has suggested it, you could also pass the path to the tests in your pytest.ini file:
[pytest]
...
testpaths = repo/tests
See documentation: https://docs.pytest.org/en/6.2.x/customize.html#pytest-ini
Side effect for Visual Studio Code: it should pick up the unit test in the UI.

We have fixed the issue by adding the following environment variable.
PYTHONPATH=${PYTHONPATH}:${PWD}/src:${PWD}/test

As pointed out by Luiz Lezcano Arialdi, the correct solution is to install your package as an editable package.
Since I am using Pipenv, I thought about adding to his answer a step-by-step how to install the current path as an edible with Pipenv, allowing to run pytest without the need of any mangling code or lose files.
You will need to have the following minimal folder structure (documentation):
package/
package/
__init__.py
module.py
tests/
module_test.py
setup.py
setup.py mostly has the following minium code (documentation):
import setuptools
setuptools.setup(name='package', # Change to your package name
packages=setuptools.find_packages())
Then you just need to run pipenv install --dev -e . and Pipenv will install the current path as an editable package (the --dev flag is optional) (documentation).
Now you should be able to run pytest without problems.

If this pytest error appears not for your own package, but for a Git-installed package in your package's requirements.txt, the solution is to switch to editable installation mode.
For example, suppose your package's requirements.txt had the following line:
git+https://github.com/foo/bar.git
You would instead replace it with the following:
-e git+https://github.com/foo/bar.git#egg=bar

If nothing works, make sure your test_module.py is listed under the correct src directory.
Sometimes it will give ModuleNotFoundError not because modules are misplaced or export PYTHONPATH="${PWD}:${PYTHONPATH}" is not working, its because test_module.py is placed into a wrong directory under the tests folder.
it should be 1-to-1 mapping relation recursively instead of the root folder should be named as "tests" and the name of the file that include test code should starts with "test_",
for example,
./nlu_service/models/transformers.py
./tests/models/test_transformers.py
This was my experience.

Very often the tests were interrupted due to module being unable to be imported.
After research, I found out that the system is looking at the file in the wrong place and we can easily overcome the problem by copying the file, containing the module, in the same folder as stated, in order to be properly imported.
Another solution proposal would be to change the declaration for the import and show MutPy the correct path of the unit. However, due to the fact that multiple units can have this dependency, meaning we need to commit changes also in their declarations, we prefer to simply move the unit to the folder.

My solution:
Create the conftest.py file in the test directory containing:
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + "/relative/path/to/code/")
This will add the folder of interest to the Python interpreter path without modifying every test file, setting environment variable or messing with absolute/relative paths.

Proper ways to set the path of my app in Python

I have a question in how to properly create a path in Python (Python 3.x).
I developed a small scraping app in Python with the following directory structure.
root
├── Dockerfile
├── README.md
├── tox.ini
├── src
│   └── myapp
│   ├── __init__.py
│   ├── do_something.py
│   └── do_something_else.py
└── tests
├── __init__.py
├── test_do_something.py
└── test_do_something_else.py
When I want to run my code, I can go to the src directory and do with
python do_something.py
But, because do_something.py has an import statement from do_something_else.py, it fails like:
Traceback (most recent call last):
File "src/myapp/do_something.py", line 1, in <module>
from src.myapp.do_something_else import do_it
ModuleNotFoundError: No module named 'src'
So, I eventually decided to use the following command to specify the python path:
PYTHONPATH=../../ python do_something.py
to make sure that the path is seen.
But, what are the better ways to feed the path so that my app can run?
I want to know this because when I run pytest via tox, the directory that I would run the command tox would be at the root so that tox.ini is seen by tox package. If I do that, then I most likely run into a similar problem due to the Python path not properly set.
Questions I want to ask specifically are:
where should I run my main code when creating my own project like this? root as like python src/myapp/do_something.py? Or, go to the src/myapp directory and run like python do_something.py?
once, the directory where I should execute my program is determined, what is the correct way to import modules from other py file? Is it ok to use from src.myapp.do_something_else import do_it (this means I must add path from src directory)? Or, different way to import?
What are ways I can have my Python recognize the path? I am aware there are several ways to make the pass accessible as below:
a. write export PYTHONPATH=<path_of_my_choice>:$PYTHONPATH to make the
path accessible temporarily, or write that line in my .bashrc to make it permanent (but it's hard to reproduce when I want to automate creating Python environment via ansible or other automation tools)
b. write import sys; sys.path.append(<root>) to have the root as an accessible path
c. use pytest-pythonpath package (but this is not really a generic answer)
Thank you so much for your inputs!
my environment
OS: MacOS and Amazon Linux 2
Python Version: 3.7
Dependency in Python: pytest, tox

I would suggest to use setup.py to make this a python package. Then you can install it in development mode python setup.py develop. This way it will be available in your python environment w/o needing to specify the PYTHONPATH.
For testing, you can simply install the package python setup.py install.
Hope that helps.

Two simple steps should make it happen. Python experts can comment if this is a good way to do it (especially going by the concluding caution raised towards the end of this post).
I would have done it like below.
First I would have put a "__init__.py" in root so that hierarchy looks like below. This way python will treat the folder as a package.
root
├── Dockerfile
├── README.md
├── tox.ini
├── __init__.py
├── src
│ └── myapp
│ ├── __init__.py
│ ├── do_something.py
│ └── do_something_else.py
└── tests
├── __init__.py
├── test_do_something.py
└── test_do_something_else.py
Then in "do_something.py", I would have added these lines at the top. In the second line please put the full path to the "root" directory.
import sys
sys.path += ['/home/SomeUserName/SomeFolderPath/root']
from src.myapp.do_something_else import do_it
Please note that the second line will essentially modify the sys.path by adding the root folder path (I guess until the interpreter quits). If this is not what you can afford then I am sorry.

Python import from sibling directories

Disclaimer: after searching through tons of very similar feeds that in the end all turn out to solve a slightly different problem I guess I have to open a new question (although I am sure there exists an answer somewhere --> so point that out if you know it ;)
The problem: I am using Python 2, am building a project with this tree:
project
├── __init__.py
├── foo
│   └── __init__.py
│   └── bar
│   └── __init__.py
├── notebooks
│ └── __init__.py
│ └── skript.py
└── test
└── __init__.py
└── foo
└── __init__.py
└── bar
└── __init__.py
└── file.py
Now I want to load test.foo.bar from within project/notebooks/skript.py. Therefore, I do in that skript
import sys
sys.path.append('../')
If I then run
import test.foo.bar # or: import test.foo
python tells me
ImportError: No module named foo.bar
(or ImportError: No module named foo respectively). Funily, import test does not throw an error, but if I then do test.foo it throws an AttributeError: 'module' object has no attribute 'foo'.
So I wonder, what is going wrong here and how to fix it?
Edit
Also, I tried adding this to skript.py
import sys
import os
MYDIR = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.join(MYDIR,'../test'))
sys.path.append(os.path.join(MYDIR,'../test/foo'))
sys.path.append(os.path.join(MYDIR,'../test/foo/bar')) #I am not sure this is entirely needed
as was pointed out below. Still,
import test.foo.bar.file
or
from test.foo.bar import file
just yield
ImportError: No module named foo.bar
Same for
sys.path.append('../test/foo/bar')
import test.foo.bar.file
I have still no clue whats going wrong?

Messing with sys.path is rarely a good idea.
Since your plan seems to be to use both foo and test from notebooks (that will probably just contain Jupyter notebooks), the cleanest solution would be to install foo and test as packages.
Remove the __init__.py from your top level directory and notebooks, since you will not want to import them. Then add a setup.py to your top level directory. Since your tests are specific to foo, you should either rename them foo_test or move them into foo itself.
A minimal setup.py would look like this
from setuptools import setup
setup(name='foo',
version='0.1',
description='descroption of fo',
author='you',
author_email='your#mail',
packages=['foo','test_foo])
Then you can simply pip install -e . in your top level directory and it will be installed into your current virtualenv. If you are not using virtualenvs, you should pip install --user -e .

It should work with
from test import foo
But you have to add a __init__.py to your project directory.
For Python 3 it would be:
from .test import foo
If you use the dot in front of the folder name, python searches for the file in the same directory as the file you are working on is placed in.
Sorry for my bad english.

Are you using an IDE? If so add the path to the Python Interpreter inside the project Properties to all the primary packages (foo, test, notebooks). Otherwise try to explicitly add the bar package to the sys path like so
import sys
import os
MYDIR = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.join(MYDIR,'test'))
sys.path.append(os.path.join(MYDIR,'test/foo'))
sys.path.append(os.path.join(MYDIR,'test/foo/bar')) #I am not sure this is entirely needed

How to use a packed python package without installing it

I have a python 3 package with the following structure:
.
├── package
│ └── bin
└── main_module
│ └── lib
│ ├── __init__.py
│ ├── module1.py
│ ├── module2.py
│ └── module3.py
│ └── test
│ ├── test1.py
│ ├── test2.py
│ └── test3.py
│ └── setup.py
Usually, one runs $ python3 setup.py install and all is good. However, I want to use this package on a cluster server, where I don't have write permissions for /usr/lib/. The following solutions came to my mind.
Somehow install the package locally in my user folder.
Modify the package such that it runs without installation.
Ask the IT guys to install the package for me.
I want to avoid 3., so my question is whether 1. is possible and if not, how I have to modify the code (particularly the imports) in order to be able to use the package without installation. I have been reading about relative imports in python all morning and I am now even more confused than before. I added __init__.py to package and bin and from what I read I assumed it has to be from package.lib import module1, but I always get ImportError: No module named lib.

In order for Python to be able to find your modules you need to add the path of your package to sys.path list. As a general way you can use following snippet:
from sys import path as syspath
from os import path as ospath
syspath.append(ospath.join(ospath.expanduser("~"), 'package_path_from_home'))
os.path.expanduser("~") will give you the path of the home directory and you can join it with the path of your package using os.path.join and then append the final path to sys.path.
If package is in home directory you can just add the following at the leading of your python code that is supposed to use this package:
syspath.append(ospath.join(ospath.expanduser("~"), 'package'))
Also make sure that you have an __init__.py in all your modules.

I had the same problem. I used the first approach
install the package locally in my user folder by running
python setup.py install --user
This will install your module in ~/.local/lib/python3/

Just add the path of your 'package' to environment variable PYTHONPATH. This will get rid of the error you are getting.
OR
programmatically add path of the package to sys.path.append()

you can add this to the "main file" of the package
import sys, os
sys.path.append(os.path.dirname(__file__) + "/..")
you can find the "main file" by looking for this pattern
if __name__ == "__main__":
some_function()