My directory layout is as follows
project\
project\setup.py
project\scripts\foo.py
project\scripts\bar.py
project\scripts\__init__.py
project\tests\test_foo.py
project\tests\__init__.py
My test file looks as follows
project\tests\test_fo.py
from ..scripts import foo
def test_one():
assert 0
I get the following error, when I do
cd C:\project
C:\virtualenvs\test_env\Scripts\activate
python setup.py install
python setup.py test
E ValueError: Attempted relative import beyond toplevel package
What am I doing wrong?
This is my setup.py
setup(
name = 'project',
setup_requires=['pytest-runner'],
tests_require=['pytest'],
packages = ["scripts","tests"],
package_data={
'scripts': ['*.py'],
'tests': ['*.py'],
},
)
Relative imports only work within a package. scripts may be a package, and so is tests, but the project directory is not (and neither should it be). This makes scripts and tests top-level packages. You can't refer to other top-level names using relative syntax.
Moreover, tests are not run with the tests package; the test runner imports the test_foo module, not the tests.test_foo module, so as far as Python is concerned test_foo is a top-level module.
scripts is a top-level name, just use that directly. You will have to add the project directory to sys.path however. You can do so at the top of your test_foo.py file with:
import os
import sys
TEST_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_DIR = os.path.abspath(os.path.join(TEST_DIR, os.pardir))
sys.path.insert(0, PROJECT_DIR)
then import from scripts with absolute paths:
from scripts import foo
Note however that when you run python setup.py then your current working directory is added to sys.path anyway, so scripts is available directly without having to fiddle with sys.path.
Moreover, pytest will already do the work for you; for any given test file it'll make sure the first parent directory with no __init__.py file in it is on sys.path. In your case that's the project/ directory, so again scripts is directly available to import from. See Good Practices:
If pytest finds a “a/b/test_module.py” test file while recursing into the filesystem it determines the import name as follows:
determine basedir: this is the first “upward” (towards the root) directory not containing an __init__.py. If e.g. both a and b contain an __init__.py file then the parent directory of a will become the basedir.
perform sys.path.insert(0, basedir) to make the test module importable under the fully qualified import name.
import a.b.test_module where the path is determined by converting path separators / into ”.” characters. This means you must follow the convention of having directory and file names map directly to the import names.
Note that in order to actually use pytest to run your tests when you use setup.py test, you need to register an alias in your setup.cfg file (create it in project/ if you do not have one):
[aliases]
test = pytest
Related
I've a directory structure like this below:
ParentPackage/
childPackage/
__init__.py
subpackageOne/
__init__.py
moduleA.py
moduleB.py
test/
__init__.py
test.py
I'm trying to import a method "methodX" from moduleA.py to my test.py residing inside test folder.
below is the code which I've tried
import os
import sys
sys.path.append(os.path.realpath('..'))
from ..subpackageOne.moduleAimport methodX
but getting error "ImportError: attempted relative import with no known parent package".
In addition to this moduleA.py is importing moduleB.py and while running the test I am getting No module named 'moduleB'.
There are already lot of question has been asked on this topic, however suggestions provided for those didn't work for me.
Based on your architecture, you can do that:
import os
import sys
# Assuming your package is on a drive called C:.
# That is what I mean by absolute path, it needs to refer the path from the disk
# so it does not depend on where you are
sys.path.append("C:/path/to/childPackage")
# Absolute import is just importing wihtout using dots at the beginning.
# In this case, python will look for installed packages in your environment
# and then any folder that is in your path.
from subpackageOne.moduleA import methodX
If you prefer, you may want to add the parentPackage in your path so you can use any childPackage in case you have several.
sys.path.append("C:/path/to/parentPackage")
from childPackage.subpackageOne.moduleA import methodX
Note: I see you have __init__ files in your architecture which suggests you may want to create an installable package that you can build after (using a setup.py or setup.cfg file). In this case, you could install your package in your virtual environment and then you would not need to add the directory to your path. setuptools Doc or some tutorial
There're a lot of threads on importing modules from sibling directories, and majority recommends to either simply add init.py to source tree, or modify sys.path from inside those init files.
Suppose I have following project structure:
project_root/
__init__.py
wrappers/
__init__.py
wrapper1.py
wrapper2.py
samples/
__init__.py
sample1.py
sample2.py
All init.py files contain code which inserts absolute path to project_root/ directory into the sys.path. I get "No module names x", no matter how I'm trying to import wrapperX modules into sampleX. And when I try to print sys.path from sampleX, it appears that it does not contain path to project_root.
So how do I use init.py correctly to set up project environment variables?
Do not run sampleX.py directly, execute as module instead:
# (in project root directory)
python -m samples.sample1
This way you do not need to fiddle with sys.path at all (which is generally discouraged). It also makes it much easier to use the samples/ package as a library later on.
Oh, and init.py is not run because it only gets run/imported (which is more or less the same thing) if you import the samples package, not if you run an individual file as script.
I'm having problems with pytest not including my projects rootdir in sys.path list. Instead it is including the directory where the tests are located by default.
Here is my project structure.
proj/
setup.py
mypackage/
__init__.py
a.py
tests/
test_a.py
when running pytest with
py.test proj/mypackage/tests
it inserts proj/mypackage/tests into sys.path which is not good because now I cannot import a since its not relative to the tests directory.
I've noticed that py.test detects a rootdir, this is recognized as the root of my project which is the proj directory. This is what I want in sys.path during tests so all my code is relative to that directory. How do I ensure py.test includes (...)/proj/ in sys.path so that I can import mypackage.a in test_a.py.
I use a conftest.py that appends the package path (relative to the conftest.py file) to sys.path so the package can be imported. Something like this:
# contents of conftest.py
import sys
from os.path import abspath, dirname
package_path = abspath(dirname(dirname(__file__)))
sys.path.insert(0, package_path)
You would then put this file in the tests directory. The conftest.py file will get run before any thing else by default, so the other tests will be able to import mypackage.
Another, probably better, approach is to add a setup.py and follow the advice given here.
I have a rather complex python build project with a PyQT GUI interface. The builder is supposed to accept configuration for a list of projects, which may differ from user to user, and then customize the build process for a selected project. As a result, parts of the builder project are versioned with the project to be built (project specific UI elements and controller files), with the rest being located in the builders directory. The issue I am running into is an overlap in namespacing for imports. For example:
The builder directory might look like:
Builder/
Controllers/
__init__.py
BuilderController1.py
BuilderController2.py
UI/
__init__.py
BuilderUI_1.py
BuilderUI_2.py
.../
__main__.py
The project directory might look like:
Project/
Project_Files/
Build/
Controllers/
__init__.py
ProjController1.py
ProjController2.py
UI/
__init__.py
ProjUI_1.py
ProjUI_2.py
.../
__init__.py
Project_setup.py
With code in __main__.py such as:
...
sys.path.append("../path/to/Project/Build")
projSetup = __import__("Project_setup)
projSetup.run()
...
And code in Project_setup.py such as:
...
from Controllers.ProjController1 import TestControl1
from Controllers.ProjController2 import TestControl2
def run():
registerStage(name = 'TestControl1', controller = TestControl1(mainApp.getController(), displayName='TestControl1'), displayName = 'Test Control Page 1', parent = None)
registerStage(name = 'TestControl2', controller = TestControl2(mainApp.getController(), displayName='TestControl2'), displayName = 'Test Control Page 2', parent = None)
...
The problem is that every time I run __main__.py, I get an error saying that:
ImportError: No module named Controllers.ProjController1
It seems that while searching the PYTHONPATH for the module ProjController1 the interpreter gets stranded in Builder/Controllers and then stops looking further down the path.
Is this reasonable behavior in python? Is there a method to obtain the same results that circumvents these issues?
I have tried using the various flavors of python importing ( import, __import__, imp, and importlib). Additionally it will work if I rename Project/Build/Controllers and Project/Build/UI, but given the possible complexity of these file structures I would rather not be stuck giving unique names to everything.
Relative paths are reliant on your current working directory. Try the following for your sys path append:
import os
dir_ = os.path.dirname(__file__)
sys.path.append(os.path.join(dir_, "/path/to/Project/Build"))
It's really better to make a package, then import via the fully qualified path. In your root project dir, put a file setup.py with contents similar to:
#!/usr/bin/env python
from setuptools import setup
from setuptools import find_packages
setup(
name='Name',
version='1.0',
packages=find_packages(exclude=('tests',)),
cmdclass={
},
package_data={
},
)
Then building and installing the package like so:
python setup.py build; python setup.py install
Perform the imports like so:
from project.hierarchy.module import Class
More information on creating packages is available in the Python docs.
Relative imports and modifying sys path have always caused me heartache.
I have the following folder structure;
myapp\
myapp\
__init__.py
tests\
test_myapp.py
and my pwd is
C:\Users\wwerner\programming\myapp\
I have the following test setup:
import sys
import pprint
def test_cool():
pprint.pprint(sys.path)
assert False
That produces the following paths:
['C:\\Users\\wwerner\\programming\\myapp\\tests',
'C:\\Users\\wwerner\\programming\\envs\\myapp\\Scripts',
'C:\\Windows\\system32\\python34.zip',
'C:\\Python34\\DLLs',
'C:\\Python34\\lib',
'C:\\Python34',
'C:\\Users\\wwerner\\programming\\envs\\myapp',
'C:\\Users\\wwerner\\programming\\envs\\myapp\\lib\\site-packages']
And when I try to import myapp I get the following error:
ImportError: No module named 'myapp'
So it looks like it's not adding the current directory to my path.
By changing my import line to look like this:
import sys
sys.path.insert(0, '.')
import myapp
I am then able to import myapp with no problems.
Why does my current directory not show up in the path when running pytest? Is my only workaround to insert . into the sys.path? (I'm using Python 3.4 if it matters)
Ahah!
After comparing the layout of my cookiecutter repo, it turns out to be way more simple (and better) than that.
tests/
__init__.py
test_myapp.py
A simple addition of the __init__.py file to my test dir allows me to run py.test from my main directory.
Using an installable package
If you have an installable package (setup.py or pyproject.toml file with a build-system defined) then you want to test against the installed package.
pip install --editable .
pytest
The simplest possible way to make the project shown in the question into an installable package would be by adding this setup.py:
from setuptools import setup
setup(
name="myapp",
version="0.1",
packages=["myapp"],
)
This will put the myapp code at /path/to/myapp/.venv/lib/python3.XY/site-packages, which is in the sys.path of the virtual environment. Now myapp can be imported from the site-packages dir, just as it would be for a user installation. It is neither necessary nor desirable for the current working directory to be present on sys.path during test execution.
Not using an installable package
The project shown in the question does not have any installer, so it can't be installed. It can still be tested by making sure the project root (i.e. the directory which contains both myapp and tests as subdirectories) is present on sys.path.
The best way to do this is to use python -m pytest, rather than invoking the bare pytest command. When you use python -m pytest it adds the current working directory to the start of sys.path. That's the normal Python behavior when executing a package as __main__ (documented here) and it's also a documented usage for pytest - see Invoking pytest versus python -m pytest.
Why does adding an __init__.py to the tests subdirectory (not) work?
The directory structure shown in the question is the "Tests outside application code" pattern, documented here. This is also the directory structure I recommend, since it creates a clear distinction between library/application code and test code.
It's not recommended to add __init__.py files inside the test directories when using a "Tests outside application code" structure, since the test files aren't intended to be "packaged" (e.g. test files do not really need to import from other test files, and they do not need to be installed at all for end users of your package).
The reason adding a myapp/__init__.py actually allows myapp to be imported by pytest, as described in Wayne's answer is actually an accident due to the way test discovery appends sys.path during the test collection phase. This is described as "problematic" in the docs
... this introduces a subtle problem: in order to load the test modules from the tests directory, pytest prepends the root of the repository to sys.path, which adds the side-effect that now mypkg is also importable
They go on to strongly recommend using the src-layout if you intend to have __init__.py files inside test directories, to avoid this confusion of the import system.
But perhaps the best reason not to rely on this side-effect is that pytest collection actually can work in multiple modes (see import modes), and Wayne's answer relies upon pytest using the default "prepend" mode. It is currently mentioned that a future version will switch to "importlib" mode as default:
We intend to make importlib the default in future releases.
The accepted answer does not work with pytest --import-mode=importlib and so will stop working altogether at some stage.
sys.path automatically has the script's directory in it, and not the current working directory.
I am guessing that your script in placed in tests directory. Based on this assumption, your code should look like this:
import sys
import os
ROOT_DIR = os.path.dirname(os.path.dirname(__file__))
sys.path.append(ROOT_DIR)
import myapp # Should work now
Use the environment variable PYTHONPATH.
In Windows:
set PYTHONPATH=.
py.test
In Unix:
PYTHONPATH=. py.test