I'm working on a project and would like to write some unit tests for my code. Here is a minimal example of my project structure
myproject
└── scripts
├── tests
│ └── test_func.py
└── tools
├── __init__.py
└── func.py
func.py contains
def f(x):
return 0
and test_func.py contains
import pytest
from ..tools import f
def test_f():
assert f(1)==0
When I try to run my tests with pytest pytest scripts/tests/test_func.py, I get the following error ImportError: attempted relative import with no known parent package.
Clearly, there is a problem with the structure of my project. There is a solution here which involves adding something to my path but I'd rather not do that if there is a way I can just structure my project to avoid this instead.
You need to add (empty) __init__.py files to the directories scripts/ and scripts/tests/. These files provide the scaffolding that lets import find modules, especially when using relative imports.
The __init__.py files are required to make Python treat directories containing the file as packages.
From https://docs.python.org/3/tutorial/modules.html
Related
I have a directory structure with 2 basic python files inside seperate directories:
├── package
│ ├── subpackage1
│ │ └── module1.py
└── subpackage2
└── module2.py
module1.py:
def module1():
print('hello world')
module2.py:
from ..subpackage1.module1 import module1
module1()
When running python3 module2.py I get the error: ImportError: attempted relative import with no known parent package
However when I run it with the imports changed to use sys.path.append() it runs successfully
import sys
sys.path.append('../subpackage1/')
from module1 import module1
module1()
Can anyone help me understand why this is and how to correct my code so that I can do this with relative imports?
To be considered a package, a Python directory has to include an __init__.py file. Since your module2.py file is not below a directory that contains an __init__.py file, it isn't considered to be part of a package. Relative imports only work inside packages.
UPDATE:
I only gave part of the answer you needed. Sorry about that. This business of running a file inside a package as a script is a bit of a can of worms. It's discussed pretty well in this SO question:
Relative imports in Python 3
The main take-away is that you're better off (and you're doing what Guido wants you to) if you don't do this at all, but rather move directly executable code outside of any module. You can usually do this by adding an extra file next to your package root dir that just imports the module you want to run.
Here's how to do that with your setup:
.
├── package
│ ├── __init__.py
│ ├── subpackage1
│ │ └── module1.py
│ └── subpackage2
│ └── module2.py
└── test.py
test.py:
import package.subpackage2.module2
You then run test.py directly. Because the directory containing the executed script is included in sys.path, this will work regardless of what the working directory is when you run the script.
You can also do basically this same thing without changing any code (you don't need test.py) by running the "script" as a module.
python3 -m package.subpackage2.module2
If you have to make what you're trying to do work, I think I'd take this approach:
import os, sys
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from subpackage1.module1 import module1
module1()
So you compute in a relative way where the root of the enclosing package is in the filesystem, you add that to the Python path, and then you use an absolute import rather than a relative import.
There are other solutions that involve extra tools and/or installation steps. I can't think why you could possibly prefer those solutions to the last solution I show.
By default, Python just considers a directory with code in it to be a directory with code in it, not a package/subpackage. In order to make it into a package, you'll need to add an __init__.py file to each one, as well as an __init__.py file to within the main package directory.
Even adding the __init__.py files won't be enough, but you should. You should also create a setup.py file next to your package directory. Your file tree would look like this:
├── setup.py
└── package
├── __init__.py
└── subpackage1
│ ├── __init__.py
│ └── module1.py
└── subpackage2
├── __init__.py
└── module2.py
This setup.py file could start off like this:
from setuptools import setup
setup(
name='package',
packages=['package'],
)
These configurations are enough to get you started. Then, on the root of your directory (parent folder to package and setup.py), you will execute next command in you terminal pip install -e . to install your package, named package, in development mode. Then you'll be able to navigate to package/subpackage2/ and execute python module2.py having your expected result. You could even execute python package/subpackage2/module2.py and it works.
The thing is, modules and packages don't work the same way they work in another programming languages. Without the creation of setup.py if you were to create a program in your root directory, named main.py for example, then you could import modules from inside package folder tree. But if you're looking to execute package\subpackage2\module2.py.
If you want relative imports without changing your directory structure and without adding a lot of boilerplate you could use my import library: ultraimport
It gives the programmer more control over their imports and lets you do file system based relative or absolute imports.
Your module2.py could then look like this:
import ultraimport
module1 = ultraimport('__dir__/../subpackage1/module1.py')
This will always work, no matter how you run your code or if you have any init files and independent of sys.path.
Following is my project structure
rolutte
├── doc
├── README.rst
├── src
│ ├── outcome.py
│ └── __pycache__
├── tests
│ ├── context.py
│ ├── __init__.py
│ ├── __pycache__
│ └── test_outcome.py
└── tox.ini
Here are the contents of my outcome.py, tests/context.py and tests/test_outcome.py
# outcome.py
from dataclasses import dataclass, field
#dataclass(frozen=True)
class Outcome:
...
# tests/context.py
import os
import sys
# Adding the project root directory to sys.path
sys.path.insert(0, os.path.abspath(
os.path.join(os.path.dirname(__file__), '..')))
import src
# tests/test_outcome.py
from .context import src
from src.outcome import Outcome
def test_outcome():
...
def test_outcome_win_amount():
...
Though pytest is working fine. But mypy shows the following error for tests/context.py
tests/context.py:9: error: Cannot find implementation or library stub for module named "src"
tests/context.py:9: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)
And following for tests/test_outcome.py:
tests/context.py:9: error: Cannot find implementation or library stub for module named "src"
tests/test_outcome.py:6: error: Cannot find implementation or library stub for module named "src.outcome"
tests/test_outcome.py:6: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 2 errors in 2 files (checked 1 source file)
But I have properly provided type hints in my src/outcome.py module while implementing the Outcome class for all the methods and attributes. I don't want to install my src directory as a site package if that could be the solution.
Please tell me how to fix this problem.
I guess you are running project from rolutte folder.
So every import has to be made from that root folder. That mean from .src.outcome import smth.
Dot represents root folder, every other dot represents subfolder/file. This means that you can see first . as .\ and any other as \ like it is in Windows.
EDIT:
From root folder rolutte import everything in the same way.
Let's say I want to import every file inside test_outcome.py, this is how it would look like:
from .src.outcome import *
from .tests.context import *
from .tests import * -> based on __init__.py I guess there is a package.
If you change all imports in each file based on this syntax and then launch script from rolutte root folder, it should work.
NOTE: If you run script from subfolder such as tests folder. The script is unable to see src folder because it is in different tree. You could use os.path.insert(....). But easier is to just launch the script from rolutte folder.
I want to know how to properly import file in my test file without using __init__.py in test root folder. Last sentence of this artice states, that test directory should not have init file.
I don't know much about python so I would like to know:
1) Why not
2) How to import file tested.py to the test_main.py in order to test its functionality without using init file as a script that insert paths to PYTHONPATHS?
My project has a following structure
.
├── __init__.py
├── my
│ ├── __init__.py
│ ├── main.py
│ └── my_sub
│ ├── __init__.py
│ └── tested.py
└── test
└── test_main.py
Files contains following code
#/Python_test/__init__.py
import my.main
#/Python_test/my/__init__.py
#empty
#/Python_test/my/main.py
from .my_sub.tested import inc
print(inc(5))
#/Python_test/my/my_sub/__init__.py
#empty
#/Python_test/my/my_sub/tested.py
def inc(x):
return x+1
#/Python_test/test/test_main.py
from Python_pytest.my.my_sub.tested import func
def test_answer():
assert func(3) == 3
When I run the code from command line python __init__.py it prints 6, which is correct.
I would like to test the function inc() in tested.py file.
1. I installed pytest package as a testing framework,
2. created test file similar to the one from tutorial here called test_main.py.
3. Added __init__.py with a code that finds path of the root directory and adds it to sys.path
It worked well but then I read that this shouldn't be done this way. But how should it be done? I have hard time reading some unit tested code from some github repositories that are tested and don't use the init file. (one of them is boto3) I can't find any clue that suggest how to properly use it.
I also tried to use relative imports this way
from ..my.main import func
but it throws ValueError: attempted relative import beyond top-level package. Which is ok. But I tried it anyway.
Now I don't know how to do that. Tutorials concerning importing usually states that we should add paths of imported modules to sys.path (if they are not present already) but how should I do that when there shouldn't be the init file which can hold the functionality?
I realize there are a slew of posts on SO related to Python and imports, but it seems like a fair number of these posts are asking about import rules/procedures with respect to creating an actual Python package (vs just a project with multiple directories and python files). I am very new to Python and just need some more basic clarification on what is and is not possible with regard to access/importing within the context of multiple py files in a project directory.
Let's say you have the following project directory (to be clear, this is not a package that is somewhere on sys.path, but say, on your Desktop):
myProject/
├── __init__.py
├── scriptA.py
└── subfolder
├── __init__.py
└── scriptB.py
└── subsubfolder
├── __init__.py
└── scriptC.py
└── foo.py
Am I correct in understanding that the only way scriptC.py could import and use methods or classes within scriptB.py if scriptC.py is run directly via $ python scriptC.py and from within the subsubfolder directory is if I add the parent directory and path to scriptB.py to the Python path at runtime via sys.path ?
It is possible, however, for scriptC.py to import foo.py or for scriptB.py to import scriptC.py or foo.py without dealing with sys.path, correct? Adjacent py files and py files in subdirectories are accessible just by using relative import paths, you just can't import python scripts that live in parent or sibling directories (without using sys.path) ?
What's Possible
Anything.
No, really. See the imp module, the the imputil module -- take a look at how the zipimport module is written if you want some inspiration.
If you can get a string with your module's code in a variable, you can get a module into sys.modules using the above, and perhaps hack around with its contents using the ast module on the way.
A custom import hook that looks in parent directories? Well within the range of possibilities.
What's Best Practice
What you're proposing isn't actually good practice. The best-practice approach looks more like the following:
myProject/
├── setup.py
└── src/
├── moduleA.py
└── submodule/
├── __init__.py
├── moduleB.py
└── subsubmodule/
├── __init__.py
└── moduleC.py
Here, the top of your project is always in myProject/src. If you use setup.py to configure moduleA:main, submodule.moduleB:main and submodule.subsubmodule.moduleC:main as entry points (perhaps named scriptA, scriptB and scriptC), then the functions named main in each of those modules would be invoked when the user ran the (automatically generated by setuptools) scripts so named.
With this layout (and appropriate setuptools use), your moduleC.py can absolutely import moduleA, or import submodule.moduleB.
Another approach, which doesn't involve entrypoints, to invoke the code in your moduleC.py (while keeping the module's intended hierarchy intact, and assuming you're in a virtualenv where python setup.py develop has been run) like so:
python -m submodule.subsubmodule.moduleC
I am having an import error problem in a test script. It looks like its something to do with the directory structure.
I have the following folder structure:
A
├── F1
│ ├── __init__.py
│ └── Src
│ └── F2
│ └── __init__.py
└── tests1
└── tests1
└── test_script.py
A/F1/Src/F2
F1 has "__init__py" in its level
F2 has "__init__.py" in its level
In the same level as F1, there is another folder "tests1"
tests1/tests1/test_script.py
in test_script.py, I have a line which says
from F1.src.F2 import C
With the above, I get an error saying, no module named "F1.src.F2"
Does someone know what is going on here?
from F1.src.F2 import C is an absolute import. In order for it to work, "F1" has to be located somewhere on your Python path (sys.path). Generally this also includes the current directory if you ran Python on the command line.
So if the A directory is not one of the directories on your Python path and is not your current working directory, there is no reason the import would work.
Module names are case sensitive. You have Src in one place and src in another, but I'm not sure that reflects your actual directory structure or just what you typed here.
Using a relative import will not work if you are running test_script.py as a script (Which is what it sounds like.) So, what you really want to do is make sure that either you run the script from the A directory, or go whole hog, make your project into a legit package with a setup.py and use a test runner such as tox.
I just had to create a shared library with the "egg" file.
As simple as that but it occurred to me late!