I am creating documentation with Sphinx. My folder structure looks as follows:
MyProject
├── mypackage
│ ├── __init__.py
│ ├── mycode.py
│ └── etc.
└── docs
├── build
├── make.bat
├── Makefile
└── source
├── conf.py
├── index.rst
├── _static
└── _templates
I begin by running make clean and make html in the docs directory. Next, to populate the documentation, I run sphinx-apidoc -o ./source ../mypackage, and all corresponding .rst files are generated as expected. Finally, I run make clean and make html once more to ensure a clean build, as suggested in the Sphinx-RTD-Tutorial. However, on this final build, I get the following output:
Running Sphinx v4.0.2
making output directory... done
[autosummary] generating autosummary for: index.rst, mypackage.rst, mypackage.mycode.rst
Extension error (sphinx.ext.autosummary):
Handler <function process_generate_options at 0x10678dee0> for event 'builder-inited' threw an exception (exception: list.remove(x): x not in list)
make: *** [html] Error 2
Removing the autosummary extension and just running autodoc with the same sequence of commands leads to a similar error:
Exception occurred:
File "/Users/myname/opt/anaconda3/envs/myenv/lib/python3.9/site-packages/sphinx/ext/autodoc/mock.py", line 151, in mock
sys.meta_path.remove(finder)
ValueError: list.remove(x): x not in list
Here is the source code method that the error comes from:
#contextlib.contextmanager
def mock(modnames: List[str]) -> Generator[None, None, None]:
"""Insert mock modules during context::
with mock(['target.module.name']):
# mock modules are enabled here
...
"""
try:
finder = MockFinder(modnames)
sys.meta_path.insert(0, finder)
yield
finally:
sys.meta_path.remove(finder)
finder.invalidate_caches()
Does anyone know what might be raising this error or have an idea as to what is happening in this method? Could it have to do with my specification of sys.path in my conf.py file?
[conf.py]
sys.path.insert(0, os.path.abspath('../../mypackage'))
I was able to resolve this error using the autodoc_mock_imports config:
autodoc_mock_imports
This value contains a list of modules to be
mocked up. This is useful when some external dependencies are not met
at build time and break the building process. You may only specify the
root package of the dependencies themselves and omit the sub-modules:
autodoc_mock_imports = ["django"]
Will mock all imports under the django package.
New in version 1.3.
Changed in version 1.6: This config value only requires to declare the
top-level modules that should be mocked.
Related
I am not very familiar with python, I only done automation with so I am a new with packages and everything.
I am creating an API with Flask, Gunicorn and Poetry.
I noticed that there is a version number inside the pyproject.toml and I would like to create a route /version which returns the version of my app.
My app structure look like this atm:
├── README.md
├── __init__.py
├── poetry.lock
├── pyproject.toml
├── tests
│ └── __init__.py
└── wsgi.py
Where wsgi.py is my main file which run the app.
I saw peoples using importlib but I didn't find how to make it work as it is used with:
__version__ = importlib.metadata.version("__package__")
But I have no clue what this package mean.
You should not use __package__, which is the name of the "import package" (or maybe import module, depending on where this line of code is located), and this is not what importlib.metadata.version() expects. This function expects the name of the distribution package (the thing that you pip-install), which is the one you write in pyproject.toml as name = "???".
You can extract version from pyproject.toml using toml package to read toml file and then display it in a webpage.
I have a project for developing a Python package where the structure of the project is similar to the following:
myproject/
├── README.md
├── examples/
│ ├── ex1.py
│ └── ex2.py
├── pyproject.toml
├── src/
│ └── mypackage/
│ ├── __init__.py
│ ├── adder.py
│ └── divider.py
└── tests/
├── test_adder.py
├── test_divider.py
└── test_examples.py
The project is for developing a Python package named mypackage which is located in the src directory. The package is uploaded to PyPI where users can pip install it. Tests for the package are run with pytest and are located in the tests directory. Examples of using the package are in the examples directory. The examples are just scripts as shown below for ex1.py
"""
Example 1
"""
from mypackage import adder
x = 2.5
y = 8
a = adder(x, y)
print('a is', a)
The purpose of test_examples.py is to test the example files, its contents are shown below:
from examples import ex1
from examples import ex2
def test_ex1():
ex1
def test_ex2():
ex2
When I run pytest in the myproject directory I get the error shown here:
$ cd myproject
$ pytest
platform darwin -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/gavinw/Desktop/test-examples
collected 2 items / 1 error
================================================================== ERRORS ==================================================================
_________________________________________________ ERROR collecting tests/test_examples.py __________________________________________________
ImportError while importing test module '/Users/gavinw/Desktop/test-examples/tests/test_examples.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/miniconda3/envs/ztest/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_examples.py:1: in <module>
from examples import ex1
E ModuleNotFoundError: No module named 'examples'
========================================================= short test summary info ==========================================================
ERROR tests/test_examples.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================= 1 error in 0.05s =============================================================
It looks like pytest is not able to run the files in the examples directory because the location does not allow them to be imported. Any suggestions on how I can test the example files? Should I even use pytest for testing examples or is there a different testing tool for this?
You should ensure that the examples directory contains __init__.py so it can be imported correctly.
If that is not enough, you can use PYTHONPATH:
PYTHONPATH="/path/to/your/code/project/:/path/to/your/code/project/examples" pytest
PYTHONPATH might be tricky, see https://stackoverflow.com/a/4580120/3800552 and https://stackoverflow.com/a/39682723/3800552 for some usage examples.
There are two approaches that fix the problem. The first approach is to run pytest using the following command:
python -m pytest
The second approach is to add the project directory in the pyproject.toml file using the pythonpath setting as shown below. Then just use the pytest command to run the tests.
[tool.pytest.ini_options]
pythonpath = ["."]
I have a question in how to properly create a path in Python (Python 3.x).
I developed a small scraping app in Python with the following directory structure.
root
├── Dockerfile
├── README.md
├── tox.ini
├── src
│ └── myapp
│ ├── __init__.py
│ ├── do_something.py
│ └── do_something_else.py
└── tests
├── __init__.py
├── test_do_something.py
└── test_do_something_else.py
When I want to run my code, I can go to the src directory and do with
python do_something.py
But, because do_something.py has an import statement from do_something_else.py, it fails like:
Traceback (most recent call last):
File "src/myapp/do_something.py", line 1, in <module>
from src.myapp.do_something_else import do_it
ModuleNotFoundError: No module named 'src'
So, I eventually decided to use the following command to specify the python path:
PYTHONPATH=../../ python do_something.py
to make sure that the path is seen.
But, what are the better ways to feed the path so that my app can run?
I want to know this because when I run pytest via tox, the directory that I would run the command tox would be at the root so that tox.ini is seen by tox package. If I do that, then I most likely run into a similar problem due to the Python path not properly set.
Questions I want to ask specifically are:
where should I run my main code when creating my own project like this? root as like python src/myapp/do_something.py? Or, go to the src/myapp directory and run like python do_something.py?
once, the directory where I should execute my program is determined, what is the correct way to import modules from other py file? Is it ok to use from src.myapp.do_something_else import do_it (this means I must add path from src directory)? Or, different way to import?
What are ways I can have my Python recognize the path? I am aware there are several ways to make the pass accessible as below:
a. write export PYTHONPATH=<path_of_my_choice>:$PYTHONPATH to make the
path accessible temporarily, or write that line in my .bashrc to make it permanent (but it's hard to reproduce when I want to automate creating Python environment via ansible or other automation tools)
b. write import sys; sys.path.append(<root>) to have the root as an accessible path
c. use pytest-pythonpath package (but this is not really a generic answer)
Thank you so much for your inputs!
my environment
OS: MacOS and Amazon Linux 2
Python Version: 3.7
Dependency in Python: pytest, tox
I would suggest to use setup.py to make this a python package. Then you can install it in development mode python setup.py develop. This way it will be available in your python environment w/o needing to specify the PYTHONPATH.
For testing, you can simply install the package python setup.py install.
Hope that helps.
Two simple steps should make it happen. Python experts can comment if this is a good way to do it (especially going by the concluding caution raised towards the end of this post).
I would have done it like below.
First I would have put a "__init__.py" in root so that hierarchy looks like below. This way python will treat the folder as a package.
root
├── Dockerfile
├── README.md
├── tox.ini
├── __init__.py
├── src
│ └── myapp
│ ├── __init__.py
│ ├── do_something.py
│ └── do_something_else.py
└── tests
├── __init__.py
├── test_do_something.py
└── test_do_something_else.py
Then in "do_something.py", I would have added these lines at the top. In the second line please put the full path to the "root" directory.
import sys
sys.path += ['/home/SomeUserName/SomeFolderPath/root']
from src.myapp.do_something_else import do_it
Please note that the second line will essentially modify the sys.path by adding the root folder path (I guess until the interpreter quits). If this is not what you can afford then I am sorry.
I'm creating a pytest unit test for a function in my software.
Before even starting to test, pyunit seems to be unable to import my "cache_offline" decorator which is indirectly imported when I import the function I'm testing in my test.
I'm using Anaconda embedding Python 3.7 and pytest 5.2.2
I tried to comment out the code where the decorator is applied to my functions, when I do so the pytest error disappear and the tests execute properly.
My test is in ./tests/scripts/test_scripts_helper.py and I run pytest at the project root .
Pytest finds properly my test (see the error message), so this is not the problem at hand here.
My test imports and wants to test the function read_tiff_tag from a package vorace.scripts_helper, which imports a function safe_mkdir from package vorace.core.misc, which imports the package vorace.core.vorace, in which 3 functions are decorated with the decorator cache_offline from package vorace.core.misc
I tried both running the tests using either py.test or python -m pytest at the root of my project.
My project have the following structure (simplified).
The code root is ./vorace
The tests root is ./tests
.
├── conftest.py
├── tests
│ ├── __init__.py
│ ├── scripts
│ │ ├── __init__.py
│ │ └── test_scripts_helper.py
│ └── tests_data
│ └── test_ROI.tif
└── vorace
├── __init__.py
├── core
│ ├── __init__.py
│ ├── misc.py
│ └── vorace.py
└── scripts
├── __init__.py
├── batch_analyzis.py
└── scripts_helper.py
I tried :
with and without empty __init__.py in each subfolder of the tests folder. -> no change
with and without an empty conftest.py at the root of the project. -> no change
executing a test which doesn't need any import in my test_scripts_helper.py file (with my test causing the problem being commented out) -> the test executes properly
I suspect kind of a circular import problem but I've always been told that it can't happen in python. Maybe the decorators are an exception to this rule ?
My vorace.core.misc code, with the decorator
from vorace.core import vorace
[...]
def cache_offline(cache_path=os.getcwd()):
[...]
def decorator(func):
[...]
def wrapper(*args, **kwargs):
[...]
return result
return wrapper
return decorator
def safe_mkdir(path):
[...]
One of the decorated functions in vorace.core.vorace
from vorace.core.misc import *
[...]
#cache_offline(cache_path=".cache")
def classify_clusters_by_connectivity(xyz_data):
[...]
[...]
The output from executing py.test in the project root
==================== test session starts ====================
platform linux -- Python 3.7.3, pytest-5.2.2, py-1.8.0, pluggy-0.12.0
rootdir: /home/flo/PycharmProjects/VorAce
plugins: arraydiff-0.3, openfiles-0.3.2, doctestplus-0.3.0, remotedata-0.3.1
collected 0 items / 1 errors
==================== ERRORS ====================
_________ ERROR collecting tests/scripts/test_scripts_helper.py _________
tests/scripts/test_scripts_helper.py:1: in <module>
import vorace.scripts.scripts_helper as sh
vorace/scripts/scripts_helper.py:6: in <module>
from vorace.core.misc import safe_mkdir
vorace/core/misc.py:8: in <module>
from vorace.core import vorace
vorace/core/vorace.py:91: in <module>
#cache_offline(cache_path=".cache")
E NameError: name 'cache_offline' is not defined
If I execute a simple 0 == 0 test in my tests/scripts/test_scripts_helper.py file without importing from my project, the test runs with success.
When you run ./tests/scripts/test_scripts_helper.py python automatically sets ./tests/scripts/ into PYTHONPATH but not sets any other directories, so all your imports from other custom files should fail.
Set all your working directories into PYTHONPATH environment variable.
Something like that on Linux shell.
PYTHONPATH="${PYTHONPATH}:$dir
EDIT: I finally got the confirmation of a circular import.
At the opposite of what I thought, importing particular names from a module
like in from x import y can be sensitive to circular imports, where import x can't.
To solve it, I just imported the module and use the syntax using the module prefixing the function call.
More information here : https://www.reddit.com/r/Python/comments/51hdup/from_import_vs_import_on_circular_import/
The problem what finally not related specifically to the decorator or pytest.
I worked around this issue by putting my cache_offline decorator in a separated package vorace.core.caching.py. Now I import this package only from vorace.core.vorace where the function needing to be decorated resides.
This way my decorator is artificially excluded from the code imported by my unit test, but still available to the rest of my code for a normal application execution.
However pytest still have an issue here, it should not fail to import. I'm still taking any answer that can explain why pytest fails to import my decorator, and I keep the resolve for an answer of this kind.
I have a package with a directory "tests" in which I'm storing my unit tests. My package looks like:
.
├── LICENSE
├── models
│ └── __init__.py
├── README.md
├── requirements.txt
├── tc.py
├── tests
│ ├── db
│ │ └── test_employee.py
│ └── test_tc.py
└── todo.txt
From my package directory, I want to be able to find both tests/test_tc.py and tests/db/test_employee.py. I'd prefer not to have to install a third-party library (nose or etc) or have to manually build a TestSuite to run this in.
Surely there's a way to tell unittest discover not to stop looking once it's found a test? python -m unittest discover -s tests will find tests/test_tc.py and python -m unittest discover -s tests/db will find tests/db/test_employee.py. Isn't there a way to find both?
In doing a bit of digging, it seems that as long as deeper modules remain importable, they'll be discovered via python -m unittest discover. The solution, then, was simply to add a __init__.py file to each directory to make them packages.
.
├── LICENSE
├── models
│ └── __init__.py
├── README.md
├── requirements.txt
├── tc.py
├── tests
│ ├── db
│ │ ├── __init__.py # NEW
│ │ └── test_employee.py
│ ├── __init__.py # NEW
│ └── test_tc.py
└── todo.txt
So long as each directory has an __init__.py, python -m unittest discover can import the relevant test_* module.
If you're okay with adding a __init__.py file inside tests, you can put a load_tests function there that will handle discovery for you.
If a test package name (directory with __init__.py) matches the
pattern then the package will be checked for a 'load_tests' function. If
this exists then it will be called with loader, tests, pattern.
If load_tests exists then discovery does not recurse into the package,
load_tests is responsible for loading all tests in the package.
I'm far from confident that this is the best way, but one way to write that function would be:
import os
import pkgutil
import inspect
import unittest
# Add *all* subdirectories to this module's path
__path__ = [x[0] for x in os.walk(os.path.dirname(__file__))]
def load_tests(loader, suite, pattern):
for imp, modname, _ in pkgutil.walk_packages(__path__):
mod = imp.find_module(modname).load_module(modname)
for memname, memobj in inspect.getmembers(mod):
if inspect.isclass(memobj):
if issubclass(memobj, unittest.TestCase):
print("Found TestCase: {}".format(memobj))
for test in loader.loadTestsFromTestCase(memobj):
print(" Found Test: {}".format(test))
suite.addTest(test)
print("=" * 70)
return suite
Pretty ugly, I agree.
First you add all subdirectories to the test packages's path (Docs).
Then, you use pkgutil to walk the path, looking for packages or modules.
When it finds one, it then checks the module members to see whether they're classes, and if they're classes, whether they're subclasses of unittest.TestCase. If they are, the tests inside the classes are loaded into the test suite.
So now, from inside your project root, you can type
python -m unittest discover -p tests
Using the -p pattern switch. If all goes well, you'll see what I saw, which is something like:
Found TestCase: <class 'test_tc.TestCase'>
Found Test: testBar (test_tc.TestCase)
Found Test: testFoo (test_tc.TestCase)
Found TestCase: <class 'test_employee.TestCase'>
Found Test: testBar (test_employee.TestCase)
Found Test: testFoo (test_employee.TestCase)
======================================================================
....
----------------------------------------------------------------------
Ran 4 tests in 0.001s
OK
Which is what was expected, each of my two example files contained two tests, testFoo and testBar each.
Edit: After some more digging, it looks like you could specify this function as:
def load_tests(loader, suite, pattern):
for imp, modname, _ in pkgutil.walk_packages(__path__):
mod = imp.find_module(modname).load_module(modname)
for test in loader.loadTestsFromModule(mod):
print("Found Tests: {}".format(test._tests))
suite.addTests(test)
This uses the loader.loadTestsFromModule() method instead of the loader.loadTestsFromTestCase() method I used above. It still modifies the tests package path and walks it looking for modules, which I think is the key here.
The output looks a bit different now, since we're adding a found testsuite at a time to our main testsuite suite:
python -m unittest discover -p tests
Found Tests: [<test_tc.TestCase testMethod=testBar>, <test_tc.TestCase testMethod=testFoo>]
Found Tests: [<test_employee.TestCase testMethod=testBar>, <test_employee.TestCase testMethod=testFoo>]
======================================================================
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s
OK
But we still get the 4 tests we expected, in both classes, in both subdirectories.
The point using init.py, is that one may encounters side effects, like file not being the script file path. Using FOR DOS command can help (not found of DOS commands, but sometimes it helps
setlocal
set CWD=%CD%
FOR /R %%T in (*_test.py) do (
CD %%~pT
python %%T
)
CD %CWD%
endlocal
/R allows for walkthrough the hierarchy from current folder.
(expr) allows for selecting test files (I use _test.py)
%%~pT is $(dirname $T) in shell.
I saved and restore my original directory, as the .bat leaves me where it ends
setlocal ... endlocal to not pollute my environment with CWD.