I have a module that sits in a namespace. Should tests and data the tests rely on go in the namespace or in the top level where setup.py sites?
./company/__init__.py
./company/namespace/__init__.py
./company/namespace/useful.py
./company/namespace/test_useful.py
./company/namespace/test_data/useful_data.xml
./setup.py
or
./company/__init__.py
./company/namespace/__init__.py
./company/namespace/useful.py
./test_useful.py
./test_data/useful_data.xml
./setup.py
Does the question amount to whether tests should be installed or not?
The Sample Project stores the tests outside the module.
The directory structure looks like this:
├── data
│ └── data_file
├── MANIFEST.in
├── README.rst
├── sample
│ ├── __init__.py
│ └── package_data.dat
├── setup.cfg
├── setup.py
└── tests
├── __init__.py
└── test_simple.py
Related: The Packing Guide: https://packaging.python.org/en/latest/
Hint: Don't follow the "The Hitchhiker's Guide to Packaging". It has not been updated since 2010!
(do not confuse both pages. The "The Hitchhiker’s Guide to Python" is a very solid book)
You should put your test module inside the module it tests according to The Hitchhiker's Guide to Packaging.
Here is their example:
TowelStuff/
bin/
CHANGES.txt
docs/
LICENSE.txt
MANIFEST.in
README.txt
setup.py
towelstuff/
__init__.py
location.py
utils.py
test/
__init__.py
test_location.py
test_utils.py
This way your module will be distributed with its tests and users can use them to verify that it works with their set up.
See http://the-hitchhikers-guide-to-packaging.readthedocs.org/en/latest/creation.html.
I personally create a single tests package as a sub package of the main package for a few reasons:
If tests is in parallel with the root package there's an off chance you, or a user may misconfigure setup.py and accidentally expose a global package named tests that will cause a great deal of confusion and headache until you realize what has happened. Putting it in the main module solves this as it's now under a (hopefully) globally unique namespace.
I don't like putting a test module within user package because test runners have to search through production code. This is probably not a problem for most. But, if you happen to be a hardware test engineer, you probably use the word 'test' a lot in your production code and don't want the unit test runner to pick that stuff up. It's much easier if all the tests are in one place separate from the production code.
I can further subdivide my tests folder into the types of tests, such as unit, functional and integration. My functional tests tend to have dependencies on weird proprietary hardware, data or are slow. So it's easy for me to continuously run just the fast unit test folder as I develop.
It can sometimes be convenient to have the tests be inside of the same package hierarchy as what it is testing.
Overall though, I think it's important to think for yourself about what's best for your particular problem domain after taking everyone's advice into account. 'Best practices' are great starting points, not end points, for developing a process.
Related
Let me say I'm still new to unit testing and using pytest.
I'm building unit tests and running them with GitHub Actions and I understand why it failed. I need to run the test modules in a specific order.
For reference, I have a following repo structure (generalized for simplicity)
mypackage/
├── __init__.py
├── foo.py
└── bar.py
tests/
├── __init__.py
├── test_foo.py
└── test_bar.py
My question (written for generalization): Is there a way to run specific test modules (not functions) in a given order with pytest? By default pytest will run in alphabetical order, i.e., test_bar before test_foo. However, in my case, test_bar uses data from test_foo, so I would like the order to be flipped. I can find other work arounds, but from my search I only found explanations of using pytest-order or pytest-ordering for specific functional orders in a given module. I thought this would be a good question for end-to-end testing while still ensuring 100% code coverage.
Thanks.
We using py.test. We try to put different conftest.py files in different folders to split our fixtures:
tests/api/
├── conftest.py
├── folder1
│ └── conftest.py
├── folder2
│ └── conftest.py
But when run the tests this error occurs:
____ ERROR collecting api/folder1/conftest.py ____
import file mismatch:
imported module 'conftest' has this __file__ attribute:
/tests/api/folder2/conftest.py
which is not the same as the test file we want to collect:
/tests/api/folder1/conftest.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
Why is that? How fix it?
PS. Removing __pycache__.pyc did not help.
PPS. __init__.py files already exist in each folder.
I had the same issue. To solve this you need to create python packages instead of directories. Then pytest will look at the conftest.py in your package instead of root directory. Hope, this will help you.
tests/api/
├── conftest.py
├── package1 # not folder
│ └── conftest.py
├── package2 # not folder
│ └── conftest.py
Your use case sounds like this example in the pytest documentation. Because of that I think it's possible to use conftest.pys at different levels to override fixtures.
The errors you're seeing may be related to incorrect imports. Is your test code importing from conftest files directly? Are your conftest files importing from your tests? Are any of your imports relative instead of absolute? If any of these are true, that may be your issue. I recommend only using absolute imports, and avoid imports between conftest.pys and test files.
Rename one (or both) of the test files Pytest is complaining about. Pytest is telling you in the error message to do this (i.e. change the basename, meaning don't name all your test files conftest.py). For example, you can fix it by doing:
tests/api/
├── conftest.py
├── folder1
│ └── test_conf1.py
├── folder2
│ └── test_conf2.py
In your case, the module names conflict (you have three conftest.pys). This is a quirk of Pytest AFAIK. Pytest could get around this by managing full package/module paths: but it doesn't do this (probably for good reason, but I do not maintain/contribute to pytest so I can't shed light on the issue). Pytest is a fantastic framework (it's even telling you exactly why it can't run your tests): I'm sure they have a good reason for not supporting this behavior.
You claim that you want to:
separate tests and fixtures by different functionalities.
So do that. Separating the test fixtures/functionalities has nothing to do with what you name the files.
I commonly run into this error when splitting up unit/integration/acceptance tests. I split them up so I can run my (fast) unit tests without having to run my (potentially slow) integration/acceptance tests. I might have some module, call it Abc. And I have something like:
tests/
├── unit
│ └── test_abc.py
├── integration
│ └── test_abc.py
But then pytest barfs with the identical error you've shown, and so I just rename integration/test_abc.py to integration/test_abc_integration.py and move on with my day. Like this:
tests/
├── unit
│ └── test_abc.py
├── integration
│ └── test_abc_integration.py
Is it annoying? A little. How long does the fix take? 5 whole seconds.
P.S. You might have to remove __pycache__ directories or you .pyc files for the first run after you get the error you've posted about (if you don't you'll just get the same error again even if you rename).
P.S.S. You can stop the Cpython interpreter (and most others) from writing out __pycache__ and .pyc files by calling python -B -m pytest .... The -B option makes the interpreter not save the bytecode to your filesystem. This results in some performance penalty whenever you run your test suite, but the penalty is usually very small (milage may vary). I typically use this option because I don't like the clutter in my repositories and the performance loss is typically negligible.
I'm writing a couple of packages that I'd like to release on PyPi for other people to use.
I've not released to PyPi before so I have been mocking up a submission template: https://github.com/chris-brown-nz/pypi-package-template
Here's a tree of the project template:
| MANIFEST.in
| README.rst
| setup.cfg
| setup.py
|
\---package
module_one.py
module_three.py
module_two.py
__init__.py
In terms of interacting with the package, this is what I would usually do - is it the best way?
To run a method:
from package import module_one
module_one.ClassOne().method_a()
To get a value from a method:
from package import module_two
print(module_two.ClassFive().method_e())
To set then use an attribute of an instance:
from package import module_three
cls = module_three.ClassSeven("Hello World")
print(cls.value)
'package' is a reserved name obviously and won't be used in the final project.
I'd be grateful for some feedback on how I've structured my project and whether it is considered standard, or if it should be modified in some way.
There are different approaches to this, whether one or the other is better is depending on a how you want to develop, usage of the package (e.g. if you ever install it using pip install -e packag_name), etc.
What is missing from your tree is the name of the directory where the setup.py resides, and that is usually the package name:
└── package
├── package
│ ├── __init__.py
│ ├── module_one.py
│ ├── module_three.py
│ └── module_two.py
├── MANIFEST.in
├── README.rst
├── setup.cfg
└── setup.py
as you can see you are doubling the 'package' name, and that means that your setup.py has to be adapted for each package, or dynamically determine the name of the directory where the module.py files resides. If you go for this route, I would suggest you put the module.py files in a generically named directory 'src' or 'lib'.
I don't like the above "standard" setup for multiple reasons:
it doesn't map well to how python programs "grow" before they are split up into packages. Before splitting up having such a 'src' directory would mean using:
from package.src.module_one import MyModuleOneClass
Instead you would have your module.py files directly under package
Having a setup.py to control installation, a README.rst for documentation and an __init__.py to satisfy Python's import is one thing, but all other stuff, apart from your module.py files containing the actual functionality, is garbage. Garbage that might be needed at some point during the package creation process, but is not necessary for the package functionality.
There are other considerations, such as being able to access the version number of the package from the setup.py as well as from the program, without the former having to import the package itself (which may lead to install complications), nor having another extra version.py file that needs importing.
In particular I always found the transition from using a directory structure under site-packages that looked like:
└── organisation
├── package1
└── package2
├── subpack1
└── subpack2
and that could intuitively be used for both importing and navigation to source files, to something like:
├── organisation_package1
│ └── src
├── organisation_package2_subpack1
│ └── src
└── organisation_package2_subpack2
└── src
unnatural. To rearrange and break a working structure to be able to package things seems wrong.
For my set of published packages I followed another way:
- I kept the natural tree structure that you can use "before packaging", 'src' or 'lib' directories.
- I have a generic setup.py which reads and parses (it does not import) the metadata (such as version number, package name, license information, whether to install a utility (and its name)), from a dictionary in the __init__.py file. A file you need anyway.
- The setup.py is smart enough to distinguish subdirectories containing other packages from subdirectories that are part of the parent package.
- setup.py generates files that are needed during package generation only (like setup.cfg), on the fly, and deletes them when no longer needed.
The above allows you to have nested namespaced packages (i.e. package2 can be a package you upload to PyPI, in addition to package2.subpack1 and package2.subpack2). The major thing it (currently) doesn't allow is using pip install -e to edit a single package (and not have the others editable). Given the way I develop, that is not a restriction.
The above embraces namespace packages, where many other approaches have problems with these (remember the last line of Zen of Python: Namespaces are one honking great idea – let’s do more of those)
Examples of the above can e.g be found in my packages ruamel.yaml (and e.g. ruamel.yaml.cmd), or generically by searching PyPI for ruamel.
As is probably obvious, the standard disclaimer: I am the author of those packages
As I use a utility to start packaging, which also runs the tests and does other sanity checks, the generic setup.py could be removed from the setup and inserted by that utility as well. But since subpackage detection is based upon setup.py availability or not, this requires some rework of the generic setup.py.
We have a Python project laid out like this:
project/
├── .pylintrc
├── module1.py
├── module2.py
└── tests/
├── test_module1.py
└── test_module2.py
Our unit and function tests reside in the folder called tests/. When it comes to tests the pylint warnings missing-docstring, invalid-name and protected-access are not relevant. On the other hand, these warnings are very useful for the actual code in the project.
My question is whether it is possible to add ignores for missing-docstring, invalid-name and protected-access in the .pylintrc-file that apply to modules in the tests/-folder only?
If possible, we do not want to add #-disables for these warnings to every test-module inside the folder.
As far as I'm aware you can't disable specific warnings for entire directories or files.
However, you can disable all warnings for specific directories using the following on the command line:
--ignore=<file[,file]>
The file here can be a directory.
Personally, and I know you said you'd rather not, I'd add a disable to the top of each file.
Yes, you can create .pylintrc in the tests folder, and another in the project folder.
Add tests to the "ignore" section of the project
[MASTER]
ignore=tests
See: https://docs.pylint.org/en/1.6.0/run.html
Then run separately:
pylint project
pylint project/tests
I'm setting up a directory structure for my Django app to separate functional and unit tests. I am using nose as the test runner for my Django project.
At the root of the Django project, I have a folder called "tests" that has this structure:
tests
├── __init__.py
├── functional
│ ├── __init__.py
└── unit
├── __init__.py
├── data.py
├── tests.py
If I want to run just the unit tests, should I not be able to use the following from the project root:
$ nosetests tests.unit
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
As you can see, this doesn't find the tests in the tests.py file.
However, when I run using the directory structure, the tests are found as they should be:
$ nosetests tests/unit/
E
# .. Some errors I expected because settings are not initialized when called this way
-----------------
Ran 1 test in 0.001s
FAILED (errors=1)
What am I missing? My main issue is that I have a setup function in tests.unit.__init__.py that should be called for creating the data in the test DB for the upcoming tests.
Thanks
This all depends on what kind of code is in tests/unit/__init__.py
When you say
nosetests tests.unit
You are pointing to unit/__init__.py not the directory unit/ thus if you had no tests in your __init__.py module then nothing would be run. So it is understandable when you say you used the directory path and then your tests started working.
You mention
What am I missing? My main issue is that I have a setup function in
tests.unit.init.py that should be called for creating the data in
the test DB for the upcoming tests.
It is likely that although you have a setup function in __init__.py you may have not ever imported your test functions into __init__.py
One quick fix to this would be to add this line in __init__.py
from tests.unit.tests import *
That said it is really not very wise to be putting any code in __init__.py at all and if you have code that returns some kind of configuration data I would recommend creating a new library module with functions that will return configuration data to your tests