We Keep Coding

Poetry run pytest only works with __init__.py in tests/ directory

Poetry and pytest are relatively new to me and I am seeking to understand a specific behavior.
I have created a project with poetry, and I have added pytest as a dependy with poetry add --group dev pytest. As a result, here are the relevant line from pyproject.toml:
[tool.poetry.dependencies]
python = "^3.10"
[tool.poetry.group.dev.dependencies]
pytest = "^7.2.0"
Here is the project/modules structure:
.
├── app
│   ├── core
│   │   ├── cli.py
│   │   └── __init__.py
│   ├── __init__.py
│   └── run.py
├── poetry.lock
├── pyproject.toml
├── README.md
└── tests
└── test_cli.py
In test_cli.py I am importing the cli module with from app.core.cli import *
With this particular setup, running poetry run pytest fails with:
tests/test_cli.py:2: in <module>
from app.core.cli import *
E ModuleNotFoundError: No module named 'app'
However, if I invoke a poetry shell and run python3 -m pytest tests/ it works.
It also works if I create an __init__.py in the tests/ directory.
I have seen a similar issue described here but I do not understand how it was resolved. One the suggestions to include include the project root name in the import statement did not help.
I also tried this answer, but it also did not work.
I was under the impression that an __init__.py is not necessary in tests/ and that poetry run pytest should work without it.

Way to add pytest options outside top level conftest?

I have a project that defines testing support modules, including py.test plugins in a package subdirectory like this:
bokeh/_testing
├── __init__.py
├── plugins
│ ├── __init__.py
│ ├── bokeh_server.py
│ ├── examples_report.jinja
│ ├── examples_report.py
│ ├── file_server.py
│ ├── implicit_mark.py
│ ├── integration_tests.py
│ ├── jupyter_notebook.py
│ ├── log_file.py
│ └── pandas.py
└── util
├── __init__.py
├── api.py
├── examples.py
├── filesystem.py
├── git.py
└── travis.py
Several of the plugins need to define new options with parser.addoption. I would like these calls to be made inside the respective plugin modules. However, if I do that, and include those plugins in a test file with, e.g.
# test_examples.py
pytest_plugins = (
"bokeh.testing.plugins.bokeh_server",
"bokeh.testing.plugins.examples_report",
)
# pytest.mark.examples test code here
Then pytest complains that any custom command line options are undefined:
(base) ❯ py.test -s -v -m examples --diff-ref FETCH_HEAD --report-path=examples.html
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: --diff-ref --report-path=examples.html
inifile: /Users/bryanv/work/bokeh/setup.cfg
rootdir: /Users/bryanv/work/bokeh
The only way I have found around this, is to collect ALL custom options in a single pytest_addoption in the top-level conftest.py:
# conftest.py
pytest_plugins = (
"bokeh.testing.plugins.implicit_mark",
"bokeh.testing.plugins.pandas",
)
# Unfortunately these seem to all need to be centrally defined at the top level
def pytest_addoption(parser):
parser.addoption(
"--upload", dest="upload", action="store_true", default=False, help="..."
)
# ... ALL other addoptions calls for all plugins here ...
As I said, this works, but is very inflexible in terms of code organization. It would be preferable to have a way for the options for the examples.py plugin to in the examples.py module, with the code it relates to.
Another possibility would presumably be to bring in all the plugins in the top-level conftest.py, but some of the plugins are very heavyweight, e.g. depend on Selenium, and I would not like to require that all that be installed just to run basic tests.
Is there another way to accomplish this, and if so, how?

As noted above, as of Pytest 3.6.2, options must be added "only in plugins or conftest.py files situated at the tests root directory due to how pytest discovers plugins during startup."

No module error in Python 3.6 with Click library

I'm trying to build a CLI in python by using the package click. The Python version I'm using is 3.6
This is the main of my application:
import os
import click
cmd_folder = os.path.join(os.path.dirname(__file__), 'commands')
class IAMCLI(click.MultiCommand):
def list_commands(self, ctx):
rv = []
for filename in os.listdir(cmd_folder):
if filename.endswith('.py') and \
filename.startswith('cmd_'):
rv.append(filename[4:-3])
rv.sort()
return rv
def get_command(self, ctx, cmd_name):
ns = {}
fn = os.path.join(cmd_folder, 'cmd_{}.py'.format(cmd_name))
with open(fn) as f:
code = compile(f.read(), fn, 'exec')
eval(code, ns, ns)
return ns['cli']
#click.command(cls=IAMCLI)
#click.option('--env', default='dev', type=click.Choice(['dev', 'staging', 'production']),
help='AWS Environment')
#click.pass_context
def cli():
"""AWS IAM roles and policies management CLI."""
pass
if __name__ == '__main__':
cli()
and this is the tree:
├── cli
│   ├── __init__.py
│   ├── aws
│   │   ├── __init__.py
│   │   ├── policy.py
│   │   └── role.py
│   ├── cli.py
│   └── commands
│   ├── __init__.py
│   └── cmd_dump.py
the cmd_dump.py looks like this:
import click
from cli.aws.role import fetch_roles
#click.command('dump', short_help='Dump IAM resources')
#click.pass_context
def cli():
pass
the problem is that when I try to run python cli/cli.py --help this is what I get:
File "cli/commands/cmd_dump.py", line 3, in <module>
from cli.aws.role import fetch_roles
ModuleNotFoundError: No module named 'cli.aws'; 'cli' is not a package
Any idea about that?

I will try to give another answer based on my approach when starting development of a new python project. Do you plan to distribute your project, or maybe just share it with someone? If you do, what do you think - will this someone be happy with needing to remember the command
$ python path/to/project/codebase/cli/cli.py --help
to use your tool? Wouldn't it be easier for him to remember the command
$ cli --help
instead?
I suggest you to start with packaging of your project right away - write a minimal setup script:
from setuptools import setup, find_packages
setup(
name='mypkg',
version='0.1',
packages=find_packages(),
install_requires=['click'],
entry_points={
'console_scripts': ['cli=cli.cli:cli'],
},
)
You can always enhance your setup script when new requirements emerge. Place the setup script in the root directory of your codebase:
├── setup.py
├── cli
│   ├── __init__.py
│   ├── aws
...
Now run python setup.py develop or even better, pip install --editable=.1 from the codebase root directory (where the setup.pyscript is). You have installed your project in the development mode and now can invoke
$ cli --help
with all the imports being resolved correctly (which would resolve your issue). But you gain much more besides that - you gain a way to package your project to be ready to be distributed to your target users and a clean command line interface which your users will invoke the same way as you just did.
Now continue with the project development. If you change the code of the cli command, it will be applied on the fly so you don't need to reinstall the project each time you change anything.
Once you are ready with the project development and want to deliver it to your users, issue:
$ python setup.py bdist_wheel
This will package your project into a installable wheel file (you will need to install wheel package to be able to invoke the command: pip install wheel --user). Usually it will reside in the dist subdirectory of the codebase root dir. Give this file to the user. To install the file, he will issue
$ pip install Downloads/mypkg-0.1-py3-none.whl --user
and can start tp use your tool right away:
$ cli --help
This is a very much simplified description and there is a lot of stuff to learn, but there is also a ton of helpful materials to guide you through the process.
If you want to learn more about the topic: as a quickstart reference, I would recommend the excellent PyPA packaging guide. For packaging click commands, their own docs are more than sufficient.
I would encourage you to use pip for distribution and packaging development where applicable as it is a standard tool for that.

Do not run scripts inside packages! Packages are made to be imported in code but not to run scripts inside them. The rest is non related with import errors.
For example:
├── cli # package
│ ├── __init__.py
│ ├── aws
│ │ ├── __init__.py
│ │ ├── policy.py
│ │ └── role.py
│ ├── cli.py
│ │ └── commands
│ │ ├── __init__.py
│ │ └── cmd_dump.py
├── run_this_module.py
Module to execute run_this_module.py:
import cli
"""Do your code here"""

I have done this a hundred times. It is tempting to name a package the same thing at multiple levels, but resist! I tend to have __main__.py in my packages these days, which helps solve the problem that you are having.
I suspect you are having a namespace issue. Try renaming your package or the internal file called cli.py. Be sure to refactor any imports that are attempting to use these.
Also, change your function name to something besides cli. Same issue.
├── cli <- rename this to 'my_app' or something else
│ ├── __init__.py
│ ├── aws
│ │ ├── __init__.py
│ │ ├── policy.py
│ │ └── role.py
│ ├── cli.py <- or maybe it is easier to rename this
│ └── commands
│ ├── __init__.py
│ └── cmd_dump.py

get site-packages folder from a python unittest

I have a standard python proj package::
proj/
├── proj
│   ├── data
│   │   └── data.csv
│   ├── __init__.py
│   └── lib.py
├── MANIFEST.in
├── setup.py
└── tests
├── __init__.py
└── lib_test.py
data/data.csv is installed with MANIFEST.in and include_package_data key in setup.py::
~/proj$ more setup.py
from setuptools import setup
setup(name='proj',
version=1.0,
packages=['proj'],
test_suite = 'tests',
include_package_data=True,
zip_safe=False)
~/proj$ more MANIFEST.in
include proj/data/*.csv
So after a python setup.py install data.csv is in the place I want in site-packages::
$ tree ~/venv/lib/python2.7/site-packages/proj-1.0-py2.7.egg
~/venv/lib/python2.7/site-packages/proj-1.0-py2.7.egg
├── EGG-INFO
│   └── ...
└── proj
├── data
│   └── data.csv
├── __init__.py
└── ...
In setup.py, test_suite is declared, so when I call test from proj first
folder, this work well::
~/proj$ python setup.py test
running test
...
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
I would like to use in my unitest the data.csv file from: ~/venv/lib/python2.7/site-packages/proj-1.0-py2.7.egg/proj/data/data.csv.
In short a way to get ~/venv/lib/python2.7/site-packages/proj-1.0-py2.7.egg/proj/data/ folder.
With this code::
$ more tests/lib_test.py
import unittest, os, proj
from proj.lib import get_data
class TestUtils(unittest.TestCase):
def test_get_data(self):
datapath = os.path.dirname(os.path.abspath(proj.__file__)) + '/data'
data = '{}/data.csv'.format(datapath)
print '\n my data:', data
if __name__ == '__main__':
unittest.main()
Using __file__ I only succed to get ~/sandbox/proj/proj/data/data.csv::
$ python setup.py test
running test
...
test_get_data (tests.lib_test.TestUtils) ...
my data: ~/sandbox/proj/proj/data/data.csv
ok
----------------------------------------------------------------------
Ran 1 test in 0.000s

In fact
python setup.py test
does not load python package from the place python setup install put it (~/venv/lib/python2.7/site-packages/proj-xxx), but from the place the python setup.py test is run.
That's why with:
import proj, os
os.path.abspath(proj.__file__)
we can't catch the site-package folder but only the working-copy files.

Python project structure for selenium test

I'm a bit rusted with Python (2.7) modules & packages so I hope to find some help. I am writing some tests using selenium. I would like to organize the tests by "scenario families" and I'd like to implement a helper class to handle some boilerplate that would be my base class for all the tests.
Basically I'd like to have a structure looking like:
.
├── assets
│ └── ressource.ext
├── tests
│ ├── __init__.py
│ ├── user
│ │ ├── __init__.py
│ │ └── upload.py
│ └── visitor
│ ├── __init__.py
│ ├── homepage.py
│ ├── login.py
│ ├── search.py
│ └── signup.py
└── utils
├── __init__.py
└── base.py
I am completely stuck on how to make the utils.base module visible to e.g. the tests.visitor.signup module and still allow to run this specific test using python tests/visitor/signup.py.
How would I do that?
Thanks!
Edit: to make things easier here is a dummy sample of what I'm trying to do:
mkdir -p {utils,tests/user}
touch {utils,tests{,/user}}/__init__.py
echo -e "import unittest\n\nclass Base(unittest.TestCase):\n pass" > utils/base.py
echo -e "from utils.base import Base\n\nclass MyTest(Base):\n pass\n\nif __name__ == '__main__':\n unittest.main()" > tests/user/upload.py
This produces the following tree with empty __init__.py files and the import I'm trying to achieve in tests/user/upload.py:
dummy/
├── tests
│   ├── __init__.py
│   └── user
│   ├── __init__.py
│   └── upload.py
└── utils
├── __init__.py
└── base.py
Now if I am in the dummy/ folder I am getting this:
(env)dummy $ python tests/user/upload.py
Traceback (most recent call last):
File "tests/user/upload.py", line 1, in <module>
from utils.base import Base
ImportError: No module named utils.base
But if I run it interactively there is obviously no problem:
(env) dummy $ python
Python 2.7.6 (default, Jan 16 2014, 16:39:48)
>>> from utils.base import Base
>>>
My problem must be really silly but I can't see what I'm doing wrong. And what I don't get also is that specifying the python path ((env) dummy $ PYTHON_PATH=. python tests/user/upload.py) doesn't fix the issue.

You can set PYTHONPATH to search the current directory:
PYTHONPATH=. python tests/user/upload.py
Or you can use a test runner, like nose:
nosetests tests/user/upload.py
Or you can create your own test runner if you don't want to use nose. For instance a runtests.py file with:
import unittest
from tests.user.upload import *
unittest.main()
Then:
python runtests.py
This test runner could import more tests or could be selective about what tests it imports. Ultimately, I'd recommend using nose over writing your own test runner.
Starting a test runner at the top of your hierarchy works because Python adds the current directory of a script to the list of paths it searches for modules.