We Keep Coding

How do I use pytest with bazel?

I have a my_module.py file that implements my_module and a file test_my_module.py that does import my_module and runs some tests written with pytest on it.
Normally I run the tests by cding into the directory that contains these two files and then doing
pytest
Now I want to use Bazel. I've added my_module.py as a py_binary but I don't know what the right way to invoke my tests is.

If you want to create a reusable code, that don't need add call to pytest add end of every python file with test. You can create py_test call that call a python file wrapping a call to pytest and keeping all argument. Then create a macro around the py_test. I explain the detailed solution in Experimentations on Bazel: Python (3), linter & pytest, with link to source code.
Create the python tool (wrapp call to pytest, or only pylint) in tools/pytest/pytest_wrapper.py
import sys
import pytest
# if using 'bazel test ...'
if __name__ == "__main__":
sys.exit(pytest.main(sys.argv[1:]))
Create the macro in tools/pytest/defs.bzl
"""Wrap pytest"""
load("#rules_python//python:defs.bzl", "py_test")
load("#my_python_deps//:requirements.bzl", "requirement")
def pytest_test(name, srcs, deps = [], args = [], data = [], **kwargs):
"""
Call pytest
"""
py_test(
name = name,
srcs = [
"//tools/pytest:pytest_wrapper.py",
] + srcs,
main = "//tools/pytest:pytest_wrapper.py",
args = [
"--capture=no",
"--black",
"--pylint",
"--pylint-rcfile=$(location //tools/pytest:.pylintrc)",
# "--mypy",
] + args + ["$(location :%s)" % x for x in srcs],
python_version = "PY3",
srcs_version = "PY3",
deps = deps + [
requirement("pytest"),
requirement("pytest-black"),
requirement("pytest-pylint"),
# requirement("pytest-mypy"),
],
data = [
"//tools/pytest:.pylintrc",
] + data,
**kwargs
)
expose some resources from tools/pytest/BUILD.bazel
exports_files([
"pytest_wrapper.py",
".pylintrc",
])
Call it from your package BUILD.bazel
load("//tools/pytest:defs.bzl", "pytest_test")
...
pytest_test(
name = "test",
srcs = glob(["*.py"]),
deps = [
...
],
)
then calling bazel test //... means that pylint, pytest and black are all part of the test flow.

Add the following code to test_my_module.py and mark the test script as a py_test instead of py_binary in your BUILD file:
if __name__ == "__main__":
import pytest
raise SystemExit(pytest.main([__file__]))
Then you can run your tests with bazel test test_my_module

Following on from #David Bernard, who wrote his answer in an awesome series of blog posts BTW, there is a curve-ball there with pytest + bazel + Windows...
Long story short, you'll need to add legacy_create_init = 0 to the py_test rule call.
This is a workaround a "feature" where bazel will create __init__.py files in the sandbox, even when none were present in your repo https://github.com/bazelbuild/rules_python/issues/55

It seems a bunch of the suggestions here have been packaged up into https://github.com/caseyduquettesc/rules_python_pytest now.
load("#rules_python_pytest//python_pytest:defs.bzl", "py_pytest_test")
py_pytest_test(
name = "test_w_pytest",
size = "small",
srcs = ["test.py"],
deps = [
# TODO Add this for the user
requirement("pytest"),
],
)
Edit: I'm the author of the above repository

How do you properly integrate unit tests for file parsing with pytest?

I'm trying to test file parsing with pytest. I have a directory tree that looks something like this for my project:
project
project/
cool_code.py
setup.py
setup.cfg
test/
test_read_files.py
test_files/
data_file1.txt
data_file2.txt
My setup.py file looks something like this:
from setuptools import setup
setup(
name = 'project',
description = 'The coolest project ever!',
setup_requires = ['pytest-runner'],
tests_require = ['pytest'],
)
My setup.cfg file looks something like this:
[aliases]
test=pytest
I've written several unit tests with pytest to verify that files are properly read. They work fine when I run pytest from within the "test" directory. However, if I execute any of the following from my project directory, the tests fail because they cannot find data files in test_files:
>> py.test
>> python setup.py pytest
The test seems to be sensitive to the directory from which pytest is executed.
How can I get pytest unit tests to discover the files in "data_files" for parsing when I call it from either the test directory or the project root directory?

One solution is to define a rootdir fixture with the path to the test directory, and reference all data files relative to this. This can be done by creating a test/conftest.py (if not already created) with some code like this:
import os
import pytest
#pytest.fixture
def rootdir():
return os.path.dirname(os.path.abspath(__file__))
Then use os.path.join in your tests to get absolute paths to test files:
import os
def test_read_favorite_color(rootdir):
test_file = os.path.join(rootdir, 'test_files/favorite_color.csv')
data = read_favorite_color(test_file)
# ...

One solution is to try multiple paths to find the files.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from coolprogram import *
import os
def test_file_locations():
"""Possible locations where test data could be found."""
return(['./test_files',
'./tests/test_files',
])
def find_file(filename):
""" Searches for a data file to use in tests """
for location in test_file_locations():
filepath = os.path.join(location, filename)
if os.path.exists(filepath):
return(filepath)
raise IOError('Could not find test file.')
def test_read_favorite_color():
""" Test that favorite color is read properly """
filename = 'favorite_color.csv'
test_file = find_file(filename)
data = read_favorite_color(test_file)
assert(data['first_name'][1] == 'King')
assert(data['last_name'][1] == 'Arthur')
assert(data['correct_answers'][1] == 2)
assert(data['cross_bridge'][1] == True)
assert(data['favorite_color'][1] == 'green')

One way is to pass a dictionary of command name and custom command class to cmdclass argument of setup function.
Another way is like here, posted it here for quick reference.
pytest-runner will install itself on every invocation of setup.py. In some cases, this causes delays for invocations of setup.py that will never invoke pytest-runner. To help avoid this contingency, consider requiring pytest-runner only when pytest is invoked:
pytest = {'pytest', 'test', 'ptr'}.intersection(sys.argv)
pytest_runner = ['pytest-runner'] if needs_pytest else []
# ...
setup(
#...
setup_requires=[
#... (other setup requirements)
] + pytest_runner,
)
Make sure all the data you read in your test module is relative to the location of setup.py directory.
In OP's case data file path would be test/test_files/data_file1.txt,
I made a project with same structure and read the data_file1.txt with some text in it and it works for me.

Python - how to make python package with argparse?

I wrote my project in python and now I want to pack it into python package like here : Python packaging tutorial. The thing is I don't know how to properly pack my main.py ( I named it __ init __.py same like in tutorial) where the arg parse is being used. My file structure currently looks like this:
information_analyzer/
information_analyzer/
__init__.py
ArticleFile.py
ArticleInspector.py
SearchEngine.py
setup.py
.gitignore
README.md
Here is my setup.py:
from setuptools import setup
setup(name='information_analyzer',
version='0.3',
description='Phrase and article analyzer',
url='https://github.com/my_repo_link',
author='Name Surname',
author_email='mail#gmail.com',
license='MIT',
packages=['information_analyzer'],
include_package_data=True,
zip_safe=False)
Here is my __ init __.py:
import argparse
import sys
from information_analyzer.information_analyzer.ResultsChecker import check_result
from information_analyzer.information_analyzer.SearchEngine import SearchEngine
from information_analyzer.information_analyzer.ArticleInspector import ArticleInspector
def main():
parser = argparse.ArgumentParser(description='Information analyzer')
parser.add_argument('--phrase', default=None, type=str, help='Phrase for verification')
parser.add_argument('--articleURL', default=None, type=str, help='URL to article')
args = parser.parse_args()
articleURL=args.articleURL # for example http://fedsalert.com/donald-trump-dead-from-a-fatal-heart-attack/
phrase=args.phrase
if(phrase is None and articleURL is None):
print("You must specify phrase or article URL to run the program.")
sys.exit()
if phrase is not None:
search_engine = SearchEngine()
check_result(phrase, search_engine)
if articleURL is not None:
article_inspector=ArticleInspector(articleURL)
article_inspector.print_all_informations()
When I installed pip install . and tried to import it in python console I got that info:
>>> import information_analyzer
>>> information_analyzer.main()
AttributeError: module 'information-analyzer' has no attribute 'main'

How to add a custom nose plugin to the `nosetests` command

So I'm very noob in dealing with nose plugins.
I've been searching a lot but docs regarding nose plugins seem scarce.
I read and tried what's in the following links to try to write a simple nose plugin
and run it with nosetests, without success:
https://nose.readthedocs.org/en/latest/doc_tests/test_init_plugin/init_plugin.html
https://nose.readthedocs.org/en/latest/plugins/writing.html
I don't want to write my own test-runner or run the tests from any other script (via run(argv=argv, suite=suite(), ...)),
like they do in the first link.
I wrote a file myplugin.py with a class like this:
import os
from nose.plugins import Plugin
class MyCustomPlugin(Plugin):
name = 'myplugin'
def options(self, parser, env=os.environ):
parser.add_option('--custom-path', action='store',
dest='custom_path', default=None,
help='Specify path to widget config file')
def configure(self, options, conf):
if options.custom_path:
self.make_some_configs(options.custom_path)
self.enabled = True
def make_some_configs(self, path):
# do some stuff based on the given path
def begin(self):
print 'Maybe print some useful stuff...'
# do some more stuff
and added a setup.py like this:
try:
from setuptools import setup, find_packages
except ImportError:
import distribute_setup
distribute_setup.use_setuptools()
from setuptools import setup, find_packages
setup(
name='mypackage',
...
install_requires=['nose==1.3.0'],
py_modules=['myplugin'],
entry_points={
'nose.plugins.1.3.0': [
'myplugin = myplugin:MyCustomPlugin'
]
}
)
Both files are in the same directory.
Every time I run nosetests --custom-path [path], I get:
nosetests: error: no such option: --custom-path
From the links mentioned above, I thought that's all that was required to register and enable a custom plugin.
But it seems that, either I'm doing something really wrong, or nose's docs are outdated.
Can someone please point me the correct way to register and enable a plugin, that I can use with nosetests?
Thanks a lot!! :)

You don't want the nose version in entry_points in setup.py. Just use nose.plugins.0.10 as the docs say. The dotted version in the entry point name is not so much a nose version as a plugin API version.

Distribute, epydoc, and setup.py

I would like to have a target (say docs) that runs epydoc for my package. I assume that I need to create a new command but I am not having much luck.
Has anyone done this before?

The Babel project provides several commands for use in setup.py files.
You need to define a distutils.commands entry point with commands; example from the Babel setup.py file:
entry_points = """
[distutils.commands]
compile_catalog = babel.messages.frontend:compile_catalog
extract_messages = babel.messages.frontend:extract_messages
init_catalog = babel.messages.frontend:init_catalog
update_catalog = babel.messages.frontend:update_catalog
"""
where the extra commands are then available as python setup.py commandname.
The entry points point to subclasses of from distutils.cmd import Command. Example again from Babel, from the babel.messages.frontend module:
from distutils.cmd import Command
from distutils.errors import DistutilsOptionError
class compile_catalog(Command):
"""Catalog compilation command for use in ``setup.py`` scripts."""
# Description shown in setup.py --help-commands
description = 'compile message catalogs to binary MO files'
# Options available for this command, tuples of ('longoption', 'shortoption', 'help')
# If the longoption name ends in a `=` it takes an argument
user_options = [
('domain=', 'D',
"domain of PO file (default 'messages')"),
('directory=', 'd',
'path to base directory containing the catalogs'),
# etc.
]
# Options that don't take arguments, simple true or false options.
# These *must* be included in user_options too, but without a = equals sign
boolean_options = ['use-fuzzy', 'statistics']
def initialize_options(self):
# Set a default for each of your user_options (long option name)
def finalize_options(self):
# verify the arguments and raise DistutilOptionError if needed
def run(self):
# Do your thing here.