When I run my testsuite with parallel executions I receive different results compared to run only one test runner.
I tried two approaches and both did not provide proper results. These commands did either only produce one partial result or less than we had runners.
I've tried with coverage and pytest combined:
COVERAGE_PROCESS_START=./my_app coverage --parallel-mode --concurrency=multiprocessing run -m pytest -m "not e2e" -n 4
Also with pytest and pytest-cov:
pytest -m "not e2e" -n 4 --cov=my_app
The second one also had the issue that some templatetags were not seen as registered even though others in the same directory were registered.
After running these I executed coverage combine and coverage report. When run in parallel the results are always incomplete compared to running it with only one test runner, which works perfectly fine:
coverage run -m pytest -m "not e2e"
This is my coveragerc:
[run]
include = my_app/*
omit = *migrations*, *tests*
plugins =
django_coverage_plugin
[report]
show_missing = true
Does some know how to get proper coverage results when running pytest in parallel?
Related
I am trying to test views and models for Django REST API written in pycharm and have installed pytest for this. I have written some tests and when I wanted to start them I got the following error message:
ERROR: usage: _jb_pytest_runner.py [options] [file_or_dir] [file_or_dir] [...]
_jb_pytest_runner.py: error: unrecognized arguments: --cov=frontend --cov-report=html
I have then checked if I have installed pytest properly and it seems I have. I have both Python 2.7.16 as well as Python 3.9.6 installed but am using Python 3. Could this be a compatibility problem or is it something else?
I have tried starting the tests both through the terminal using py.test and just in the IDE itself. I keep getting this same error message.
I have tried the approach below:
py.test: error: unrecognized arguments: --cov=ner_brands --cov-report=term-missing --cov-config
yet I seem to get the same error.
ERROR: usage: _jb_pytest_runner.py [options] [file_or_dir] [file_or_dir] [...]
_jb_pytest_runner.py: error: unrecognized arguments: --cov=frontend --cov-report=html
Does anyone know how I could solve this issue?
Thanks in advance.
First of all, yes, Python 3.9.6 is compatible with pytest 6.2.5, however, you appear to be missing a few dependencies. pytest is one of many different Python packages, and you appear to have installed that successfully, so you're halfway there.
There are a few different coverage plugins that work with pytest, and those need to be installed separately. Here are the two most common coverage plugins for Python and pytest:
https://pypi.org/project/coverage/
https://pypi.org/project/pytest-cov/
The first one, coverage is installed with:
pip install coverage
The second one, pytest-cov is installed with:
pip install pytest-cov
Based on your run command, you appear to want to use pytest-cov. After you've installed that, you can verify that pytest has those new options by calling pytest --help:
> pytest --help
...
coverage reporting with distributed testing support:
--cov=[SOURCE] Path or package name to measure during execution (multi-allowed). Use --cov= to
not do any source filtering and record everything.
--cov-reset Reset cov sources accumulated in options so far.
--cov-report=TYPE Type of report to generate: term, term-missing, annotate, html, xml (multi-
allowed). term, term-missing may be followed by ":skip-covered". annotate, html
and xml may be followed by ":DEST" where DEST specifies the output location.
Use --cov-report= to not generate any output.
--cov-config=PATH Config file for coverage. Default: .coveragerc
--no-cov-on-fail Do not report coverage if test run fails. Default: False
--no-cov Disable coverage report completely (useful for debuggers). Default: False
--cov-fail-under=MIN Fail if the total coverage is less than MIN.
...
Alternatively, you might be able to get the same results you're looking for using coverage:
coverage run -m pytest
coverage html
coverage report
And that will also give you a coverage report even if not using pytest-cov options.
In pytest-cov documentation it says:
Note that this plugin controls some options and setting the option in
the config file will have no effect. These include specifying source
to be measured (source option) and all data file handling (data_file
and parallel options).
However it doesn't say how to change these options. Is there a way to change it (parallel=True)?
I want to change this because after coverage is upgraded from < 5 to latest (5.1) I got these:
Failed to generate report: Couldn't use data file '/path/to/jenkins/workspace/pr/or/branch/.coverage': no such table: line_bits
Note: using coverage < 5 do not have this problem
I have also tried adding .coveragerc with the following but still get the same issue.
[run]
parallel = True
The way it is run in jenkins:
pytest ./tests --mpl -n 4 \
--junitxml=pyTests.xml --log-cli-level=DEBUG -s \
--cov=. --cov-report --cov-report html:coverage-reports
This is due to pytest-cov using coverage combine, which combines all coverage results: In parallel it mixes results from other runs, that may or may not be completed, and in any cases are irrelevant.
I think if you're having the issue, it may be because you're running multiple tests in parallel, like multiple versions of Python.
In which case it's easily solved by specifying a unique COVERAGE_FILE for each run, like:
export COVERAGE_FILE=.coverage.3.7
for the Python 3.7 run, an so on.
See: https://github.com/nedbat/coveragepy/issues/883#issuecomment-650562896
I would like to be able to run some python unit tests apart from other tests by means of CTest and the following command:
make unit_tests
I tried the following combination but it does not work:
ADD_TEST(unit_test_1 ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/unit_test_1.py --verbose)
ADD_TEST(unit_test_2 ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/unit_test_2.py --verbose)
ADD_CUSTOM_TARGET(unit_tests COMMAND ${CMAKE_CTEST_COMMAND}
DEPENDS unit_test_1 unit_test_2)
Do you know how to do it?
This works for me (I replaced the test commands with some dummy statements, but adjusting it to invoke python should be doable):
cmake_minimum_required(VERSION 3.11)
enable_testing()
add_test(unit_test_1 echo "Unit test 1")
add_test(unit_test_2 echo "Unit test 2")
add_custom_target(unit_tests COMMAND ${CMAKE_CTEST_COMMAND})
No need to add any dependencies to the unit_tests target. By default, ctest runs all tests.
How can I generate test report using pytest? I searched for it but whatever i got was about coverage report.
I tried with this command:
py.test sanity_tests.py --cov=C:\Test\pytest --cov-report=xml
But as parameters represents it generates coverage report not test report.
Ripped from the comments: you can use the --junitxml argument.
$ py.test sample_tests.py --junitxml=C:\path\to\out_report.xml
You can use a pytest plugin 'pytest-html' for generating html reports which can be forwarded to different teams as well
First install the plugin:
$ pip install pytest-html
Second, just run your tests with this command:
$ pytest --html=report.html
You can also make use of the hooks provided by the plugin in your code.
import pytest
from py.xml import html
def pytest_html_report_title(report)
report.title = "My very own title!"
Reference: https://pypi.org/project/pytest-html/
I haven't tried it yet but you can try referring to https://github.com/pytest-dev/pytest-html. A python library that can generate HTML output.
py.test --html=Report.html
Here you can specify your python file as well. In this case, when there is no file specified it picks up all the files with a name like 'test_%' present in the directory where the command is run and executes them and generates a report with the name Report.html
You can also modify the name of the report accordingly.
I have a problem running nose tests and get results inside Jenkins.
The job has a shell script like this:
. /var/lib/jenkins/envs/itravel/bin/activate
python api/manage.py syncdb --noinput
DJANGO_SETTINGS_MODULE=ci_settings nosetests --verbosity=0 --processes=1 --with-xunit --xunit-file=nosetests.xml
deactivate
Part of the test suite is run using the django_nose.NoseTestSuiteRunner.
All the tests are run and the resulting nosetests.xml file is created but does not seem to be filled with the tests results:
<?xml version="1.0" encoding="UTF-8"?><testsuite name="nosetests" tests="0" errors="0" failures="0" skip="0"></testsuite>
I noticed that on an import Error fail the file is filled with one error, but otherwise, nothing...
Any idea? Is there something special to do from the tests side? Any property to set or so?
Thanks.
As far as I know, the --processes option is not compatible with --with-xunit. When you ask nosetests to run with the processes plugin, the tests are run in specified number of subprocesses. The xunit plugin does not know how to gather results into the xml file.
Just remove the --processes option and you should be fine.
Nose has had an open and unresolved GitHub issue for this since 2011. As #sti said, everything works fine if you don't use --processes. For everyone else, consider using Ignas/nose_xunitmp instead:
pip install nose_xunitmp
nosetests --with-xunitmp
nosetests --xunitmp-file results.xml