How can I generate test report using pytest? I searched for it but whatever i got was about coverage report.
I tried with this command:
py.test sanity_tests.py --cov=C:\Test\pytest --cov-report=xml
But as parameters represents it generates coverage report not test report.
Ripped from the comments: you can use the --junitxml argument.
$ py.test sample_tests.py --junitxml=C:\path\to\out_report.xml
You can use a pytest plugin 'pytest-html' for generating html reports which can be forwarded to different teams as well
First install the plugin:
$ pip install pytest-html
Second, just run your tests with this command:
$ pytest --html=report.html
You can also make use of the hooks provided by the plugin in your code.
import pytest
from py.xml import html
def pytest_html_report_title(report)
report.title = "My very own title!"
Reference: https://pypi.org/project/pytest-html/
I haven't tried it yet but you can try referring to https://github.com/pytest-dev/pytest-html. A python library that can generate HTML output.
py.test --html=Report.html
Here you can specify your python file as well. In this case, when there is no file specified it picks up all the files with a name like 'test_%' present in the directory where the command is run and executes them and generates a report with the name Report.html
You can also modify the name of the report accordingly.
Related
I am trying to test views and models for Django REST API written in pycharm and have installed pytest for this. I have written some tests and when I wanted to start them I got the following error message:
ERROR: usage: _jb_pytest_runner.py [options] [file_or_dir] [file_or_dir] [...]
_jb_pytest_runner.py: error: unrecognized arguments: --cov=frontend --cov-report=html
I have then checked if I have installed pytest properly and it seems I have. I have both Python 2.7.16 as well as Python 3.9.6 installed but am using Python 3. Could this be a compatibility problem or is it something else?
I have tried starting the tests both through the terminal using py.test and just in the IDE itself. I keep getting this same error message.
I have tried the approach below:
py.test: error: unrecognized arguments: --cov=ner_brands --cov-report=term-missing --cov-config
yet I seem to get the same error.
ERROR: usage: _jb_pytest_runner.py [options] [file_or_dir] [file_or_dir] [...]
_jb_pytest_runner.py: error: unrecognized arguments: --cov=frontend --cov-report=html
Does anyone know how I could solve this issue?
Thanks in advance.
First of all, yes, Python 3.9.6 is compatible with pytest 6.2.5, however, you appear to be missing a few dependencies. pytest is one of many different Python packages, and you appear to have installed that successfully, so you're halfway there.
There are a few different coverage plugins that work with pytest, and those need to be installed separately. Here are the two most common coverage plugins for Python and pytest:
https://pypi.org/project/coverage/
https://pypi.org/project/pytest-cov/
The first one, coverage is installed with:
pip install coverage
The second one, pytest-cov is installed with:
pip install pytest-cov
Based on your run command, you appear to want to use pytest-cov. After you've installed that, you can verify that pytest has those new options by calling pytest --help:
> pytest --help
...
coverage reporting with distributed testing support:
--cov=[SOURCE] Path or package name to measure during execution (multi-allowed). Use --cov= to
not do any source filtering and record everything.
--cov-reset Reset cov sources accumulated in options so far.
--cov-report=TYPE Type of report to generate: term, term-missing, annotate, html, xml (multi-
allowed). term, term-missing may be followed by ":skip-covered". annotate, html
and xml may be followed by ":DEST" where DEST specifies the output location.
Use --cov-report= to not generate any output.
--cov-config=PATH Config file for coverage. Default: .coveragerc
--no-cov-on-fail Do not report coverage if test run fails. Default: False
--no-cov Disable coverage report completely (useful for debuggers). Default: False
--cov-fail-under=MIN Fail if the total coverage is less than MIN.
...
Alternatively, you might be able to get the same results you're looking for using coverage:
coverage run -m pytest
coverage html
coverage report
And that will also give you a coverage report even if not using pytest-cov options.
In pytest-cov documentation it says:
Note that this plugin controls some options and setting the option in
the config file will have no effect. These include specifying source
to be measured (source option) and all data file handling (data_file
and parallel options).
However it doesn't say how to change these options. Is there a way to change it (parallel=True)?
I want to change this because after coverage is upgraded from < 5 to latest (5.1) I got these:
Failed to generate report: Couldn't use data file '/path/to/jenkins/workspace/pr/or/branch/.coverage': no such table: line_bits
Note: using coverage < 5 do not have this problem
I have also tried adding .coveragerc with the following but still get the same issue.
[run]
parallel = True
The way it is run in jenkins:
pytest ./tests --mpl -n 4 \
--junitxml=pyTests.xml --log-cli-level=DEBUG -s \
--cov=. --cov-report --cov-report html:coverage-reports
This is due to pytest-cov using coverage combine, which combines all coverage results: In parallel it mixes results from other runs, that may or may not be completed, and in any cases are irrelevant.
I think if you're having the issue, it may be because you're running multiple tests in parallel, like multiple versions of Python.
In which case it's easily solved by specifying a unique COVERAGE_FILE for each run, like:
export COVERAGE_FILE=.coverage.3.7
for the Python 3.7 run, an so on.
See: https://github.com/nedbat/coveragepy/issues/883#issuecomment-650562896
I am trying to run a specific test method with pytest on Python 3.6 with the following command:
pytest sanity_test.py::test_check
(as mentioned in pytest docs)
and it fails with the following error message:
ERROR: not found: <rootdir>/tests/automation/sanity/sanity_test.py::test_check
(no name '<rootdir>/tests/automation/sanity/sanity_test.py::test_check' in any of [<DoctestModule 'tests/automation/sanity/sanity_test.py'>, <Module
'tests/automation/sanity/sanity_test.py'>])
when I run the whole test file the tests are running properly with:
pytest sanity_test.py
When I tried running a specific method in isolated project it works perfectly.
Any reason why pytest can't find specifically my test methods on my project?
The file is very simple and looks like this:
def test_check():
pass
Thank to hoefling, I recognized that I had --doctest-modules option in my setup.cfg file. Removing this option fixed the problem.
hoefling opened an issue in pytest for that problem.
I'd like to only run selenium tests in my test suite, in addition to filtering it down to only run tests in a specific file/folder. It seems like I should be able to accomplish this with the -m option, and the path positional argument. Furthermore, I'm doing this in a bash script.
So for example, I tried something like this:
#!/bin/bash
# ...some logic here for determining `EXTRA` arg
EXTRA = "not selenium"
py.test -m $EXTRA -v -s --tb=long --no-flaky-report ~/project/mytests/test_blerg.py
And then my test looks like this (still using xunit-style classes):
#pytest.mark.selenium
class BaseTest(UnitTest):
pass
class ChildTest(BaseTest):
def test_first_case(self):
pass
When I run the py.test command as I described above, I get this:
============================================================================ no tests ran in 0.01 seconds ============================================================================
ERROR: file not found: selenium"
Not completely sure why this doesn't work. I'll try manually overriding pytest_runtest_setup(), but I'm feel like I should be able to accomplish what I want without doing that. Also, just FYI, this is a django project, using Django==1.8.7 and pytest-django==2.9.1.
Any help would be greatly appreciated :)
Figured it out. This has nothing to do with py.test itself. I had an error in how I was calling the py.test command in my bash script. The amended command looks like this:
py.test -m "$EXTRA" -v -s --tb=long --no-flaky-report ~/project/mytests/test_blerg.py
Works as expected!
I have a problem running nose tests and get results inside Jenkins.
The job has a shell script like this:
. /var/lib/jenkins/envs/itravel/bin/activate
python api/manage.py syncdb --noinput
DJANGO_SETTINGS_MODULE=ci_settings nosetests --verbosity=0 --processes=1 --with-xunit --xunit-file=nosetests.xml
deactivate
Part of the test suite is run using the django_nose.NoseTestSuiteRunner.
All the tests are run and the resulting nosetests.xml file is created but does not seem to be filled with the tests results:
<?xml version="1.0" encoding="UTF-8"?><testsuite name="nosetests" tests="0" errors="0" failures="0" skip="0"></testsuite>
I noticed that on an import Error fail the file is filled with one error, but otherwise, nothing...
Any idea? Is there something special to do from the tests side? Any property to set or so?
Thanks.
As far as I know, the --processes option is not compatible with --with-xunit. When you ask nosetests to run with the processes plugin, the tests are run in specified number of subprocesses. The xunit plugin does not know how to gather results into the xml file.
Just remove the --processes option and you should be fine.
Nose has had an open and unresolved GitHub issue for this since 2011. As #sti said, everything works fine if you don't use --processes. For everyone else, consider using Ignas/nose_xunitmp instead:
pip install nose_xunitmp
nosetests --with-xunitmp
nosetests --xunitmp-file results.xml