Ok, I'm struggling with something that is literally blowing my mind.
Although my actual code is different, this basically nails down the issue. Assume this sample code:
import pytest
#pytest.mark.parametrize('type',(
pytest.param('stability', marks=pytest.mark.stability),
pytest.param('integration', marks=pytest.mark.integration),
))
#pytest.mark.integration
#pytest.mark.stability
def test_meh(type):
assert type == 'integration'
And the following is the output of running that test.
$pytest -m integration test_meeh.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.6.4, pytest-3.1.2, py-1.5.2, pluggy-0.4.0
rootdir: /Users/yzT/Desktop, inifile:
collected 2 items
test_meeh.py F
============================================================== FAILURES ===============================================================
_________________________________________________________ test_meh[stability] _________________________________________________________
type = 'stability'
#pytest.mark.parametrize('type',(
pytest.param('stability', marks=pytest.mark.stability),
pytest.param('integration', marks=pytest.mark.integration),
))
#pytest.mark.integration
#pytest.mark.stability
def test_meh(type):
> assert type == 'integration'
E AssertionError: assert 'stability' == 'integration'
E - stability
E + integration
test_meeh.py:10: AssertionError
========================================================= 1 tests deselected ==========================================================
=============================================== 1 failed, 1 deselected in 0.07 seconds ================================================
$ pytest -m stability test_meeh.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.6.4, pytest-3.1.2, py-1.5.2, pluggy-0.4.0
rootdir: /Users/yzT/Desktop, inifile:
collected 2 items
test_meeh.py .
========================================================= 1 tests deselected ==========================================================
=============================================== 1 passed, 1 deselected in 0.02 seconds ================================================
What is going on here? Why when I use -m integration it uses stability and when I use -m stability it uses integration ?
Looks like there are two issues with the code sample that you provided.
First, you shouldn't mark both individual parametrized tests and the test function. As soon as the -m option you provided matches the mark decorator on the function, the test will be selected.
Here's a minimal comparison. With both the marks on the function and individual parametrized tests:
# parametrized_tests.py
import pytest
#pytest.mark.parametrize('smiley', [
pytest.param(':)', marks=[pytest.mark.happy]),
pytest.param(':(', marks=[pytest.mark.unhappy]),
])
#pytest.mark.happy
#pytest.mark.unhappy
def test_meh(smiley):
assert smiley == ':)'
You will have two tests collected and selected:
$ pytest -m happy --collect-only parametrized_tests.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: [redacted], inifile:
collected 2 items
<Module 'params-individual-tests.py'>
<Function 'test_smiley[:)]'>
<Function 'test_smiley[:(]'>
====================================================================================== no tests ran in 0.01 seconds ======================================================================================
But if only the parametrized tests are marked:
# parametrized_tests.py
import pytest
#pytest.mark.parametrize('smiley', [
pytest.param(':)', marks=[pytest.mark.happy]),
pytest.param(':(', marks=[pytest.mark.unhappy]),
])
def test_smiley(smiley):
assert smiley == ':)'
You will get two tests collected, but only one selected, as you expect:
$ pytest -m happy --collect-only parametrized_tests.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: [redacted], inifile:
collected 2 items / 1 deselected
<Module 'params-individual-tests.py'>
<Function 'test_smiley[:)]'>
====================================================================================== 1 deselected in 0.01 seconds ======================================================================================
Second, there seems to be a puzzling bug (or undocumented "feature") in pytest.param, where using the exact name of the mark as the parameter value makes the test not selected.
From your (slightly modified) code:
# mark_name.py
import pytest
#pytest.mark.parametrize('type_', [
pytest.param('integration', marks=[pytest.mark.integration]),
pytest.param('stability', marks=[pytest.mark.stability]),
])
def test_meh(type_):
assert type_ == 'integration'
If I try to run the integration tests only, it won't select any:
$ pytest -m integration mark_name.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: [redacted], inifile:
collected 2 items / 2 deselected
But simply modifying the value (in this case I only made it uppercase) makes everything work as expected:
# mark_name.py
import pytest
#pytest.mark.parametrize('type_', [
pytest.param('INTEGRATION', marks=[pytest.mark.integration]),
pytest.param('STABILITY', marks=[pytest.mark.stability]),
])
def test_meh(type_):
assert type_ == 'INTEGRATION'
And now I can select the tests properly:
$ pytest -m integration mark_name.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: [redacted], inifile:
collected 2 items / 1 deselected
mark_name.py . [100%]
================================================================================= 1 passed, 1 deselected in 0.01 seconds =================================================================================
$ pytest -m stability mark_name.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: [redacted], inifile:
collected 2 items / 1 deselected
mark_name.py F [100%]
================================================================================================ FAILURES ================================================================================================
__________________________________________________________________________________________ test_meh[STABILITY] ___________________________________________________________________________________________
type_ = 'STABILITY'
#pytest.mark.parametrize('type_', [
pytest.param('INTEGRATION', marks=[pytest.mark.integration]),
pytest.param('STABILITY', marks=[pytest.mark.stability]),
])
def test_meh(type_):
> assert type_ == 'INTEGRATION'
E AssertionError: assert 'STABILITY' == 'INTEGRATION'
E - STABILITY
E + INTEGRATION
mark_name.py:9: AssertionError
================================================================================= 1 failed, 1 deselected in 0.08 seconds =================================================================================
I suspect that's related to the string value being used as the ID of the test, but would suggest opening a GitHub issue if that's something you don't want to work around.
Also, totally unrelated, but it's better not to define the argument as type, because you are shadowing the builtin function type. PEP-8 suggests using a trailing underscore to prevent a name clash.
Related
Is the following nested structure discoverable by unittest ?
class HerclTests(unittest.TestCase):
def testJobs(self):
def testJobSubmit():
jid = "foobar"
assert jid,'hercl job submit failed no job_id'
return jid
def testJobShow(jid):
jid = "foobar"
out,errout=bash(f"hercl job show --jid {jid} --form json")
assert 'Job run has been accepted by airflow successfully' in out,'hercl job show failed'
Here is the error when trying to run unittest :
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.4.3, py-1.10.0, pluggy-0.13.1 -- /Users/steve/git/hercl/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/steve/git/hercl/tests
collecting ... collected 0 items
ERROR: not found: /Users/steve/git/hercl/tests/hercl_flow_test.py::HerclTests::testJobs::testJobSubmit
(no name '/Users/steve/git/hercl/tests/hercl_flow_test.py::HerclTests::testJobs::testJobSubmit' in any of [<TestCaseFunction testJobs>])
============================ no tests ran in 0.01s =============================
Can this structure be tweaked to work with unittest or must each test method be elevated to the level of the HerclTests class?
This can't work - functions defined inside another function ("inner functions") only "exist" as variables inside the outer function's local scope. They're not accessible to any other code. The unittest discovery won't find them, and couldn't call them even if it knew about them.
When I call pytest.main(...) in Python, it will display the unit test messages in the output window, such as
============================= test session starts =============================
platform win32 -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: xxx, inifile: pytest.ini
plugins: cov-2.5.1, hypothesis-3.38.5
collected 1 item
..\test_example.py F [100%]
================================== FAILURES ===================================
_______________________________ test_something ________________________________
def test_something():
> assert 1 == 0
E assert 1 == 0
..\test_example.py:5: AssertionError
========================== 1 failed in 0.60 seconds ===========================
My question is simply that how can I get the message above into a string object. The documentation doesn't say anything about this. And the what pytest.main() returns is only an integer that represents the error code.
https://docs.pytest.org/en/latest/usage.html#calling-pytest-from-python-code
I don't know pytest but you can try to redirect the standard output to a file. Something like this:
import sys
sys.stdout = open("log.txt", "w")
After this you can read the file to get the strings. What do you want to do with the strings?
I try to make a simple test with assert statement like this
def sqr(x):
return x**2 + 1
def test_sqr():
assert kvadrat(2) == 4
! pytest
and it returns
If anyone has any ideas what might be wrong.
For pytest to find the tests these have to be in a file in the current directory or a sub-directory and the filename has to begin with test (although that's probably customizable).
So if you use (inside a Jupyter notebook):
%%file test_sth.py
def sqr(x):
return x**2 + 1
def test_sqr():
assert kvadrat(2) == 4
That creates a file named test_sth.py with the given contents.
Then run ! pytest and it should work (well, fail):
============================= test session starts =============================
platform win32 -- Python 3.6.3, pytest-3.3.0, py-1.5.2, pluggy-0.6.0
rootdir: D:\JupyterNotebooks, inifile:
plugins: hypothesis-3.38.5
collected 1 item
test_sth.py F [100%]
================================== FAILURES ===================================
__________________________________ test_sqr ___________________________________
def test_sqr():
> assert kvadrat(2) == 4
E NameError: name 'kvadrat' is not defined
test_sth.py:5: NameError
========================== 1 failed in 0.12 seconds ===========================
I'd like to define a custom marker (my_marker containing a bunch of attributes. While triggering the tests, how do I select/specify a set of tests with a specific attribute?
For e.g., consider the following:
import pytest
#pytest.mark.my_marker(foo='bar')
def test_001():
pass
def test_002():
pass
#pytest.mark.my_marker(foo='bar', cat='dog')
def test_003():
pass
Now, py.test -m my_marker will select test_001 and test_003. I'd like to specify something like:
py.test -s -m "my_marker(cat='dog')"
This should select only test_003. But py.test throws an error with this.
Any suggestions?
Regards
Sharad
I was able to get reasonably close with this:
conftest.py
import ast
import pytest
def pytest_collection_modifyitems(session, config, items):
mymarkerFilter = config.option.markexpr
if not mymarkerFilter:
return
mymarkerFilter = ast.literal_eval(mymarkerFilter).get('my_marker')
if not mymarkerFilter:
return
selected = []
deselected = []
for item in items:
testMymarker = item.get_marker('my_marker')
if not testMymarker:
deselected.append(item)
continue
found = False
for filterKey in mymarkerFilter:
if not testMymarker.kwargs.get(filterKey):
deselected.append(item)
found = True
break
if mymarkerFilter[filterKey] != testMymarker.kwargs.get(filterKey):
deselected.append(item)
found = True
break
if not found:
selected.append(item)
if deselected:
config.hook.pytest_deselected(items=deselected)
items[:] = selected
print('Deselected: {}'.format(deselected))
print('Selected: {}'.format(items))
test_bar.py
import pytest
#pytest.mark.my_marker(foo='bar')
def test_001():
print('test_001')
def test_002():
print('test_002')
#pytest.mark.my_marker(foo='bar', cat='dog')
def test_003():
print('test_003')
Output
# Without specifying any marker
py.test -vs test_bar.py
Test session starts (platform: linux, Python 3.5.1, pytest 2.8.7, pytest-sugar 0.5.1)
cachedir: .cache
rootdir: /home/sharad/tmp, inifile:
plugins: cov-2.2.1, timeout-1.0.0, sugar-0.5.1, json-0.4.0, html-1.8.0, spec-1.0.1
test_bar.pytest_001 0% test_001
test_bar.pytest_001 ✓ 33% ███▍
test_bar.pytest_002 33% ███▍ test_002
test_bar.pytest_002 ✓ 67% ██████▋
test_bar.pytest_003 67% ██████▋ test_003
test_bar.pytest_003 ✓ 100% ██████████
Results (0.06s):
3 passed
# Specifying foo=bar
py.test -vs -m "{'my_marker': {'foo': 'bar'}}" test_bar.py
Test session starts (platform: linux, Python 3.5.1, pytest 2.8.7, pytest-sugar 0.5.1)
cachedir: .cache
rootdir: /home/sharad/tmp, inifile:
plugins: cov-2.2.1, timeout-1.0.0, sugar-0.5.1, json-0.4.0, html-1.8.0, spec-1.0.1
Deselected: [<Function 'test_002'>]
Selected: [<Function 'test_001'>, <Function 'test_003'>]
test_bar.pytest_001 0% test_001
test_bar.pytest_001 ✓ 50% █████
test_bar.pytest_003 50% █████ test_003
test_bar.pytest_003 ✓ 100% ██████████
=================================================================================== 1 tests deselected by '-m "{\'my_marker\': {\'foo\': \'bar\'}}"' ===================================================================================
Results (0.04s):
2 passed
1 deselected
# Specifying foo=bar, cat=dog
py.test -vs -m "{'my_marker': {'foo': 'bar', 'cat': 'dog'}}" test_bar.py
Test session starts (platform: linux, Python 3.5.1, pytest 2.8.7, pytest-sugar 0.5.1)
cachedir: .cache
rootdir: /home/sharad/tmp, inifile:
plugins: cov-2.2.1, timeout-1.0.0, sugar-0.5.1, json-0.4.0, html-1.8.0, spec-1.0.1
Deselected: [<Function 'test_001'>, <Function 'test_002'>]
Selected: [<Function 'test_003'>]
test_bar.pytest_003 0% test_003
test_bar.pytest_003 ✓ 100% ██████████
========================================================================== 2 tests deselected by '-m "{\'my_marker\': {\'foo\': \'bar\', \'cat\': \'dog\'}}"' ==========================================================================
Results (0.06s):
1 passed
2 deselected
I have this testing code:
import pytest
def params():
dont_skip = pytest.mark.skipif(False, reason="don't skip")
return [dont_skip("foo"), dont_skip("bar")]
#pytest.mark.skipif(True, reason="always skip")
#pytest.mark.parametrize("param", params())
#pytest.mark.skipif(True, reason="really always skip please")
def test_foo(param):
assert False
Yet test_foo is not skipped, even though there are skipif decorators attached to test_foo (I tried in both orders, as you can see above):
============================= test session starts ==============================
platform darwin -- Python 3.5.0, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /Volumes/Home/Users/Waleed/tmp/python/explainerr/test, inifile:
collected 2 items
test/test_example.py FF
=================================== FAILURES ===================================
________________________________ test_foo[foo] _________________________________
param = 'foo'
#pytest.mark.skipif(True, reason="always skip")
#pytest.mark.parametrize("param", params())
#pytest.mark.skipif(True, reason="really always skip")
def test_foo(param):
> assert False
E assert False
test/test_example.py:13: AssertionError
________________________________ test_foo[bar] _________________________________
param = 'bar'
#pytest.mark.skipif(True, reason="always skip")
#pytest.mark.parametrize("param", params())
#pytest.mark.skipif(True, reason="really always skip")
def test_foo(param):
> assert False
E assert False
test/test_example.py:13: AssertionError
=========================== 2 failed in 0.01 seconds ===========================
If I change this line
dont_skip = pytest.mark.skipif(False, reason="don't skip")
to
dont_skip = pytest.mark.skipif(True, reason="don't skip")
then it skips the test cases:
============================= test session starts ==============================
platform darwin -- Python 3.5.0, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /Volumes/Home/Users/Waleed/tmp/python/explainerr/test, inifile:
collected 2 items
test/test_example.py ss
========================== 2 skipped in 0.01 seconds ===========================
How do I get pytest.mark.skipif to work when also using skippable parameters with pytest.mark.parametrize? I'm using Python 3.5.0 with Pytest 2.8.5.
The pytest version that you are using is very old. I used your code in my env and two tests are skipped at first skipif marks.
python3 -m pytest test_b.py
==================================================================================== test session starts =====================================================================================
platform darwin -- Python 3.6.5, pytest-3.9.1, py-1.7.0, pluggy-0.8.0
rootdir:<redacted>, inifile:
collected 2 items
test_b.py ss [100%]
====================================================================================== warnings summary ======================================================================================
test_b.py:9: RemovedInPytest4Warning: Applying marks directly to parameters is deprecated, please use pytest.param(..., marks=...) instead.
For more details, see: https://docs.pytest.org/en/latest/parametrize.html
#pytest.mark.skipif(True, reason="always skip")
test_b.py:9: RemovedInPytest4Warning: Applying marks directly to parameters is deprecated, please use pytest.param(..., marks=...) instead.
For more details, see: https://docs.pytest.org/en/latest/parametrize.html
#pytest.mark.skipif(True, reason="always skip")
-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================================================================== 2 skipped, 2 warnings in 0.03 seconds ============================================================================