I am writing tests in pytest bdd with selenium. I am using pytest-html to generate report. For debug purpose or just to have a proper logging, I want selenium screenshots and rest of the logs in html report. But I am unable to have selenium screenshot in passed report.
Here are the things I am trying.
There is a pytest-html hook wrapper in conftest.py
conftest.py
#pytest.mark.hookwrapper
def pytest_runtest_makereport(item, call):
pytest_html = item.config.pluginmanager.getplugin('html')
outcome = yield
report = outcome.get_result()
print("printing report")
extra = getattr(report, 'extra', [])
if report.when == 'call':
mylogs = ""
with open('/tmp/test.log', 'r') as logfile:
for line in logfile:
mylogs = mylogs + line + "<br>"
extra.append(pytest_html.extras.html('<html><body>{}</body></html>'.format(mylogs)))
report.extra = extra
This code is adding logs in my report.html
Similarly, I will be adding few selenium screenshots in my test code.
I want to know if we can generate a report containing all selenium screenshots.
Following is my test file
test_file.py
def test_case():
logger.info("I will now open browser")
driver = webdriver.Chrome()
driver.get('http://www.google.com')
driver.save_screenshot('googlehome.png')
time.sleep(3)
driver.quit()
I want googlehome.png and all other png file to be part of html report. I will be great if the we can generate a robot framework like html report.
Is there any way in pytest we can do that?
Following is the command I use to generate report
py.test -s --html=report.html --self-contained-html -v
You have to pass webdriver from test into pytest reporting system.
In my case I use webdriver as fixtuer. That have a lot of other advantages - for example you can test for any set of browsers and control that from one place.
#pytest.fixture(scope='session', params=['chrome'], ids=lambda x: 'Browser: {}'.format(x))
def web_driver(request):
browsers = {'chrome': webdriver.Chrome}
return browsers[]()
def test_case(web_driver):
logger.info("I will now open browser")
web_driver.get('http://www.google.com')
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
outcome = yield
rep = outcome.get_result()
if rep.when == 'call' and not rep.failed:
try:
if 'web_driver' in item.fixturenames:
web_driver = item.funcargs['web_driver']
else:
return # This test does not use web_driver and we do need screenshot for it
# web_driver.save_screenshot and other magic to add screenshot to your report
except Exception as e:
print('Exception while screen-shot creation: {}'.format(e))
Here is how I solved mine:
Okay so here is how you access webdriver from the report generation hook:
#pytest.mark.hookwrapper
def pytest_runtest_makereport(item, call):
timestamp = datetime.now().strftime('%H-%M-%S')
pytest_html = item.config.pluginmanager.getplugin('html')
outcome = yield
report = outcome.get_result()
extra = getattr(report, 'extra', [])
if report.when == 'call':
feature_request = item.funcargs['request']
driver = feature_request.getfuncargvalue('browser')
driver.save_screenshot('D:/report/scr'+timestamp+'.png')
extra.append(pytest_html.extras.image('D:/report/scr'+timestamp+'.png'))
# always add url to report
extra.append(pytest_html.extras.url('http://www.example.com/'))
xfail = hasattr(report, 'wasxfail')
if (report.skipped and xfail) or (report.failed and not xfail):
# only add additional html on failure
extra.append(pytest_html.extras.image('D:/report/scr.png'))
extra.append(pytest_html.extras.html('<div>Additional HTML</div>'))
report.extra = extra
Related
I'm trying to customize report.html of pytest using pytest-html plugin.
I searched up many sites(including pytest-html documentation) and found that the code below is commonly used.(The code is in conftest.py)
(https://pytest-html.readthedocs.io/en/latest/user_guide.html#extra-content)
#pytest.hookimpl(hookwrapper = True)
def pytest_runtest_makereport(item, call):
pytest_html = item.config.pluginmanager.getplugin("html")
outcome = yield
report = outcome.get_result()
extra = getattr(report, "extra", [])
if report.outcome == "call":
#always add url to report
xfail = hasattr(report, "wasxfail")
if (report.skipped and xfail) or (report.failed and not xfail):
extra.append(pytest_html.extras.url("http://www.google.com/"))
extra.append(pytest_html.extras.text('Hi', name = 'TEXT'))
# only add additional html on failure
# extra.append(pytest_html.extras.html("<div>Additional HTML</div>"))
report.extra = extra
However, I have no idea of each lines.
No one explained what the line does actually.
Why does the script allocates yield keyword to outcome with out any variable(e.g. yield 1), and what does yield.get_result() actually do?
Also, I have no idea of xfail("wasxfail").
I found that #pytest.xfail makes the test function fail in the pytest run, but I think it has nothing to do with the above code.
Why don't we use 'fail' not 'xfail'?
Anyway, what I need is
First, the meaning of each line and what it does.
Second, I wanna set different message in the report.html depending on the pass/fail.
I tried python report.outcome == 'failed', report.outcome == 'passed' to divide conditions, but it didn't work.
Third, when adding the text not url, it becomes tag and helps redirecting the page containing the text.
However, if I click the page in the html, it opens about:blank page not the desired one.
Using right click and open in new tab redirects to the desired one.
Any help is welcomed. Thanks.
+ I have more questions, I tried
if report.passed:
extra.append(pytest_html.extras.url("https://www.google.com/")
report.extra = extra
It attaches 3 same links in the report.html(Results table) How can I handle it?
+ I could log a message when test is failed like msg = 'hi', pytest.fail(msg) However, I cannot get a clue to do it when the test is passed.
Trying to answer as many lines as possible.
Pytest uses generators to iterate over the report steps.
The function pytest_runtest_makereport iterates over every result.when (not .outcome, this is a bug in the documentation) which according to pytest are as follows: 'collect', 'setup', 'call', and 'teardown'.
The get_result is how pytest implements its hooks.
The confusion about failed and xfail (expected to fail) is how you define a test failure: It is an error if it was skipped but was expected to fail or if it failed but was not expected to fail.
The thing with the about:blank could also be a bug.
What you want to use your if statements on is not the call info but the report:
if report.failed:
do_stuff()
if report.passed:
do_stuff_different()
One way to get more info about code and its context would be to debug it using breakpoint().
So the snippet you are looking for is:
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
pytest_html = item.config.pluginmanager.getplugin("html")
outcome = yield
report = outcome.get_result()
extra = getattr(report, "extra", [])
if report.when == "call":
xfail = hasattr(report, "wasxfail")
if (report.skipped and xfail) or (report.failed and not xfail):
extra.append(pytest_html.extras.url("http://www.google.com/"))
if report.passed:
extra.append(pytest_html.extras.url("http://www.stackoverflow.com/"))
report.extra = extra
I'd like to run pytest and then store results and present them to users on demand (e.g. store pytest results to a db and then expose them through web service)
I could run pytest from a command line with option to save results report into file, then find and parse the file, but feels silly to have the results in a (pytest) python app, then store them to a file and then instantly look for the file, parse it back into python code for further processing. I know I can run pytest programatically via pytest.main(args) however it only return some exit code and not details about tests results - how can I retrieve the results when using pytest.main()?
I'm looking for smt like
args = # arguments
ret_code = pytest.main(args=args) # pytest.main() as is only returns trivial return code
my_own_method_to_process(pytest.results) # how to retrieve any kind of pytest.results object that would contain test execution results data (list of executed tests, pass fail info, etc as pytest is displaying into console or saves into file reports)
There are couple of similar questions but always with some deviation that doesn't work for me. I simply want to run pytest from my code and - whatever format the output would be - directly grab it and further process.
(Note I'm in a corporate environment where installing new packages (i.e. pytest plugins) is limited, so I'd like to achieve this without installing any other module/pytest plugin into my environment)
Write a small plugin that collects and stores reports for each test. Example:
import time
import pytest
class ResultsCollector:
def __init__(self):
self.reports = []
self.collected = 0
self.exitcode = 0
self.passed = 0
self.failed = 0
self.xfailed = 0
self.skipped = 0
self.total_duration = 0
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(self, item, call):
outcome = yield
report = outcome.get_result()
if report.when == 'call':
self.reports.append(report)
def pytest_collection_modifyitems(self, items):
self.collected = len(items)
def pytest_terminal_summary(self, terminalreporter, exitstatus):
print(exitstatus, dir(exitstatus))
self.exitcode = exitstatus.value
self.passed = len(terminalreporter.stats.get('passed', []))
self.failed = len(terminalreporter.stats.get('failed', []))
self.xfailed = len(terminalreporter.stats.get('xfailed', []))
self.skipped = len(terminalreporter.stats.get('skipped', []))
self.total_duration = time.time() - terminalreporter._sessionstarttime
def run():
collector = ResultsCollector()
pytest.main(plugins=[collector])
for report in collector.reports:
print('id:', report.nodeid, 'outcome:', report.outcome) # etc
print('exit code:', collector.exitcode)
print('passed:', collector.passed, 'failed:', collector.failed, 'xfailed:', collector.xfailed, 'skipped:', collector.skipped)
print('total duration:', collector.total_duration)
if __name__ == '__main__':
run()
I am trying to generate a self contained html report using pytest-html and selenium. I have been trying to imbedded screenshots into the report but they are not being displayed.
My conftest.py looks like this
#pytest.fixture()
def chrome_driver_init(request, path_to_chrome):
driver = webdriver.Chrome(options=opts, executable_path=path_to_chrome)
request.cls.driver = driver
page_object_init(request, driver)
driver.get(URL)
driver.maximize_window()
yield driver
driver.quit()
# Hook that takes a screenshot of the web browser for failed tests and adds it to the HTML report
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item):
pytest_html = item.config.pluginmanager.getplugin("html")
outcome = yield
report = outcome.get_result()
extra = getattr(report, "extra", [])
if report.when == "call":
feature_request = item.funcargs['request']
driver = feature_request.getfixturevalue('chrome_driver_init')
nodeid = item.nodeid
xfail = hasattr(report, "wasxfail")
if (report.skipped and xfail) or (report.failed and not xfail):
file_name = f'{nodeid}_{datetime.today().strftime("%Y-%m-%d_%H_%M")}.png'.replace("/", "_").replace("::", "_").replace(".py", "")
driver.save_screenshot("./reports/screenshots/"+file_name)
extra.append(pytest_html.extras.image("/screenshots/"+file_name))
report.extra = extra
I am convinced the problem is with the path to the image, and I have tried so many str combinations, os.path and pathlib but nothing has worked. The screenshot is being saved in the expected location and I can open it like any other image. Its just not displaying on the report.
<div class="image"><img src="data:image/png;base64,screenshots\scr_tests_test_example_TestExample_test_fail_example_2022-01-18_16_26.png"/></div>
EDIT: For addional clairification. I have tried to use absolute path in the extra.append but it kept giving me a Cant Resolve File error in the HTML file. My absoulte path was(with some personal details redacted) C:\Users\c.Me\OneDrive - Me\Documents\GitHub\project\build\reports\screenshots\filename.png I have tried it with both '/' and '\'
Also my File structure
project
├───build
│ ├───reports
│ ├───screenshots
│ ├───filename.png
| ├───report.html
| ├───run.py # I am running the test suite from here
├───scr
| ├───settings.py
│ ├───tests
│ ├───confest.py
run.py
if __name__ == "__main__":
os.system(f"pytest --no-header -v ../scr/tests/ --html=./reports/Test_Report_{today}.html --self-contained-html")
For Prophet, may be bless me this day
To get the Cannot Resolve Directory error my code is the following
file_name = f'{nodeid}_{datetime.today().strftime("%Y-%m-%d_%H_%M")}.png'.replace("/", "_").replace("::", "_").replace(".py", "")
img_path = os.path.join(REPORT_PATH, 'screenshots', file_name)
driver.save_screenshot(img_path)
extra.append(pytest_html.extras.image(img_path))
The variable REPORT_PATH is imported from the settings.py (see directory tree above) and is created by
PROJ_PATH = Path(__file__).parent.parent
REPORT_PATH = PROJ_PATH.joinpath("build\reports")
also fun fact if I do img_path.replace("\\", "/") the error changes to Cannot Resolve File
I have learned so much in this painful journey. Mostly I have learned I am an idiot. The problem was that I wanted to make a self contained HTML. Pytest-html does not work as expected with adding images to a self contained report. Before you can you have to convert the image into its text base64 version first. So the answers to all my owes was a single line of code.
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item):
pytest_html = item.config.pluginmanager.getplugin("html")
outcome = yield
report = outcome.get_result()
extra = getattr(report, "extra", [])
if report.when == "call":
feature_request = item.funcargs['request']
driver = feature_request.getfixturevalue('chrome_driver_init')
nodeid = item.nodeid
xfail = hasattr(report, "wasxfail")
if (report.skipped and xfail) or (report.failed and not xfail):
file_name = f'{nodeid}_{datetime.today().strftime("%Y-%m-%d_%H_%M")}.png'.replace("/", "_").replace("::", "_").replace(".py", "")
img_path = os.path.join(REPORT_PATH, "screenshots", file_name)
driver.save_screenshot(img_path)
screenshot = driver.get_screenshot_as_base64() # the hero
extra.append(pytest_html.extras.image(screenshot, ''))
report.extra = extra
Thank you Prophet for guiding on this pilgrimage. Now I must rest.
I'm not completely sure how it works with PyTest, however we have similar issue with Java Extent Manager.
There you have to pass the absolute path of the image file, not the relative path.
As I can see here the current working directory can be achieved as following:
import pathlib
pathlib.Path().resolve()
So, if I understand that correctly you should change your code from
extra.append(pytest_html.extras.image("/screenshots/"+file_name))
to
working_root = pathlib.Path().resolve()
extra.append(pytest_html.extras.image(working_root + "/screenshots/"+file_name))
UPD
I think you are missing a reports subfolder here.
Instead of
working_root = pathlib.Path().resolve()
extra.append(pytest_html.extras.image(working_root + "/screenshots/"+file_name))
Try using
working_root = pathlib.Path().resolve()
extra.append(pytest_html.extras.image(working_root + "/reports/screenshots/"+file_name))
I have an issue where I have a dictionary stub inside a mock and my test crashes. Shouldn't the mock replace everything to begin with so I shouldn't even run into this issue?
script.py
from selenium import webdriver
def nothing_here_really_just_setting_up_the_logic_flow():
"""
This function is completely irrelevant. It just loads a website
returns the webdriver session as "driver" and a dictionary
"add extra xpath" with the addtional xpath syntax for the function
where my issue is at
"""
# Load a website
driver = webdriver.Chrome()
driver.get(#some website)
# Prepare extra xpath syntax
add_extra_xpath = {}
add_extra_xpath['add me'] = 1 # The actual dict is more complex then this and
# this number keeps changing so that's why I'm using a
# dict, if you're wondering
return driver, add_extra_xpath
def function_where_I_have_a_problem(driver, add_extra_xpath):
"""
Simply find an element on the site via a connotated xpath
"""
# Find an element by xpath with the help of "add_extra_path"
find = driver.find_element_by_xpath('begin writing xpath' + str(add_extra_xpath['add me']) + 'finish off the xpath')
test_script.py
import pytest
import script
from unittest.mock import MagicMock
# Mock the "driver" and stub the "add_extra_xpath" dictionary
#pytest.fixture
def fixture():
driver = MagicMock()
driver.find_element_by_xpath.side_effect = driver
add_extra_xpath = {}
add_extra_xpath['add me'] = 1
return driver, add_extra_xpath
def test_function_where_I_have_a_problem(fixture):
driver, add_extra_xpath = fixture
script.function_where_I_have_a_problem(driver, add_extra_xpath)
assert driver.call_count == 1
Running the test crashes due to an:
AttributeError: 'dict' object has no attribute 'find_element_by_xpath'
at
find = driver.find_element_by_xpath('begin writing xpath' + str(add_extra_xpath['add me']) + 'finish off the xpath')
Shouldn't the side_effect I set back in my fixture override this 'find_element_by_xpath'?
I'm writing a small fixture for implementing regression tests. The function under test does not contain any assert statements but produces output which is compared to a recorded output which is assumed to be correct.
This is a simplfied snippet to demonstrate what I'm doing:
#pytest.yield_fixture()
def regtest(request):
fp = cStringIO.StringIO()
yield fp
reset, full_path, id_ = _setup(request)
if reset:
_record_output(fp.getvalue(), full_path)
else:
failed = _compare_output(fp.getvalue(), full_path, request, id_)
if failed:
pytest.fail("regression test %s failed" % id_, pytrace=False)
In general my approach works works but I want to improve error reporting so that the fixture indicates the failure of a test and not the testing function itself: this implementation always prints a . because the testing function does not raise any exception, and then an extra E if pytest.fail is called in the last line.
So what I want is to supress the output of . triggered by the function under test and let my fixture code output the approriate character.
Update:
I was able to improve output, but it still I have to many "." in the output when the tests are running. It is uploaded at https://pypi.python.org/pypi/pytest-regtest
you can find the repository at https://sissource.ethz.ch/uweschmitt/pytest-regtest/tree/master
Sorry for posting links, but the files got a bit bigger now.
Solution:
I came up with a solution by implementing an hook which handles the regtest result in hook. The code is then (simplified):
#pytest.yield_fixture()
def regtest(request):
fp = cStringIO.StringIO()
yield fp
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
try:
outcome = yield
except Exception:
raise
else:
# we only handle regtest fixture if no other other exception came up during testing:
if outcome.excinfo is not None:
return
regtest = item.funcargs.get("regtest")
if regtest is not None:
_handle_regtest_result(regtest)
And _handle_regtest_result either stores the recorded values or does the appropriate checks. The plugin is now available at https://pypi.python.org/pypi/pytest-regtest
Your are mixing two things there: the fixture itself (setting up conditions for your test) and the expected behavior _compare_output(a, b). You are probably looking for something along the lines:
import pytest
#pytest.fixture()
def file_fixture():
fp = cStringIO.StringIO()
return fp.getvalue()
#pytest.fixture()
def request_fixture(request, file_fixture):
return _setup(request)
def test_regression(request_fixture, file_fixture):
reset, full_path, id_ = request_fixture
if reset:
_record_output(file_fixture, full_path)
else:
failed = _compare_output(file_fixture, full_path, request, id_)
assert failed is True, "regression test %s failed" % id_