I am using pytest framework and trying to integrate test case with Testrail.
I need to do mapping of test case with test case IDs in testrail.
Consider below scenario where there are 2 test cases.
def test_login1():
test_case_id= 10
def test_login2():
test_case_id= 20
I need variable (test_case_id), once each and every test case executed. "test_case_id" is basically testcase ID in testrail. At the end of testcase execution, I am checking testcase result (passed or failed) and integrating with testrail using TC Id. Test case ID will be passed from Testcase and in conftest.py.
I have 2 hooks created in conftest file.
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item):
"""
This function will be executed after execution of each and every test case.
Results dictionary will contain details of executed TC
"""
outcome = yield
result = outcome.get_result()
if result.when == 'call':
item.session.results[item] = result
def pytest_sessionfinish(session):
"""
This function will be execution post execution of all test cases
"""
passed_amount = sum(1 for result in session.results.values() if result.passed)
failed_amount = sum(1 for result in session.results.values() if result.failed)
logger.info('**** Test Summary ****')
logger.info(f'Passed Test Cases Count -> {passed_amount}')
logger.info(f'Failed Test Cases Count -> {failed_amount}')
At the end of execution of these 2 hooks, I am expecting output in list or dictionary containing
<test_case_id: <tc_result> eg. (10: Passed and 20: Failed) or ( 10:True and 20:False)
I am reading my test data from a Python file as follows.
//testdata.py -- its a list of sets.
TEST_DATA = [
(
{"test_scenario":"1"}, {"test_case_id":1}
),
(
{"test_scenario":"2"}, {"test_case_id":2}
)
]
Now I use this test data as part of a pytest test file.
// test.py
// import testdata
test_data = testdata.TEST_DATA
start = 0
class TestOne():
#pytest.mark.parametrize(("test_scenario,testcase_id"),test_data)
#testcaseid.marktc[test_data[start][1]["test_case_id"]]
def testfunction():
global start
start = start + 1
// Doing test here.
Now when I print start, it changes its value continuoulsy. But when I try to retrieve the pytest results, I still keep getting start = 0 due to which my test case ID isnt being recorded properly.
Can I either
Pass marker from within the function.
Or is there a way to change the count of start dynamically in this example?
P.S. This is the best way that I am able to store my test data currently.
Here's how i have my testcaseid.marktc defined.
// testrailthingy.py
class testcaseid(object):
#staticmethod
def marktc(*ids):
return pytest.mark.testrail(ids=ids)
What I am trying to do is to skip tests that are not supported by the code I am testing. My PyTest is running tests against an embedded system that could have different versions of code running. What I want to do mark my test such that they only run if they are supported by the target.
I have added a pytest_addoption method:
def pytest_addoption(parser):
parser.addoption(
'--target-version',
action='store', default='28',
help='Version of firmware running in target')
Create a fixture to decide if the test should be run:
#pytest.fixture(autouse = True)
def version_check(request, min_version: int = 0, max_version: int = 10000000):
version_option = int(request.config.getoption('--target-version'))
if min_version and version_option < min_version:
pytest.skip('Version number is lower that versions required to run this test '
f'({min_version} vs {version_option})')
if max_version and version_option > max_version:
pytest.skip('Version number is higher that versions required to run this test '
f'({max_version} vs {version_option})')
Marking the tests like this:
#pytest.mark.version_check(min_version=24)
def test_this_with_v24_or_greater():
print('Test passed')
#pytest.mark.version_check(max_version=27)
def test_not_supported_after_v27():
print('Test passed')
#pytest.mark.version_check(min_version=13, max_version=25)
def test_works_for_range_of_versions():
print('Test passed')
In the arguments for running the test I just want to add --target-version 22 and have only the right tests run. I haven't been able to figure out how to pass the arguments from #pytest.mark.version_check(max_version=27), to version_check.
Is there a way to do this or am I completely off track and should be looking at something else to accomplish this?
You are not far from a solution, but you're mixing up markers with fixtures; they are not the same, even if you give them the same name. You can, however, read markers of each test function in your version_check fixture and skip the test depending on what was provided by the version_check marker if set. Example:
#pytest.fixture(autouse=True)
def version_check(request):
version_option = int(request.config.getoption('--target-version'))
# request.node is the current test item
# query the marker "version_check" of current test item
version_marker = request.node.get_closest_marker('version_check')
# if test item was not marked, there's no version restriction
if version_marker is None:
return
# arguments of #pytest.mark.version_check(min_version=10) are in marker.kwargs
# arguments of #pytest.mark.version_check(0, 1, 2) would be in marker.args
min_version = version_marker.kwargs.get('min_version', 0)
max_version = version_marker.kwargs.get('max_version', 10000000)
# the rest is your logic unchanged
if version_option < min_version:
pytest.skip('Version number is lower that versions required to run this test '
f'({min_version} vs {version_option})')
if version_option > max_version:
pytest.skip('Version number is higher that versions required to run this test '
f'({max_version} vs {version_option})')
in my project, I want to reconcile selected data retrieved from various databases and print out differences. I
create a list containing datasets to be reconciled (dataset1, dataset2, dataset3)
create a dictionary of lists where I store output from each database (i.e. list_db1, list_db2, list_db3)
start loop where for each dataset a call is issued to n databases (sql queries are read from external file)
modify database returns (formatting but also qualitative changes)
do some set operations among returned data
print out differences
clear lists in dictionary and go to next dataset
I would like to move step 4 from main script outside, to achieve better maintainability. For dataset A, I want to modify data from database X in way x_A, from database Y in way y_A, from database Z in way z_A. For dataset B, I only want to modify data from database Z in way z_B.
The function which modifies database return is currently part of main script and is a somewhat complex if-then statement.
def retrieve_db1(dataset):
...
list_db1.append(sql_output)
...
def retrieve_db2(dataset):
...
list_db2.append(sql_output)
...
def retrieve_db3(dataset):
...
list_db3.append(sql_output)
...
def myfunction(arg1,arg2):
if arg1 == 'dataset1':
if arg2 == 'list_db1':
list_db1.append('x')
elif arg2 == 'list_db3':
list_db2.append('y')
else:
None
elif arg1 == 'dataset2':
if arg2 == 'list_db3':
list_db3.append('z')
else:
None
...
return arg2
datasets = ['dataset1','dataset2','dataset3']
for d in datasets:
mydict = {'list_db1':[],'list_db2':[],'list_db3':[]}
retrieve_db1(d)
myfunction(d,list_db1)
retrieve_db2(d)
myfunction(d,list_db2)
retrieve_db3(d)
myfunction(d,list_db3)
#... merge list_db1 and list_db2, find differences against list_db3, print out etc.
As you see, the function itself is a decision tree dependent on both arguments. Is it possible to maintain only the part of code with decision tree in a separate file (script)?
import other_script
def my_function(arg1,arg2):
...
other_script(arg1,arg2)
...
The tree is quite complex and this helps with maintenance of both scripts. I am not sure how to ensure that list created with main script is modified when running other_script, that is data is still there when I return to main.
How to locate slow django tests? How to locate tests, on which test runner can 'stuck'? Do you know any good custom django test runners, that can provide more detailed information on test performance?
You can get Django to print the tests it's running with:
./manage.py test -v 3
This will print the name of the test, run it, then print "ok". So you can figure out which test is slow.
You could try nose. Plenty of tutorials available on installing it along with Django. To get a high level overview of testing time, look into the pinocchio nose extensions, specifically the stopwatch one.
To identify slow tests, you would measure the duration it takes for each test to run and define a threshold of what slow means to you.
By default, Django's test runner does not show detailed timing information. You could use an alternative test runner, some of which will show timing data for test runs.
However, you could easily cook your own, as Django uses Python's unittest facilities, which are well documented and can be extended.
The following code can be found in one file here:
https://gist.github.com/cessor/44446799736bbe801dc5565b28bfe58b
To run tests, Django uses a DiscoverRunner (which wraps Python's TestRunner). The runner will discover testcases from your project structure by convention. Cases are collected in a TestSuite and executed. For each test method, a TestResult will be created which is then used to communicate the result to the developer.
To measure how long it takes for a test to run, you may exchange the TestResult for a custom class:
import unittest
from django.test.runner import DiscoverRunner
class StopwatchTestResult(unittest.TextTestResult):
...
class StopwatchTestRunner(DiscoverRunner):
def get_resultclass(self):
return StopwatchTestResult
In this example, the code above is placed in a Django app called example in a module called testrunner.py. To execute it, invoke Django's test command like this:
$ python manage.py test --testrunner=example.testrunner.StopwatchTestRunner -v 2
The StopwatchTestResult can be made to register timing data:
import time
...
class StopwatchTestResult(unittest.TextTestResult):
"""
Times test runs and formats the result
"""
# Collection shared between all result instaces to calculate statistics
timings = {}
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.start = 0
self.stop = 0
self.elapsed = 0
def startTest(self, test):
self.start = time.time()
super().startTest(test)
def stopTest(self, test):
super().stopTest(test)
self.stop = time.time()
self.elapsed = self.stop - self.start
self.timings[test] = self.elapsed
def getDescription(self, test):
"""
Format test result with timing info
e.g. `test_add (module) [0.1s]`
"""
description = super().getDescription(test)
return f'{description} [{self.elapsed:0.4f}s]'
For each test method, TestResult.startTest is called and registers a time stamp. The duration of each test run is calculated in TestResult.stopTest.
When the verbosity is increased, the description is printed, which allows you to format the timing data:
test_student_can_retract_their_request_commit (person.tests.test_view_new.test_view_person_new.NewViewTest) [0.0080s] ... ok
test_student_can_see_their_request_status (person.tests.test_view_new.test_view_person_new.NewViewTest) [0.0121s] ... ok
test_student_can_submit_request (person.tests.test_view_new.test_view_person_new.NewViewTest) [0.0101s] ... ok
To identify slow tests, you can either eyeball the solution or use a systematic approach:
class StopwatchTestRunner(DiscoverRunner):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._stats = kwargs['stats']
#classmethod
def add_arguments(cls, parser):
DiscoverRunner.add_arguments(parser)
parser.add_argument(
"--stats",
action="store_true",
help="Print timing statistics",
)
def run_tests(self, test_labels, extra_tests=None, **kwargs):
super().run_tests(test_labels, extra_tests, **kwargs)
if self._stats:
StopwatchTestResult.print_stats()
def get_resultclass(self):
...
This test runner adds a new option to the test command. The command will print additional statistics, when the --stats option is set:
$ python manage.py test --testrunner=example.testrunner.StopwatchTestRunner -v 2 --stats
The actual calculation and output are performed in StopwatchTestResult.print_stats(). It is implemented as a class method because it refers to data belonging to all TestResults:
import statistics
class StopwatchTestResult(unittest.TextTestResult):
...
#classmethod
def print_stats(cls):
"""
Calculate and print timings
These data are likely skewed, as is normal for reaction time data,
therefore mean and standard deviation are difficult to interpret.
Thus, the IQR is used to identify outliers.
"""
timings = StopwatchTestResult.timings.values()
count = len(timings)
mean = statistics.mean(timings)
stdev = statistics.stdev(timings)
slowest = max(timings)
q1, median, q3 = statistics.quantiles(timings)
fastest = min(timings)
total = sum(timings)
print()
print("Statistics")
print("==========")
print("")
print(f"count: {count:.0f}")
print(f" mean: {mean:.4f}s")
print(f" std: {stdev:.4f}s")
print(f" min: {fastest:.4f}s")
print(f" 25%: {q1:.4f}s")
print(f" 50%: {median:.4f}s")
print(f" 75%: {q3:.4f}s")
print(f" max: {slowest:.4f}s")
print(f"total: {total:.4f}s")
# https://en.wikipedia.org/wiki/Interquartile_range
iqr = q3 - q1
fast = q1 - 1.5 * iqr
slow_threshold = q3 + 1.5 * iqr
slow_tests = [
(test, elapsed)
for test, elapsed
in StopwatchTestResult.timings.items()
if elapsed >= slow_threshold
]
if not slow_tests: return
print()
print("Outliers")
print("========")
print("These were particularly slow:")
print()
for test, elapsed in slow_tests:
print(' ', test, f"[{elapsed:0.4f}s]")
To identify slow tests, you can analyze the obtained timing data for outliers. Usually, one would calculate the mean and a standard deviation and define outliers as all values beyond a certain threshold, such as +1.5 Standard deviations from the mean.
However, when collecting timing data, the distribution will be skewed, because responses can't be faster than 0 seconds and slow outliers will draw the distribution's mean to the right in a long tail. Under these circumstances, mean and standard deviation can be difficult to interpret, thus the code uses a different approach to identify outliers.
Use statistics.quantiles from Python's statistics module (requires Python 3.8). This yields the 25%, 50% and 75% boundaries of the distribution. To identify outliers, you define the threshold as:
slow_threshold = q3 + 1.5 * iqr
You may want to finetune this formular for your purposes. The IQR is a reference for the dispersion of your distribution (interquartile range, q3 - q1). Instead of with q3, you might calculate the threshold starting from the median. You could reduce or increase the constant factor, which yould make the outlier detection more or less sensitive.
All together, this produces the following output:
...
test_bad_request (content.tests.test_image_upload_view.ImageUploadTests) [0.1159s] ... ok
test_view_editor_can_access_create_page (edv.knowledgebase.tests.test_view_create_page.CreatePageTests) [3.5587s] ... ok
test_admins_are_redirected_to_where_they_were_before (web.tests.test_view_login.LoginOfficialUser) [0.4595s] ... ok
test_ldap_users_are_redirected_to_their_profile_pages (web.tests.test_view_login.LoginOfficialUser) [0.4522s] ... ok
...
Statistics
==========
count: 708
mean: 0.0222s
std: 0.2177s
min: 0.0000s
25%: 0.0000s
50%: 0.0000s
75%: 0.0156s
max: 4.5008s
total: 15.6883s
Outliers
========
These were particularly slow:
test_account_view (account.tests.test_view_account.AccountViewTest) [0.0690s]
test_headline_en (arbeitseinheit.tests.test_arbeitseinheit_page.ArbeitseinheitTest) [0.0846s]
test_admins_see_admin_menu (arbeitseinheit.tests.test_arbeitseinheit_superuser.ArbeitseinheitTest) [0.1628s]
test_editors_can_post_in_their_arbeitseinheit (arbeitseinheit.tests.test_view_create_news.CreateArticleViewTests) [0.0625s]
test_bad_request (content.tests.test_image_upload_view.ImageUploadTests) [0.1159s]
...