We Keep Coding

How to test if a function calls range in python?

I'm a Python instructor and I wanted to give my students a task: write a function that computes the average of a list using a for loop and the range object.
I wanted to run a test on their function to see whether it actually uses the range object. How can I do that?
It should be something like this:
def avg(L):
Pass
def test_range(avg):
...
If avg contains range, then test_range should return True.
I tried solutions that utilize func_code, but apparantly range doesn't have that.

You can use Python's unittest.mock module to wrap around the range function from the builtins module, and then let your test assert that the wrapped range was indeed called.
For example, using Python's unittest framework for writing the test:
import builtins
import unittest
from unittest.mock import patch
# I don't know what L is supposed to be, and I know that there
# are better ways to compute average of a list, but the code
# for calculating the average is not important for this question.
def avg(L):
total = 0
for index in range(len(L)):
total += L[index]
return total / len(L)
class TestAverage(unittest.TestCase):
def test_avg(self):
with patch("builtins.range", wraps=builtins.range) as wrapped_patch:
expected = 47
actual = avg([1,49,91])
self.assertEqual(expected, actual)
wrapped_patch.assert_called()
if __name__ == '__main__':
unittest.main()
$ python -m unittest -v main.py
test_avg (main.TestAverage) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
It uses unittest.mock's patch targetting the builtins.range function. Normally, patch is used in tests to replace the behavior and/or return value of the target, but in this case, you can pass wraps=builtins.range (which gets passed to the underlying Mock object), which means "I just want to spy on the call, but not modify its behavior":
wraps: Item for the mock object to wrap. If wraps is not None then calling the Mock will pass the call through to the wrapped object (returning the real result).
By wrapping it in a Mock object, you can use any of Mock's assert functions to check calls to range, like assert_called which checks whether the target was called at least once. You can be more specific by asserting the number of times range was called:
self.assertTrue(wrapped_patch.call_count == 1)
The assertion would fail if it wasn't called at all:
# Here, `range` wasn't used at all.
def avg(L):
return sum(L) / len(L)
class TestAverage(unittest.TestCase):
# same as the code above
$ python -m unittest -v main.py
test_avg (main.TestAverage) ... FAIL
======================================================================
FAIL: test_avg (main.TestAverage)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/main.py", line 15, in test_avg
wrapped_patch.assert_called()
File "/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/unittest/mock.py", line 888, in assert_called
raise AssertionError(msg)
AssertionError: Expected 'range' to have been called.
The most important thing to note when using patch, is to know exactly where to patch. In this case, you can check the docs or use __module__ to know range's module:
>>> range
<class 'range'>
>>> range.__module__
'builtins'
But the test is a bit naive, because it can still pass even though avg didn't really use range:
def avg(L):
range(len(L)) # Called but really unused. Sneaky!
return sum(L) / len(L)
class TestAverage(unittest.TestCase):
# same as the code above
$ python -m unittest -v main.py
test_avg (main.TestAverage) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
A slightly confusing workaround would be a test that "breaks" range such that, if the function was really using range, then it wouldn't work anymore:
def avg(L):
range(len(L)) # Called but really unused. Sneaky!
return sum(L) / len(L)
class TestAverage(unittest.TestCase):
def test_avg(self):
# same as above
def test_avg_is_really_using_range(self):
L = [10,20,90]
# Is it returning the correct result?
self.assertEqual(avg(L), 40)
# OK, but did it really use `range`?
# Let's try breaking `range` so it always yields 0,
# so we expect the return value to be *different*
with patch("builtins.range", return_value=[0,0,0]):
self.assertNotEqual(avg(L), 40)
So, if avg was sneakily calling but not really using range, then test_avg_is_really_using_range would now fail because avg still yields the correct value even with a broken range:
$ python -m unittest -v main.py
test_avg (main.TestAverage) ... ok
test_avg_really_using_range (main.TestAverage) ... FAIL
======================================================================
FAIL: test_avg_really_using_range (main.TestAverage)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/main.py", line 19, in test_avg_really_using_range
self.assertNotEqual(avg(L), 40)
AssertionError: 40.0 == 40
Lastly, as a side note, I'm using assertEqual here in all the example because the test for the return value is not the focus, but do read up on proper ways to assert possible float values, ex. How to perform unittest for floating point outputs? - python

Python Unit Test to Assert Type Annotation of Object

In Python versions <3.11 where the assert_type (source) isn't available, how does one assert a type annotation via the unittest TestCase class? Example of the problem:
from typing import List
from unittest import TestCase
def dummy() -> List[str]:
return ["one", "two", "three"]
class TestDummy(TestCase):
def test_dummy(self):
self.assertEqual(
List[str],
type(dummy())
)
The test fails with the following output:
<class 'list'> != typing.List[str]
Expected :typing.List[str]
Actual :<class 'list'>
<Click to see difference>
Traceback (most recent call last):
File "C:\Users\z\dev\mercata\scratch\mock_testing.py", line 12, in test_dummy
self.assertEqual(
AssertionError: typing.List[str] != <class 'list'>
The approach I currently use is as follows:
data = dummy()
self.assertTrue(
type(data) == list
)
self.assertTrue(all([
type(d) == str for d in data
]))
This works but requires iterating the entirety of the object which is unwieldy with larger datasets. Is there a more efficient approach for Python versions <3.11 (not requiring a third-party package)?

assert_type is used to ask a static type checker to confirm a value is of some type. At normal runtime this method doesn't do anything. If you want to use it, then you should use static analysis tooling, for example mypy or pyright. Checking assertEqual is a runtime operation, and unlike some languages, instances of generics in python do not retain their type info at runtime, which is why the class is being shown as the standard <class 'list'> and not the generic one from the method type annotation.
Because assert_type doesn't perform anything at runtime, it will not check for the contents of the actual list. It is used to add an explicit typecheck into the code, and only useful if all of the inputs for how a variable was constructed have been properly type checked as well. So it would also not be useful within unit testing as you have it.
For example, the following script only produces one error:
from typing import assert_type
def dummy() -> list[str]:
return [1]
res = dummy()
assert_type(res, list[str])
(venv) $ mypy test.py
test.py:4: error: List item 0 has incompatible type "int"; expected "str" [list-item]
Found 1 error in 1 file (checked 1 source file)
This detects the error of an int list being returned by dummy, but the assert_type succeeds because it would be correct if dummy had respected its contract.
If we fixed dummy like below, then at this point we would get the expected assert_type error:
from typing import assert_type
def dummy() -> list[int]:
return [1]
res = dummy()
assert_type(res, list[str])
(venv) $ mypy test.py
test.py:7: error: Expression is of type "List[int]", not "List[str]" [assert-type]
Found 1 error in 1 file (checked 1 source file)

While I agree with the general sentiment the commenters have expressed that this type of thing should probably be left to static type checkers rather than unit tests, just for academic purposes, you can construct your own assertion without too much effort.
Something like list[str] is a specified version of the generic type list. By subscripting a type like like list, you are actually calling its __class_getitem__ method, which returns the specified type. The type argument is actually stored and the typing module provides the get_args/get_origin functions to extract more detailed type information from generic types at runtime.
from typing import get_args
print(get_args(list[str])) # (<class 'str'>,)
The problem is more that any concrete list object (like ["one", "two", "three"]) does not store any information about the type of the items it holds (for obvious reasons). This means, at runtime, we would have to check the type of the elements ourselves.
The question thus becomes how pedantic you want your check to be. Lists for example can be as long as you want them to be (or as your memory allows). If you a list object of a million elements, do you really want to check each and every one of them? A possible compromise might be to only check the very first element's type or something like that.
Here is an example of a function checking arbitrary iterable types that are parameterized by "regular" types only (i.e. not something like list[tuple[int]]):
from collections.abc import Iterable
from types import GenericAlias
from typing import Union, cast, get_origin, get_args
def is_of_iter_type(
value: object,
type_: Union[type[Iterable[object]], GenericAlias],
pedantic: bool = False,
) -> bool:
if isinstance(type_, type): # something like unspecified `list`
return isinstance(value, type_)
if isinstance(type_, GenericAlias): # a specified generic like `list[str]`
origin, args = get_origin(type_), get_args(type_)
if not isinstance(origin, type) or not issubclass(origin, Iterable):
raise TypeError
arg = cast(type, args[0])
if not isinstance(arg, type): # type arg is a type var or another generic alias
raise TypeError
if not isinstance(value, origin):
return False
if pedantic:
return all(isinstance(item, arg) for item in value)
else:
return isinstance(next(iter(value)), arg)
raise TypeError
Note also that depending on what iterable you actually pass to this function, it may be a terrible idea to (try to) consume the resulting iterator (via next or all). It would be up to you to ensure that this does not have any bad side effects.
Here is a demo:
print(is_of_iter_type("a", list[str])) # False
print(is_of_iter_type(["a"], list[str])) # True
print(is_of_iter_type(["a"], list)) # True
print(is_of_iter_type(["a", 1], list[str])) # True
print(is_of_iter_type(["a", 1], list[str], pedantic=True)) # False
To incorporate it into a unittest.TestCase you could do this:
...
from unittest import TestCase
class ExtendedTestCase(TestCase):
def assert_is_of_iter_type(
self,
value: object,
type_: Union[type[Iterable[object]], GenericAlias],
pedantic: bool = False,
) -> None:
if not is_of_iter_type(value, type_, pedantic=pedantic):
self.fail(f"{value} is not of type {type_}")
def test(self) -> None:
self.assert_is_of_iter_type(["a", 1], list[str], pedantic=True)
But again, this is most likely not a good idea because something like mypy in --strict mode will probably do a better job at ensuring type safety throughout your code than you could hope to do at runtime. Meaning if you declare def dummy() -> list[str]: ..., but the in the function body you return ["a", 1], then mypy will pick that up and yell at you. Thus, there would be no need for such a test.

Comparison of multi-line strings in Python unit test

When I compare two Unicode strings in a Python unit test, it gives a nice failure message highlighting which lines and characters are different. However, comparing two 8-bit strings just shows the two strings with no highlighting.
How can I get the highlighting for both Unicode and 8-bit strings?
Here is an example unit test that shows both comparisons:
import unittest
class TestAssertEqual(unittest.TestCase):
def testString(self):
a = 'xax\nzzz'
b = 'xbx\nzzz'
self.assertEqual(a, b)
def testUnicode(self):
a = u'xax\nzzz'
b = u'xbx\nzzz'
self.assertEqual(a, b)
if __name__ == '__main__':
unittest.main()
The results of this test show the difference:
FF
======================================================================
FAIL: testString (__main__.TestAssertEqual)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/mnt/data/don/workspace/scratch/scratch.py", line 7, in testString
self.assertEqual(a, b)
AssertionError: 'xax\nzzz' != 'xbx\nzzz'
======================================================================
FAIL: testUnicode (__main__.TestAssertEqual)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/mnt/data/don/workspace/scratch/scratch.py", line 12, in testUnicode
self.assertEqual(a, b)
AssertionError: u'xax\nzzz' != u'xbx\nzzz'
- xax
? ^
+ xbx
? ^
zzz
----------------------------------------------------------------------
Ran 2 tests in 0.001s
FAILED (failures=2)
Update for Python 3
In Python 3, string literals are Unicode by default, so this is mostly irrelevant. assertMultiLineEqual() no longer supports byte strings, so you're pretty much stuck with regular assertEqual() unless you're willing to decode the byte strings to Unicode.

A little digging in the Python source code shows that TestCase registers a bunch of methods to test equality for different types.
self.addTypeEqualityFunc(dict, 'assertDictEqual')
self.addTypeEqualityFunc(list, 'assertListEqual')
self.addTypeEqualityFunc(tuple, 'assertTupleEqual')
self.addTypeEqualityFunc(set, 'assertSetEqual')
self.addTypeEqualityFunc(frozenset, 'assertSetEqual')
try:
self.addTypeEqualityFunc(unicode, 'assertMultiLineEqual')
except NameError:
# No unicode support in this build
pass
You can see that unicode is registered to use assertMultiLineEqual(), but str is not registered for anything special. I have no idea why str is left out, but so far I have been happy with either of the following two methods.
Call Directly
If an 8-bit string isn't registered to use assertMultiLineEqual() by default, you can still call it directly.
def testString(self):
a = 'xax\nzzz'
b = 'xbx\nzzz'
self.assertMultiLineEqual(a, b)
Register String Type
You can also register it yourself. Just add an extra line to your test case's setUp() method. Do it once, and all your test methods will use the right method to test equality. If your project has a common base class for all test cases, that would be a great place to put it.
class TestAssertEqual(unittest.TestCase):
def setUp(self):
super(TestAssertEqual, self).setUp()
self.addTypeEqualityFunc(str, self.assertMultiLineEqual)
def testString(self):
a = 'xax\nzzz'
b = 'xbx\nzzz'
self.assertEqual(a, b)
def testUnicode(self):
a = u'xax\nzzz'
b = u'xbx\nzzz'
self.assertEqual(a, b)
Either of these methods will include highlighting when the string comparison fails.

Equality test failing for eval(repr(object)) in python3

Here's the __repr__ method inside a class called Grid
def __repr__(self):
return 'Grid(%r, %r)' % (self.rows, self.cols)
and I've put some basic tests inside a unittest module to check if eval can do an equality test without failure, and that looks like this:
# One of the tests inside a unittest.TestCase subclass
def test_grid(self):
grid = Grid(3, 4)
self.assertEqual(eval(repr(grid)), grid)
Now, this is the test report (I've isolated this test from others):
======================================================================
FAIL: test_grid (tests.test_core.TestCore)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/user/Desktop/sample/tests/test_core.py", line 14, in test_grid
self.assertEqual(eval(repr(grid)), grid)
AssertionError: Grid(3, 4) != Grid(3, 4)
----------------------------------------------------------------------
Ran 1 test in 0.003s
FAILED (failures=1)
The Assertion exception message is even more confusing to me. Isn't Grid(3, 4) != Grid(3, 4) supposed to be False?

I think the core of the issue is that you are creating a new object, and even though the values are the same inside - python cannot tell that they are, so it compares object by references. And they are different. I believe you need to override python's magic comparison operator to be able to compare by internal values.

Python 2.7 mock/patch: understanding assert_called_XYZ()

I'm relatively new to Python and unit testing in Python. From the Java world I know the concept of mocking but it seem to be much different from what I can see in Python.
I found this guide, which I found very helpful: http://www.voidspace.org.uk/python/mock/index.html
But as I wrote my (a bit more complex) tests with mocked out dependencies I noticed a strage behavior.
I decided to create a reduced, simple example which also does not work as I expect it.
Take a look at this, the result and my expectation I have added as comments:
import unittest
from mock import patch, Mock, MagicMock
class BasicTest(unittest.TestCase):
#patch("StringIO.StringIO")
def testSomethingNotWorkingAsExpected(self, StringIOMock):
StringIOMock.assert_called_once() # asserts, but why?
#patch("StringIO.StringIO")
def testSomethingSomehowWorking(self, StringIOMock):
# self.instantiateStringIO() # intentionally commented out
assert StringIOMock.called # does not assert (leading to failure of this test); as expected. If the above line is not commented, this asserts as expected.
def instantiateStringIO(self):
import StringIO
StringIO.StringIO()
Why is assert_called_once() asserting the instantiation of StringIO even it has not been instantiated yet?
And why does assert ClassMock.called bring the expected results?
Using assert not ... to assert a method has not been called I found here: Assert a function/method was not called using Mock. I inverted this pattern in my case by omitting the not.
Somewhere I found the pattern ClassMock.return_value to reference an instance. But I understand this as a way to manupulate the instance of a Mock before it will be called, not as a way to access the instance that might an underliing code have internally created. Or am I wrong?
My environment:
Python 2.7.3
mock 0.8.8
Fedora 19
Probably my understanding of the mock/patch thing is wrong. Could please someone aditionally explain what a class mock does and how it works?
Edit1: Added output
... and added paraphrase in parens to comment in testSomethingSomehowWorking
This is the output:
.F
======================================================================
FAIL: testSomethingSomehowWorking (test_test.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mock.py", line 1224, in patched
return func(*args, **keywargs)
File "test_test.py", line 15, in testSomethingSomehowWorking
assert StringIOMock.called # does not assert; as expected
AssertionError
----------------------------------------------------------------------
Ran 2 tests in 0.001s
FAILED (failures=1)

The method assert_called_once does not exist and it does not perform an assertion. It's no different from writing StringIOMock.assert_foo_bar_does_not_exist() or any other method. The mock library doesn't check whether the method called on the mock actually exists.
If you use assert_called_once_with then it fails as expected.
You can use the spec parameter to raise an error when you call a non-existent method:
#patch("StringIO.StringIO", spec=StringIO.StringIO)
def testSomethingNotWorkingAsExpected(self, StringIOMock):
StringIOMock.assert_called_once() # will fail as the method doesn't exist