I'm trying to set up a grading script for an intro CS class using unittest. Essentially, students submit a python file student.py, which has some number of functions in it that are generally interdependent (meaning that func3() may use func1() in it's computations).
I'm writing unit tests for each method by comparing the output of student.func1 to the output of correct.func1, a method that's known to be the correct implementation (from a file correct.py).
As an example, say that func2 uses func1 in it's computation.
So, either by default or upon student.func1 failing some test, I want to overwrite student.func1 with correct.func1, so student.func2 uses a known correct implementation (and therefore isn't just wrong by default). How could I go about doing that? It seems like the setUp() and tearDown() are similar to what I want, but I don't know how to "unimport" modules in python, and haven't found any resources about it so far.
I'm interested in both the case where student.py contains classes, and func1,func2 are methods of a specific class, and when func1 and func2 are just defined generically within student.py.
The easiest way would be to import student into your module, then catch an AssertionError if a test fails, and replace the bad code inside the student module with your own good code:
import student
import unittest
def safe_f1():
print("Safe f1")
return 1
class TestSomething(unittest.TestCase):
def test_f1(self):
try:
self.assertEqual(student.func1(), 1)
except AssertionError:
student.func1 = safe_f1
raise
def test_f2(self):
self.assertEqual(student.func2(), 2)
Here's a dummy student.py that fails/works:
def func1():
print("Bad f1")
return 2
def func2():
return func1() + 1
return 2
When I run this, I get:
$ python -m unittest test.py
Bad f1
FSafe f1
.
======================================================================
FAIL: test_f1 (test.TestSomething)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/austin/Code/so/test.py", line 13, in test_f1
self.assertEqual(student.func1(), 1)
AssertionError: 2 != 1
----------------------------------------------------------------------
Ran 2 tests in 0.001s
FAILED (failures=1)
Related
I'm a Python instructor and I wanted to give my students a task: write a function that computes the average of a list using a for loop and the range object.
I wanted to run a test on their function to see whether it actually uses the range object. How can I do that?
It should be something like this:
def avg(L):
Pass
def test_range(avg):
...
If avg contains range, then test_range should return True.
I tried solutions that utilize func_code, but apparantly range doesn't have that.
You can use Python's unittest.mock module to wrap around the range function from the builtins module, and then let your test assert that the wrapped range was indeed called.
For example, using Python's unittest framework for writing the test:
import builtins
import unittest
from unittest.mock import patch
# I don't know what L is supposed to be, and I know that there
# are better ways to compute average of a list, but the code
# for calculating the average is not important for this question.
def avg(L):
total = 0
for index in range(len(L)):
total += L[index]
return total / len(L)
class TestAverage(unittest.TestCase):
def test_avg(self):
with patch("builtins.range", wraps=builtins.range) as wrapped_patch:
expected = 47
actual = avg([1,49,91])
self.assertEqual(expected, actual)
wrapped_patch.assert_called()
if __name__ == '__main__':
unittest.main()
$ python -m unittest -v main.py
test_avg (main.TestAverage) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
It uses unittest.mock's patch targetting the builtins.range function. Normally, patch is used in tests to replace the behavior and/or return value of the target, but in this case, you can pass wraps=builtins.range (which gets passed to the underlying Mock object), which means "I just want to spy on the call, but not modify its behavior":
wraps: Item for the mock object to wrap. If wraps is not None then calling the Mock will pass the call through to the wrapped object (returning the real result).
By wrapping it in a Mock object, you can use any of Mock's assert functions to check calls to range, like assert_called which checks whether the target was called at least once. You can be more specific by asserting the number of times range was called:
self.assertTrue(wrapped_patch.call_count == 1)
The assertion would fail if it wasn't called at all:
# Here, `range` wasn't used at all.
def avg(L):
return sum(L) / len(L)
class TestAverage(unittest.TestCase):
# same as the code above
$ python -m unittest -v main.py
test_avg (main.TestAverage) ... FAIL
======================================================================
FAIL: test_avg (main.TestAverage)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/main.py", line 15, in test_avg
wrapped_patch.assert_called()
File "/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/unittest/mock.py", line 888, in assert_called
raise AssertionError(msg)
AssertionError: Expected 'range' to have been called.
The most important thing to note when using patch, is to know exactly where to patch. In this case, you can check the docs or use __module__ to know range's module:
>>> range
<class 'range'>
>>> range.__module__
'builtins'
But the test is a bit naive, because it can still pass even though avg didn't really use range:
def avg(L):
range(len(L)) # Called but really unused. Sneaky!
return sum(L) / len(L)
class TestAverage(unittest.TestCase):
# same as the code above
$ python -m unittest -v main.py
test_avg (main.TestAverage) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
A slightly confusing workaround would be a test that "breaks" range such that, if the function was really using range, then it wouldn't work anymore:
def avg(L):
range(len(L)) # Called but really unused. Sneaky!
return sum(L) / len(L)
class TestAverage(unittest.TestCase):
def test_avg(self):
# same as above
def test_avg_is_really_using_range(self):
L = [10,20,90]
# Is it returning the correct result?
self.assertEqual(avg(L), 40)
# OK, but did it really use `range`?
# Let's try breaking `range` so it always yields 0,
# so we expect the return value to be *different*
with patch("builtins.range", return_value=[0,0,0]):
self.assertNotEqual(avg(L), 40)
So, if avg was sneakily calling but not really using range, then test_avg_is_really_using_range would now fail because avg still yields the correct value even with a broken range:
$ python -m unittest -v main.py
test_avg (main.TestAverage) ... ok
test_avg_really_using_range (main.TestAverage) ... FAIL
======================================================================
FAIL: test_avg_really_using_range (main.TestAverage)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/main.py", line 19, in test_avg_really_using_range
self.assertNotEqual(avg(L), 40)
AssertionError: 40.0 == 40
Lastly, as a side note, I'm using assertEqual here in all the example because the test for the return value is not the focus, but do read up on proper ways to assert possible float values, ex. How to perform unittest for floating point outputs? - python
Here's the __repr__ method inside a class called Grid
def __repr__(self):
return 'Grid(%r, %r)' % (self.rows, self.cols)
and I've put some basic tests inside a unittest module to check if eval can do an equality test without failure, and that looks like this:
# One of the tests inside a unittest.TestCase subclass
def test_grid(self):
grid = Grid(3, 4)
self.assertEqual(eval(repr(grid)), grid)
Now, this is the test report (I've isolated this test from others):
======================================================================
FAIL: test_grid (tests.test_core.TestCore)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/user/Desktop/sample/tests/test_core.py", line 14, in test_grid
self.assertEqual(eval(repr(grid)), grid)
AssertionError: Grid(3, 4) != Grid(3, 4)
----------------------------------------------------------------------
Ran 1 test in 0.003s
FAILED (failures=1)
The Assertion exception message is even more confusing to me. Isn't Grid(3, 4) != Grid(3, 4) supposed to be False?
I think the core of the issue is that you are creating a new object, and even though the values are the same inside - python cannot tell that they are, so it compares object by references. And they are different. I believe you need to override python's magic comparison operator to be able to compare by internal values.
The docs say:
If you would rather define test functions directly at module level you can also use the following functions to implement fixtures:
def setup_function(function):
""" setup any state tied to the execution of the given function.
Invoked for every test function in the module.
"""
def teardown_function(function):
""" teardown any state that was previously setup with a setup_function
call.
"""
but actually it's unclear how you're supposed to use them.
I tried putting them in my test file exactly as shown above, but they don't get called. Looking closer at them, with that function arg they look like decorators. So I tried to find a way to do:
#pytest.setup_function
def my_setup():
# not called
I couldn't find anywhere to import one from as a decorator though, so that can't be right.
I found this thread on grokbase where someone explains you can do:
#pytest.fixture(autouse=True)
def setup_function(request):
# this one gets called
It works, but it doesn't seem related to the docs any more... there's no reason to call it setup_function here.
You just implement the body of setup_function(), that function is called for each function whose name starts with test_ and the function is handed in as a parameter:
def setup_function(fun):
print ("in setup function: " + fun.__name__)
def test_test():
assert False
This will give as output, when run with py.test:
============================= test session starts ==============================
platform linux2 -- Python 2.7.6 -- pytest-2.3.5
collected 1 items
test/test_test.py F
=================================== FAILURES ===================================
__________________________________ test_test ___________________________________
def test_test():
> assert False
E assert False
test/test_test.py:7: AssertionError
------------------------------- Captured stdout --------------------------------
in setup function: test_test
=========================== 1 failed in 0.01 seconds ===========================
The line before the last one shows the output from the actual call to setup_function()
A slightly more useful example, actually doing something that influences the test function:
def setup_function(function):
function.answer = 17
def teardown_function(function):
del function.answer
def test_modlevel():
assert modlevel[0] == 42
assert test_modlevel.answer == 17
This was taken from py.test's own tests, always a good (and hopefully complete) set of examples of all the features of py.test.
I'm relatively new to Python and unit testing in Python. From the Java world I know the concept of mocking but it seem to be much different from what I can see in Python.
I found this guide, which I found very helpful: http://www.voidspace.org.uk/python/mock/index.html
But as I wrote my (a bit more complex) tests with mocked out dependencies I noticed a strage behavior.
I decided to create a reduced, simple example which also does not work as I expect it.
Take a look at this, the result and my expectation I have added as comments:
import unittest
from mock import patch, Mock, MagicMock
class BasicTest(unittest.TestCase):
#patch("StringIO.StringIO")
def testSomethingNotWorkingAsExpected(self, StringIOMock):
StringIOMock.assert_called_once() # asserts, but why?
#patch("StringIO.StringIO")
def testSomethingSomehowWorking(self, StringIOMock):
# self.instantiateStringIO() # intentionally commented out
assert StringIOMock.called # does not assert (leading to failure of this test); as expected. If the above line is not commented, this asserts as expected.
def instantiateStringIO(self):
import StringIO
StringIO.StringIO()
Why is assert_called_once() asserting the instantiation of StringIO even it has not been instantiated yet?
And why does assert ClassMock.called bring the expected results?
Using assert not ... to assert a method has not been called I found here: Assert a function/method was not called using Mock. I inverted this pattern in my case by omitting the not.
Somewhere I found the pattern ClassMock.return_value to reference an instance. But I understand this as a way to manupulate the instance of a Mock before it will be called, not as a way to access the instance that might an underliing code have internally created. Or am I wrong?
My environment:
Python 2.7.3
mock 0.8.8
Fedora 19
Probably my understanding of the mock/patch thing is wrong. Could please someone aditionally explain what a class mock does and how it works?
Edit1: Added output
... and added paraphrase in parens to comment in testSomethingSomehowWorking
This is the output:
.F
======================================================================
FAIL: testSomethingSomehowWorking (test_test.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mock.py", line 1224, in patched
return func(*args, **keywargs)
File "test_test.py", line 15, in testSomethingSomehowWorking
assert StringIOMock.called # does not assert; as expected
AssertionError
----------------------------------------------------------------------
Ran 2 tests in 0.001s
FAILED (failures=1)
The method assert_called_once does not exist and it does not perform an assertion. It's no different from writing StringIOMock.assert_foo_bar_does_not_exist() or any other method. The mock library doesn't check whether the method called on the mock actually exists.
If you use assert_called_once_with then it fails as expected.
You can use the spec parameter to raise an error when you call a non-existent method:
#patch("StringIO.StringIO", spec=StringIO.StringIO)
def testSomethingNotWorkingAsExpected(self, StringIOMock):
StringIOMock.assert_called_once() # will fail as the method doesn't exist
I have a file called test_web.py containing a class TestWeb and many methods named like test_something().
I can run every test in the class like so:
$ nosetests test_web.py
...
======================================================================
FAIL: checkout test
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/me/path/here/test_web.py", line 187, in test_checkout
...
But I can’t seem to run individual tests. These give me “No such test” errors when run in the same PWD:
$ nosetests test_web.py:test_checkout
$ nosetests TestWeb:test_checkout
What could be wrong here?
You must specify it like so: nosetests <file>:<Test_Case>.<test_method>, or
nosetests test_web.py:TestWeb.test_checkout
See the docs
You can also specify a module:
nosetests tests.test_integration:IntegrationTests.test_user_search_returns_users
Specifying names on the command line like the other answers suggest does work and is useful. However, when I'm in the midst of writing tests, I often find that I want to run just the test I'm working on, and the names that I would have to write on the command line get pretty long and cumbersome to write. In such case, I prefer to use a custom decorator and flag.
I define wipd ("work in progress decorator") like this:
from nose.plugins.attrib import attr
def wipd(f):
return attr('wip')(f)
This defines a decorator #wipd which will set the wip attribute on objects it decorates. For instance:
import unittest
class Test(unittest.TestCase):
#wipd
def test_something(self):
pass
Then -a wip can be used at the command line to narrow the execution of the test to the ones marked with #wipd.
Note on names...
I'm using the name #wipd for the decorator rather than #wip to avoid this kind of problem:
import unittest
class Test(unittest.TestCase):
from mymodule import wip
#wip
def test_something(self):
pass
def test_something_else(self):
pass
The import will make the wip decorator a member of the class, and all tests in the class will be selected. The attrib plugin checks the parent class of a test method to see if the attribute selected exists there too, and the attributes that are created and tested by attrib do not exist in a segregated space. So if you test with -a foo and your class contains foo = "platypus", then all tests in the class will be selected by the plugin.
To run multiple specific tests, you can just add them to the command line, separated by space.
nosetests test_web.py:TestWeb.test_checkout test_web.py:TestWeb.test_another_checkout
In my tests, specifying tests with module names do not work
You must specify the actual path to the .py:
nosetests /path/to/test/file.py:test_function
This with nose==1.3.7
My requirement was to run a single test in a test file that was in another windows directory - this was done from the anaconda command prompt as follows:
ran nosetests from:
(base) C:\Users\ABC\Documents\work\
but test_MyTestFile.py and methodsFile.py were in:
(base) C:\Users\ABC\Documents\work\daily\
run single test by including path with quotes as follows:
(base) C:\Users\ABC\Documents\work>nosetests "daily\\test_MyTestFile.py:MyTestClass.test_add_integers"
test_MyTestFile.py looked like this:
import methodsFile
import unittest
class MyTestClass(unittest.TestCase):
def test_add_integers(self):
assert methodsFile.add(5, 3) == 8
def test_add_integers_zero(self):
assert methodsFile.add(3, 0) == 3
methodsFile.py looked like this:
def add(num1, num2):
return num1 + num2