I am trying to automate testing across several modules. All of these modules have a "test()" function with their unit-tests. Some modules are basic and their tests contain simple statements, but most modules have unittest or doctest. I'm having the most trouble dynamically importing and running doctest.
For example, here is a module sample.py
class sample:
"""
>>> import sample
>>> print sample.hello()
Hello
"""
def hello():
return "Hello"
def test():
import doctest
doctest.testmod(name=__name__, verbose=True)
And here is my file run_all_tests.py:
# assume I already have a list of all my files to test
for file in all_my_files:
temp_module = __import__(file)
temp_module.test()
This doesn't work and I always get this error:
1 items had no tests:
sample
0 tests in 1 items.
0 passed and 0 failed.
Test passed.
Please help me understand the problem.
Would Nose be a good alternative? I don't want to use it because I won't know beforehand if a module uses doctests, unittests or simple statements. But do let me know if that's not true/you have another alternative entirely!
Use doctest.DocTestSuite. It takes a module, extracts all doctests that exist there, and returns it as a unittest.TestSuite. Then, running the tests is a piece of pie. Your code would look like this:
for f in all_my_files:
temp_module = __import__(f)
test_suite = doctest.DocTestSuite(temp_module)
unittest.TextTestRunner().run(test_suite)
Why your code wasn't working
From doctest's testmod documentation:
Test examples in docstrings in functions and classes reachable from
module m (or module __main__ if m is not supplied or is None),
starting with m.__doc__.
So, since you left out the first argument (m), the module __main__ is passed to testmod. So the doctests that were run were the doctests in the module that contain the for loop. You can see this for yourself:
run_tests.py
"""
>>> print 'oops'
oops
"""
# assume I already have a list of all my files to test
for file in all_my_files:
temp_module = __import__(file)
temp_module.test()
If you run your example now (before fixing it) you'll see that you'll get:
Trying:
print 'oops'
Expecting:
oops
ok
1 items passed all tests:
1 tests in sample
1 tests in 1 items.
1 passed and 0 failed.
Test passed.
Clearly showing that the doctests running are those in run_tests.py. The name argument only changes the name that appears in the message (sample).
Related
CAUTION: This situation was happened because of a mistake. Check my answer.
I have a python file (myfile.py) which I want to test its content with Pytest. (The content is changing dynamically)
I've wrote this code:
import importlib
def test_myfile(capsys, monkeypatch):
monkeypatch.setattr('builtins.input', lambda s: "some_input")
# Create a module from the custom Problem file and import it
my_module = importlib.import_module("myfile")
# Rest of the test script
when I run the test I'm getting this error:
OSError: reading from stdin while output is captured
The error has been produced because there is an input() instruction in myfile.py and it means that mocking that function was futile.
My Question:
How can I mock out some functions inside a module I want to import?
Finally I found that I was looking wrong place to find a solution.
actually I use pytest to check a learner response and grade it. this was the way I did that at the moment I asked this question.
py.test test_runner.py test_case.py -v
This will test the user response which is saved in test_case.py ( I get the second parameter inside my test method and load its content and for example run a desired function ). Then I was examining the report of pytest to see if there was an error or failure to decide about the result. (pass/failure)
normally pytest was failing if there was an error in users code (e.g. syntax error) or if the test was failing (e.g. didn't return what it must return)
This time there was an error I didn't want to stop the test. I want to mock out input() function in users code. when I was running the command, before running my test method, pytest imported both files to collect test methods. It was failing to import test_case.py because of the input() function. It was not even reach the line I asked to mock that function and failed in the init stage.
At last to fix the problem I've added a parameter to py.test and now I'm running test process like this:
py.test test_runner.py --program test_case.py -v
In this form pytest doesn't look in test_case.py for test methods and doesn't fail.
hope this experience helps someone else
I have a Python package (Python 3.6, if it makes a difference) that I've designed to run as 'python -m package arguments' and I'd like to write unit tests for the __main__.py module. I specifically want to verify that it sets the exit code correctly. Is it possible to use runpy.run_module to execute my __main__.py and test the exit code? If so, how do I retrieve the exit code?
To be more clear, my __main__.py module is very simple. It just calls a function that has been extensively unit tested. But when I originally wrote __main__.py, I forgot to pass the result of that function to exit(), so I would like unit tests where the main function is mocked to make sure the exit code is set correctly. My unit test would look something like:
#patch('my_module.__main__.my_main', return_value=2)
def test_rc2(self, _):
"""Test that rc 2 is the exit code."""
sys.argv = ['arg0', 'arg1', 'arg2', …]
runpy.run_module('my_module')
self.assertEqual(mod_rc, 2)
My question is, how would I get what I’ve written here as ‘mod_rc’?
Thanks.
Misko Hevery has said before (I believe it was in Clean Code Talks: Don't Look for Things but I may be wrong) that he doesn't know how to effectively unit test main methods, so his solution is to make them so simple that you can prove logically that they work if you assume the correctness of the (unit-tested) code that they call.
For example, if you have a discrete, tested unit for parsing command line arguments; a library that does the actual work; and a discrete, tested unit for rendering the completed work into output, then a main method that calls all three of those in sequence is assuredly going to work.
With that architecture, you can basically get by with just one big system test that is expected to produce something other than the "default" output and it'll either crash (because you wired it up improperly) or work (because it's wired up properly and all of the individual parts work).
At this point, I'm dropping all pretense of knowing what I'm talking about. There is almost assuredly a better way to do this, but frankly you could just write a shell script:
python -m package args
test $? -eq [expected exit code]
That will exit with error iff your program outputs incorrectly, which TravisCI or similar will regard as build failing.
__main__.py is still subject to normal __main__ global behavior — which is to say, you can implement your __main__.py like so
def main():
# Your stuff
if __name__ == "__main__":
main()
and then you can test your __main__ in whatever testing framework you like by using
from your_package.__main__ import main
As an aside, if you are using argparse, you will probably want:
def main(arg_strings=None):
# …
args = parser.parse_args(arg_strings)
# …
if __name__ == "__main__":
main()
and then you can override arg strings from a unit test simply with
from your_package.__main__ import main
def test_main():
assert main(["x", "y", "z"]) == …
or similar idiom in you testing framework.
With pytest, I was able to do:
import mypkgname.__main__ as rtmain
where mypkgname is what you've named your app as a package/module. Then just running pytest as normal worked. I hope this helps some other poor soul.
I sometimes run into the following problem. I have a function, which returns something I am interested in and prints something I do not care about. E.g.
def f(x):
print('Some complicated printing stuff')
important_result = 42
return important_result
I want to write a doctest, which checks, that it indeed returns the correct result. But whose code is not obfuscated by the complicated printing stuff.
Something along the following lines would be cool:
def f(x):
"""
>>> f(0)
...
42
"""
print('Some complicated printing stuff')
important_result = 42
return important_result
Is there an elegant way to acomplish this?
Given that, your question has tag 'doctest' - I am assuming you want to run doctest for your function (enlighten me in the comment section, for any presumption or trivia) - as the text is ambiguous.
doctest basically works by looking for text in the form of Python interactive session. This text can be written in the docstring (like you have in the second code block/sample) or in a separate file.
using docstrings - all you have to do is specify few (or at-least one) examples of function i.e. passing the required parameters and the expected result, exactly in the same format as of Python interactive session. A good practice is to run your function in the interactive session and copy paste it in the docstring of the function.
To run the doctest, first specify the following code (for your mentioned code example, you just have to type the exact lines):
if __name__ = "__main__":
import doctest
doctest.testmod()
Line 1 - run the following code, only if the module (your *.py file)
is run as script i.e. not imported, etc. (a more detailed answer here)
Line 2 - importing the doctest module.
Line 3 - looks for any interactive session style text in the
docstring and runs it.
Once you have the above code included in your module (*.py file), just run in as a script:
python yourmodule.py
OR,
You can run the doctest directly (without the above 3 lines of code):
pyhton -m doctest yourmodule.py
using a separate file - add the following line in your file:
if __name__ = "__main__"
import doctest
doctest.testfile("somefile.txt")
it will identify and execute any interactive Python text in the file. By default the testfile() will look for 'somefile.txt' in the same directory where the module (.py file) is located (using options we can look for files in other locations).
Coming back to your question
I want to write a doctest, which checks, that it indeed returns the
correct result. But whose code is not obfuscated by the complicated
printing stuff. Something along the following lines would be cool:
NO (directly not possible) - Scenarios for doctest are set/written by specifying examples in form of Python interactive sessions in your docstring - exactly the i.e. as mentioned above a good practice is to run your function with various inputs in the interactive session and copy paste those lines in the docstring - and all the print statements will definitely be there, and part of the out for doctest to mark test as passed.
Indirect ways:
use an optional parameter in function, for example printing - the overhead 'd be you'll need to make changes to the function i.e. moving the print statements under if.
def f(x, printing=True):
"""
f(0, printing=False)
...
42
"""
if printing:
print('Some complicated printing stuff')
important_result = 42
return important_result
pass the None argument to print function - beware, it'll be passed to all the print calls in the module i.e. the whole *.py file.
def f(x):
"""
f(0)
...
42
"""
print('Some complicated printing stuff')
important_result = 42
return important_result
if name == 'main':
import doctest
print = lambda *args, **kwargs: None
doctest.testmod()
Source: jonrsharpe's answer
To get some feedback, I ended up printing the result of doctest.testmod()
Hello I have a test module like the following under "test.py":
class TestBasic(unittest.TestCase):
def setUp(self):
# set up in here
class TestA(TestBasic):
def test_one(self):
self.assertEqual(1,1)
def test_two(self):
self.assertEqual(2,1)
if __name__ == "__main__":
unittest.main()
And this works pretty good, but I need a way to print which test passed, for example I could print the output to the console:
test_one: PASSED
test_two: FAILED
Now the twist, I could add a print statement right after the self.assertEqual() and that would be a passed test and I could just print it, but I need to run the test from a different module, let's say "test_reporter.py" where I have something like this:
import test
suite = unittest.TestLoader().loadTestsFromModule(test)
results = unittest.TextTestRunner(verbosity=0).run(suite)
at this point with results is when I build a report.
So please any suggestion is welcome
Thanks !!
Like Corey's comment mentioned, if you set verbosity=2 unittest will print the result of each test run.
results = unittest.TextTestRunner(verbosity=2).run(suite)
If you want a little more flexibility - and you might since you are creating suites and using test runners - I recommend that you take a look at Twisted Trial. It extends Python's unittest module and provides a few more assertions and reporting features.
Writing your tests will be exactly the same (besides subclassing twisted.trial.unittest.TestCase vs python's unittest) so your workflow won't change. You can still use your TestLoader but you'll have the options of many more TestReporters http://twistedmatrix.com/documents/11.1.0/api/twisted.trial.reporter.html.
For example, the default TestReporter is TreeReporter which returns the following output:
I am trying to get into testing in Python using the doctest module. At the moment I do
Write the tests for the functions.
implement the functions code.
If Tests pass, write more tests and more code.
When the function is done move on to the next function to implement.
So after 3 or 4 (independent) functions in the same module with many tests I get a huge output by doctest. And it is a little annoysing.
Is there a way to tell doctest "don't test functions a(), b() and c()", so that it runs only the unmarked functions?
I only found the doctest.SKIP flag, which is not sufficient for my needs. I would have to place this flag in a lot of lines. And if I would want to check a marked function again, I would have to go manually through the code and remove any flag I set inside.
looks like you could pass the function to run_docstring_examples:
def f(a, b, c):
'''
>>> f(1,2,3)
42
'''
if __name__ == '__main__':
import doctest
# doctest.testmod()
doctest.run_docstring_examples(f, globals())
example found via google.
I put together a helper script to make this a little less painful. It can be installed using:
pip install doctestfn
It can then be used as follows:
usage: doctestfn [-h] [-v] module function
Run doctests for one function
positional arguments:
module Module to load
function Function to test
optional arguments:
-h, --help show this help message and exit
-v, --verbose Enable verbose doctest output