We Keep Coding

Unit testing __main__.py

I have a Python package (Python 3.6, if it makes a difference) that I've designed to run as 'python -m package arguments' and I'd like to write unit tests for the __main__.py module. I specifically want to verify that it sets the exit code correctly. Is it possible to use runpy.run_module to execute my __main__.py and test the exit code? If so, how do I retrieve the exit code?
To be more clear, my __main__.py module is very simple. It just calls a function that has been extensively unit tested. But when I originally wrote __main__.py, I forgot to pass the result of that function to exit(), so I would like unit tests where the main function is mocked to make sure the exit code is set correctly. My unit test would look something like:
#patch('my_module.__main__.my_main', return_value=2)
def test_rc2(self, _):
"""Test that rc 2 is the exit code."""
sys.argv = ['arg0', 'arg1', 'arg2', …]
runpy.run_module('my_module')
self.assertEqual(mod_rc, 2)
My question is, how would I get what I’ve written here as ‘mod_rc’?
Thanks.

Misko Hevery has said before (I believe it was in Clean Code Talks: Don't Look for Things but I may be wrong) that he doesn't know how to effectively unit test main methods, so his solution is to make them so simple that you can prove logically that they work if you assume the correctness of the (unit-tested) code that they call.
For example, if you have a discrete, tested unit for parsing command line arguments; a library that does the actual work; and a discrete, tested unit for rendering the completed work into output, then a main method that calls all three of those in sequence is assuredly going to work.
With that architecture, you can basically get by with just one big system test that is expected to produce something other than the "default" output and it'll either crash (because you wired it up improperly) or work (because it's wired up properly and all of the individual parts work).
At this point, I'm dropping all pretense of knowing what I'm talking about. There is almost assuredly a better way to do this, but frankly you could just write a shell script:
python -m package args
test $? -eq [expected exit code]
That will exit with error iff your program outputs incorrectly, which TravisCI or similar will regard as build failing.

__main__.py is still subject to normal __main__ global behavior — which is to say, you can implement your __main__.py like so
def main():
# Your stuff
if __name__ == "__main__":
main()
and then you can test your __main__ in whatever testing framework you like by using
from your_package.__main__ import main
As an aside, if you are using argparse, you will probably want:
def main(arg_strings=None):
# …
args = parser.parse_args(arg_strings)
# …
if __name__ == "__main__":
main()
and then you can override arg strings from a unit test simply with
from your_package.__main__ import main
def test_main():
assert main(["x", "y", "z"]) == …
or similar idiom in you testing framework.

With pytest, I was able to do:
import mypkgname.__main__ as rtmain
where mypkgname is what you've named your app as a package/module. Then just running pytest as normal worked. I hope this helps some other poor soul.

How can I test the standard input and standard output in Python Script with a Unittest test?

I'm trying to test a Python script (2.7) where I work with the standar input (readed with raw_input() and writed with a simple print) but I don't find how do this and I'm sure that this issue is very simple.
This is a very very very resume code of my script:
def example():
number = raw_input()
print number
if __name__ == '__main__':
example()
I want to write a unittest test to check this, but I don't find how. I've trying with StringIO and other things but I don't find the solution to do this really simple.
Somebody have a idea?
PD: Of course in the real script I use data blocks with several lines and other kind of data.
Thank you so much.
EDIT:
Thank you so much for the first really specific answer, it works perfectly, only I've had a little problem importing StringIO, because I was doing import StringIO and I needed to import like from StringIO import StringIO (I don't understand really why), but be that as It may, it works.
But I I've found another problem using this way, in my project I need test a scripts with this way (that work perfectly thanks to your support) but I want do this:
I have a file with a lot of test to pass over a script, so I open the file and read blocks of info with their result blocks and I would like to do that the code will be able to process a block checking their result and do the same with other and another...
Something like this:
class Test(unittest.TestCase):
...
#open file and process saving data like datablocks and results
...
allTest = True
for test in tests:
stub_stdin(self, test.dataBlock)
stub_stdouts(self)
runScrip()
if sys.stdout.getvalue() != test.expectResult:
allTest = False
self.assertEqual(allTest, True)
I know that maybe unittest doesn't has sense now, but you can do a idea about I want. So, this way fails and I don't know why.

Typical techniques involve mocking the standard sys.stdin and sys.stdout with your desired items. If you do not care for Python 3 compatibility you can just use the StringIO module, however if you want forward thinking and is willing to restrict to Python 2.7 and 3.3+, supporting for this both Python 2 and 3 in this way becomes possible without too much work through the io module (but requires a bit of modification, but put this thought on hold for now).
Assuming you already have a unittest.TestCase going, you can create a utility function (or method in the same class) that will replace sys.stdin/sys.stdout as outlined. First the imports:
import sys
import io
import unittest
In one of my recent projects I've done this for stdin, where it take a str for the inputs that the user (or another program through pipes) will enter into yours as stdin:
def stub_stdin(testcase_inst, inputs):
stdin = sys.stdin
def cleanup():
sys.stdin = stdin
testcase_inst.addCleanup(cleanup)
sys.stdin = StringIO(inputs)
As for stdout and stderr:
def stub_stdouts(testcase_inst):
stderr = sys.stderr
stdout = sys.stdout
def cleanup():
sys.stderr = stderr
sys.stdout = stdout
testcase_inst.addCleanup(cleanup)
sys.stderr = StringIO()
sys.stdout = StringIO()
Note that in both cases, it accepts a testcase instance, and calls its addCleanup method that adds the cleanup function call that will reset them back to where they were when the duration of a test method is concluded. The effect is that for the duration from when this was invoked in the test case until the end, sys.stdout and friends will be replaced with the io.StringIO version, meaning you can check its value easily, and don't have to worry about leaving a mess behind.
Better to show this as an example. To use this, you can simply create a test case like so:
class ExampleTestCase(unittest.TestCase):
def test_example(self):
stub_stdin(self, '42')
stub_stdouts(self)
example()
self.assertEqual(sys.stdout.getvalue(), '42\n')
Now, in Python 2, this test will only pass if the StringIO class is from the StringIO module, and in Python 3 no such module exists. What you can do is use the version from the io module with a modification that makes it slightly more lenient in terms of what input it accepts, so that the unicode encoding/decoding will be done automatically rather than triggering an exception (such as print statements in Python 2 will not work nicely without the following). I typically do this for cross compatibility between Python 2 and 3:
class StringIO(io.StringIO):
"""
A "safely" wrapped version
"""
def __init__(self, value=''):
value = value.encode('utf8', 'backslashreplace').decode('utf8')
io.StringIO.__init__(self, value)
def write(self, msg):
io.StringIO.write(self, msg.encode(
'utf8', 'backslashreplace').decode('utf8'))
Now plug your example function plus every code fragment in this answer into one file, you will get your self contained unittest that works in both Python 2 and 3 (although you need to call print as a function in Python 3) for doing testing against stdio.
One more note: you can always put the stub_ function calls in the setUp method of the TestCase if every single test method requires that.
Of course, if you want to use various mocks related libraries out there to stub out stdin/stdout, you are free to do so, but this way relies on no external dependencies if this is your goal.
For your second issue, test cases have to be written in a certain way, where they must be encapsulated within a method and not at the class level, your original example will fail. However you might want to do something like this:
class Test(unittest.TestCase):
def helper(self, data, answer, runner):
stub_stdin(self, data)
stub_stdouts(self)
runner()
self.assertEqual(sys.stdout.getvalue(), answer)
self.doCleanups() # optional, see comments below
def test_various_inputs(self):
data_and_answers = [
('hello', 'HELLOhello'),
('goodbye', 'GOODBYEgoodbye'),
]
runScript = upperlower # the function I want to test
for data, answer in data_and_answers:
self.helper(data, answer, runScript)
The reason why you might want to call doCleanups is to prevent the cleanup stack from getting as deep as all the data_and_answers pairs are there, but that will pop everything off the cleanup stack so if you had any other things that need to be cleaned up at the end this might end up being problematic - you are free to leave that there as all of the stdio related objects will be restored at the end in the same order, so the real one will always be there. Now the function I wanted to test:
def upperlower():
raw = raw_input()
print (raw.upper() + raw),
So yes, a bit of explanation for what I did might help: remember within a TestCase class, the framework relies strictly on the instance's assertEqual and friends for it to function. So to ensure testing being done at the right level you really want to call those asserts all the time so that helpful error messages will be shown at the moment the error occurred with the inputs/answers that didn't quite show up right, rather than until the very end like what you did with the for loop (that will tell you something was wrong, but not exactly where out of the hundreds and now you are mad). Also the helper method - you can call it anything you want, as long as it doesn't start with test because then the framework will try to run it as one and it will fail terribly. So just follow this convention and you can basically have templates within your test case to run your test with - you can then use it in a loop with a bunch of inputs/outputs like what I did.
As for your other question:
only I've had a little problem importing StringIO, because I was doing import StringIO and I needed to import like from StringIO import StringIO (I don't understand really why), but be that as It may, it works.
Well, if you look at my original code I did show you how did import io and then overrode the io.StringIO class by defining class StringIO(io.StringIO). However it works for you because you are doing this strictly from Python 2, whereas I do try to target my answers to Python 3 whenever possible given that Python 2 will (probably definitely this time) not be supported in less than 5 years. Think of the future users that might be reading this post who had similar problem as you. Anyway, yes, the original from StringIO import StringIO works, as that's the StringIO class from the StringIO module. Though from cStringIO import StringIO should work as that imports the C version of the StringIO module. It works because they all offer close enough interfaces, and so they will basically work as intended (until of course you try to run this under Python 3).
Again, putting all this together along with my code should result in a self-contained working test script. Do remember to look at documentation and follow the form of the code, and not invent your own syntax and hoping things to work (and as for exactly why your code didn't work, because the "test" code was defined at where the class was being constructed, so all of that was executed while Python was importing your module, and since none of the things that are needed for the test to run are even available (namely the class itself doesn't even exist yet), the whole thing just dies in fits of twitching agony). Asking questions here help too, even though the issue you face is something really common, not having a quick and simple name to search for your exact problem does make it difficult to figure out where you went wrong, I supposed? :) Anyway good luck, and good on you for taking the effort to test your code.
There are other methods, but given that the other questions/answers I looked at here at SO doesn't seem to help, I hope this one this. Other ones for reference:
How to supply stdin, files and environment variable inputs to Python unit tests?
python mocking raw input in unittests
Naturally, it bares repeating that all of this can be done using unittest.mock available in Python 3.3+ or the original/rolling backport version on pypi, but given that those libraries hides some of the intricacies on what actually happens, they may end up hiding some of the details on what actually happens (or need to happen) or how the redirection actually happens. If you want, you can read up on unittest.mock.patch and go down slightly to the StringIO patching sys.stdout section.

How should I indicate that a test hasn't been written yet in Python?

I'm doing TDD using Python and the unittest module. In NUnit you can Assert.Inconclusive("This test hasn't been written yet").
So far I haven't been able to find anything similar in Python to indicate that "These tests are just placeholders, I need to come back and actually put the code in them."
Is there a Pythonic pattern for this?

With the new and updated unittest module you can skip tests:
#skip("skip this test")
def testSomething(self):
pass # TODO
def testBar(self):
self.skipTest('We need a test here, really')
def testFoo(self):
raise SkipTest('TODO: Write a test here too')
When the test runner runs these, they are counted separately ("skipped: (n)").

I would not let them pass or show OK, because you will not find them easily back.
Maybe just let them fail and the reason (not written yet), which seems logical because you have a test that is not finished.

I often use self.fail as a todo list
def test_some_edge_case(self):
self.fail('Need to check for wibbles')
Works well for me when I'm doing tdd.

Auto generate doctest output with Sphinx extension

I think I am missing something about the sphinx extension for doctest.
The typical example in the documentation is:
.. doctest::
>>> print 1
1
Isn't there a way to let sphinx generate the output (here: 1) automatically?
As far as I understood, it is possible to run:
$ make doctest
which has the effect to test the code snippets, and compare the real output with the expected output. For example, if you have
.. doctest::
>>> print 1
3
doctest will warn you that it got 1 while it was expecting 3.
Instead, I would like sphinx to insert the real output alone in my docstring or in my .rst file. For example, if we have something like:
.. doctest::
>>> print 1
>>> print [2*x for x in range(3)]
I would like that when we run make doctest with an option, it changes the docstring to:
.. doctest::
>>> print 1
1
>>> print [2*x for x in range(3)]
[0,2,4]
I'm sure it's possible, and would be very convenient!

I have to strongly (but kindly) advise against what you're trying to do.
What you're asking is against the "test part" of the doctest module:
The doctest module searches for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown.
These tests have a reasons to be if you write the input and the expected output and let Python check if the expected output match the actual output.
If you let Python produce the expected output, well.. it will no longer be expected (by the user/author), so the doctests will never fail, hence those tests will be useless.
Note: If inside a function there's no logic (if/else, while-loops, appends, etc..) there's no need to test them. And tests must not reproduce the testing logic, otherwise they're not testing the function anymore.
I found this video about test driven development very interesting, maybe it could be of interest to you if you want to know more about this argument.

Here is a suggestion on how you could achieve what I suspect you might be looking for:
Doug Hellmann has written an interesting article called Writing Technical Documentation with Sphinx, Paver, and Cog. It has a section describing how the Cog tool can be used to automatically run code examples and capture the output for inclusion in Sphinx-built documentation.
There is also a contributed Sphinx extension called autorun that can execute code in a special
runblock directive and attach the output to the documentation.

This feature is available as part of pytest-accept, and extension of pytest: https://github.com/max-sixty/pytest-accept
Quote:
pytest-accept is a pytest plugin for automatically updating doctest
outputs. It runs doctests, observes the generated outputs, and writes
them to the doctests' documented outputs.
It's designed for a couple of use cases:
People who work with doctests and don't enjoy manually copying generated outputs from the pytest error log and pasting them into
their doctests' documented outputs. pytest-accept does the copying &
pasting for you.
People who generally find writing tests a bit annoying, and prefer to develop by "running the code and seeing whether it works". This
library aims to make testing a joyful part of that development loop.

Is using Python modules main function for validation testing a bad idea?

I'll quickly explain exactly what I mean by this.
I'm working on a project using python, where I have multiple modules doing segments of work. Let's say for example I have a module called Parser.py and this module has a function parseFile() which my main module Main.py calls in order to parse some files.
As of right now, I'm using a main method inside of the Parser.py
if __name__ == "__main__":
line_list = parseFile(sys.argv[1])
out_file = open(sys.argv[2], "w")
for i in range(len(line_list)):
out_file.write(line_list[i].get_string(True))
It's not important what exactly the parsing does, but the important part is if you call it, the first argument will be the input file for the parsing, the second argument is the output file for parsing.
So, what I'm doing essentially, is I'm using a batch file to validate the results of my parser by a typical input, output, baseline system...
ECHO Set the test, source, input, output and baseline directories
set TESTDIR=%CD%
set SRCDIR=%CD%\..\pypro\src
set INDIR=%CD%\input
set OUTDIR=%CD%\output
set BASEDIR=%CD%\baseline
:: Parser.py main method is base for unit testing on parsing
ECHO Begin Parser testing
cd %INDIR%\Parser
FOR %%G IN (*.psma) DO %SRCDIR%\Parser.py %%G %OUTDIR%\Parser\%%G
ECHO Parser testing complete
cd %TESTDIR%
"C:\Program Files\WinMerge\winmergeU.exe" "%OUTDIR%" "%BASEDIR%"
As you can see it diffs the results against the baseline, so if anything is changed the programmer knows it is no longer valid, or the requirements are wrong.
Is there anything wrong with this method? I did it because it would be easy. My plan is to continue doing this with as many modules that I can which are valid and make sense to do this way, as well as a suite of pyunit tests inside pydev...

I think it's a good idea, and it does seem to be a common use case for if __name__ == '__main__' construct. Though this is a more usual structure:
def main(argv=None):
if argv is None:
argv = sys.argv
# etc.
if __name__ == "__main__":
sys.exit(main() or 0)
This gives you the additional flexibility to use your main from within the interactive interpreter. There are a few more nice examples from Guido and others here.

Personally, what I do in these situations is creating test cases (although these would could more as integration test cases and not only unit test cases).
So, usually (in my workflow), those would be regular test cases (which diff the actual output with the expected output). Although probably in a separate source folder which is not run as often as the unit-test cases.
The bad part of having it as the __main__ is that you'll have to remember to run it as the entry point and you'll probably forget to do it later on as the project grows and you have many of those files -- or at least have a test case that calls that main() :)