We Keep Coding

Why doesn't the test work without if __name__ == '__main__'?

I can't figure out why without the line if __name__ == '__main__': before unittest.main() the test does not find?
I am using the latest version of PyCharm. I know that in order for the test to work in PyCharm, you can not add these lines at all, but I want to deal with the logic itself: why without the line if __name__ == '__main__': the result is as in the screenshot, but if you add it, then everything works?
Code:
import unittest
from name_function import get_formatted_name
class NamesTestCase(unittest.TestCase):
"""Tests for 'name_function.py'."""
def test_first_last_name(self):
"""Are names like 'Janis Joplin' working correctly?"""
formatted_name = get_formatted_name('janis', 'joplin')
self.assertEqual(formatted_name, 'Janis Joplin')
unittest.main()
There is only one function in the name_function module:
def get_formatted_name(first, last):
"""Builds a formatted full name."""
full_name = f"{first} {last}"
return full_name.title()
Result:
No tests were found
/Users/xxx/Documents/PycharmProjects/Book/venv/bin/python
"/Applications/PyCharm
CE.app/Contents/plugins/python-ce/helpers/pycharm/_jb_unittest_runner.py"
--path /Users/xxx/Documents/PycharmProjects/Book/testing.py
Testing started at 00:22 ...
-------------------------------------------------------------------> Ran 0 tests in 0.000s
OK
Launching unittests with arguments python -m unittest
/Users/xxx/Documents/PycharmProjects/Book/testing.py in
/Users/xxx/Documents/PycharmProjects/Book
Process finished with exit code 0
Empty suite
Empty suite
I am running the testing.py module as the main program, but judging by the answer line PyCharm is running the test via python -m unittest testing.NamesTestCase
I additionally checked the value of the global variable __name__ and indeed it has the value testing, as if testing was imported. Although I launch it initially.
Please explain why in this case the startup script differs from the standard one and when testing.py starts it runs it through unittest? I really want to finally understand this issue. Also don't understand why, in this case, if it initially runs through unittest, unittest.main() doesn't run normally without additional checking if __name__ == '__main__':?

Unit testing __main__.py

I have a Python package (Python 3.6, if it makes a difference) that I've designed to run as 'python -m package arguments' and I'd like to write unit tests for the __main__.py module. I specifically want to verify that it sets the exit code correctly. Is it possible to use runpy.run_module to execute my __main__.py and test the exit code? If so, how do I retrieve the exit code?
To be more clear, my __main__.py module is very simple. It just calls a function that has been extensively unit tested. But when I originally wrote __main__.py, I forgot to pass the result of that function to exit(), so I would like unit tests where the main function is mocked to make sure the exit code is set correctly. My unit test would look something like:
#patch('my_module.__main__.my_main', return_value=2)
def test_rc2(self, _):
"""Test that rc 2 is the exit code."""
sys.argv = ['arg0', 'arg1', 'arg2', …]
runpy.run_module('my_module')
self.assertEqual(mod_rc, 2)
My question is, how would I get what I’ve written here as ‘mod_rc’?
Thanks.

Misko Hevery has said before (I believe it was in Clean Code Talks: Don't Look for Things but I may be wrong) that he doesn't know how to effectively unit test main methods, so his solution is to make them so simple that you can prove logically that they work if you assume the correctness of the (unit-tested) code that they call.
For example, if you have a discrete, tested unit for parsing command line arguments; a library that does the actual work; and a discrete, tested unit for rendering the completed work into output, then a main method that calls all three of those in sequence is assuredly going to work.
With that architecture, you can basically get by with just one big system test that is expected to produce something other than the "default" output and it'll either crash (because you wired it up improperly) or work (because it's wired up properly and all of the individual parts work).
At this point, I'm dropping all pretense of knowing what I'm talking about. There is almost assuredly a better way to do this, but frankly you could just write a shell script:
python -m package args
test $? -eq [expected exit code]
That will exit with error iff your program outputs incorrectly, which TravisCI or similar will regard as build failing.

__main__.py is still subject to normal __main__ global behavior — which is to say, you can implement your __main__.py like so
def main():
# Your stuff
if __name__ == "__main__":
main()
and then you can test your __main__ in whatever testing framework you like by using
from your_package.__main__ import main
As an aside, if you are using argparse, you will probably want:
def main(arg_strings=None):
# …
args = parser.parse_args(arg_strings)
# …
if __name__ == "__main__":
main()
and then you can override arg strings from a unit test simply with
from your_package.__main__ import main
def test_main():
assert main(["x", "y", "z"]) == …
or similar idiom in you testing framework.

With pytest, I was able to do:
import mypkgname.__main__ as rtmain
where mypkgname is what you've named your app as a package/module. Then just running pytest as normal worked. I hope this helps some other poor soul.

PyCharm doesn't appear to run all unit tests

I have a problem running unittests in pycharm. The first class 'KnownValues' runs but the other class doesn't get checked at all.
import roman
import unittest
class KnownValues(unittest.TestCase):
def test_too_large(self):
'''to_roman should fail with large input'''
self.assertRaises(roman.OutOfRangeError, roman.to_roman, 4000)
def test_too_small(self):
ls = [0,-1,-25,-60]
for x in ls:
self.assertRaises(roman.OutOfRangeError, roman.to_roman, x)
def test_non_int(self):
ls = [1.5, -6.5, 6.8,12.9, "hello wold", "nigga123"]
for x in ls:
self.assertRaises(roman.TypeError, roman.to_roman, x)
class Test2(unittest.TestCase):
def test1(self):
assert 1 == 1
if __name__ == '__main__':
unittest.main()

Start all of your test functions with test. Many people use underscores to separate words, so a lot of people end up with tests starting with test_, but test is all that is required.
When having trouble in the GUI, you can check how your tests are running from the command line.
python test.py
or
python -m test
One problem that you might run into is that you have defined your tests within classes, and when running them through the GUI, the GUI has automatically discovered them for you. Be sure to include the lines at the end of your test file directing the interpreter to use the main function built into unittest.
if __name__ == '__main__':
unittest.main()
Keep in mind, you can optionally run the tests in only one of your classes at a time:
python tests.py KnownValues
python tests.py Test2
In PyCharm, it should automatically discover all the test classes. You still have the option of running only one class at a time. Choose Run->Edit Configurations to see the options that you are currently running under. Using command line parameters you can control running fewer or more tests.
As you can see, you can choose to run a script, a class, or a method. Be sure to set the name of your run configuration such that it reflects the scope of what you are running.

Python CLI program unit testing

I am working on a python Command-Line-Interface program, and I find it boring when doing testings, for example, here is the help information of the program:
usage: pyconv [-h] [-f ENCODING] [-t ENCODING] [-o file_path] file_path
Convert text file from one encoding to another.
positional arguments:
file_path
optional arguments:
-h, --help show this help message and exit
-f ENCODING, --from ENCODING
Encoding of source file
-t ENCODING, --to ENCODING
Encoding you want
-o file_path, --output file_path
Output file path
When I made changes on the program and want to test something, I must open a terminal,
type the command(with options and arguments), type enter, and see if any error occurs
while running. If error really occurs, I must go back to the editor and check the code
from top to end, guessing where the bug positions, make small changes, write print lines,
return to the terminal, run command again...
Recursively.
So my question is, what is the best way to do testing with CLI program, can it be as easy
as unit testing with normal python scripts?

I think it's perfectly fine to test functionally on a whole-program level. It's still possible to test one aspect/option per test. This way you can be sure that the program really works as a whole. Writing unit-tests usually means that you get to execute your tests quicker and that failures are usually easier to interpret/understand. But unit-tests are typically more tied to the program structure, requiring more refactoring effort when you internally change things.
Anyway, using py.test, here is a little example for testing a latin1 to utf8 conversion for pyconv::
# content of test_pyconv.py
import pytest
# we reuse a bit of pytest's own testing machinery, this should eventually come
# from a separatedly installable pytest-cli plugin.
pytest_plugins = ["pytester"]
#pytest.fixture
def run(testdir):
def do_run(*args):
args = ["pyconv"] + list(args)
return testdir._run(*args)
return do_run
def test_pyconv_latin1_to_utf8(tmpdir, run):
input = tmpdir.join("example.txt")
content = unicode("\xc3\xa4\xc3\xb6", "latin1")
with input.open("wb") as f:
f.write(content.encode("latin1"))
output = tmpdir.join("example.txt.utf8")
result = run("-flatin1", "-tutf8", input, "-o", output)
assert result.ret == 0
with output.open("rb") as f:
newcontent = f.read()
assert content.encode("utf8") == newcontent
After installing pytest ("pip install pytest") you can run it like this::
$ py.test test_pyconv.py
=========================== test session starts ============================
platform linux2 -- Python 2.7.3 -- pytest-2.4.5dev1
collected 1 items
test_pyconv.py .
========================= 1 passed in 0.40 seconds =========================
The example reuses some internal machinery of pytest's own testing by leveraging pytest's fixture mechanism, see http://pytest.org/latest/fixture.html. If you forget about the details for a moment, you can just work from the fact that "run" and "tmpdir" are provided for helping you to prepare and run tests. If you want to play, you can try to insert a failing assert-statement or simply "assert 0" and then look at the traceback or issue "py.test --pdb" to enter a python prompt.

Start from the user interface with functional tests and work down towards unit tests. It can feel difficult, especially when you use the argparse module or the click package, which take control of the application entry point.
The cli-test-helpers Python package has examples and helper functions (context managers) for a holistic approach on writing tests for your CLI. It's a simple idea, and one that works perfectly with TDD:
Start with functional tests (to ensure your user interface definition) and
Work towards unit tests (to ensure your implementation contracts)
Functional tests
NOTE: I assume you develop code that is deployed with a setup.py file or is run as a module (-m).
Is the entrypoint script installed? (tests the configuration in your setup.py)
Can this package be run as a Python module? (i.e. without having to be installed)
Is command XYZ available? etc. Cover your entire CLI usage here!
Those tests are simplistic: They run the shell command you would enter in the terminal, e.g.
def test_entrypoint():
exit_status = os.system('foobar --help')
assert exit_status == 0
Note the trick to use a non-destructive operation (e.g. --help or --version) as we can't mock anything with this approach.
Towards unit tests
To test single aspects inside the application you will need to mimic things like command line arguments and maybe environment variables. You will also need to catch the exiting of your script to avoid the tests to fail for SystemExit exceptions.
Example with ArgvContext to mimic command line arguments:
#patch('foobar.command.baz')
def test_cli_command(mock_command):
"""Is the correct code called when invoked via the CLI?"""
with ArgvContext('foobar', 'baz'), pytest.raises(SystemExit):
foobar.cli.main()
assert mock_command.called
Note that we mock the function that we want our CLI framework (click in this example) to call, and that we catch SystemExit that the framework naturally raises. The context managers are provided by cli-test-helpers and pytest.
Unit tests
The rest is business as usual. With the above two strategies we've overcome the control a CLI framework may have taken away from us. The rest is usual unit testing. TDD-style hopefully.
Disclosure: I am the author of the cli-test-helpers Python package.

So my question is, what is the best way to do testing with CLI program, can it be as easy as unit testing with normal python scripts?
The only difference is that when you run Python module as a script, its __name__ attribute is set to '__main__'. So generally, if you intend to run your script from command line it should have following form:
import sys
# function and class definitions, etc.
# ...
def foo(arg):
pass
def main():
"""Entry point to the script"""
# Do parsing of command line arguments and other stuff here. And then
# make calls to whatever functions and classes that are defined in your
# module. For example:
foo(sys.argv[1])
if __name__ == '__main__':
main()
Now there is no difference, how you would use it: as a script or as a module. So inside your unit-testing code you can just import foo function, call it and make any assertions you want.

Maybe too little too late,
but you can always use
import os.system
result = os.system(<'Insert your command with options here'>
assert(0 == result)
In that way, you can run your program as if it was from command line, and evaluate the exit code.
(Update after I studied pytest)
You can also use capsys.
(from running pytest --fixtures)
capsys
Enable text capturing of writes to sys.stdout and sys.stderr.
The captured output is made available via ``capsys.readouterr()`` method
calls, which return a ``(out, err)`` namedtuple.
``out`` and ``err`` will be ``text`` objects.

This isn't for Python specifically, but what I do to test command-line scripts is to run them with various predetermined inputs and options and store the correct output in a file. Then, to test them when I make changes, I simply run the new script and pipe the output into diff correct_output -. If the files are the same, it outputs nothing. If they're different, it shows you where. This will only work if you are on Linux or OS X; on Windows, you will have to get MSYS.
Example:
python mycliprogram --someoption "some input" | diff correct_output -
To make it even easier, you can add all these test runs to your 'make test' Makefile target, which I assume you already have. ;)
If you are running many of these at once, you could make it a little more obvious where each one ends by adding a fail tag:
python mycliprogram --someoption "some input" | diff correct_output - || tput setaf 1 && echo "FAILED"

The short answer is yes, you can use unit tests, and should. If your code is well structured, it should be quite easy to test each component separately, and if you need to to can always mock sys.argv to simulate running it with different arguments.

pytest-console-scripts is a Pytest plugin for testing python scripts installed via console_scripts entry point of setup.py.

For Python 3.5+, you can use the simpler subprocess.run to call your CLI command from your test.
Using pytest:
import subprocess
def test_command__works_properly():
try:
result = subprocess.run(['command', '--argument', 'value'], check=True, capture_output=True, text=True)
except subprocess.CalledProcessError as error:
print(error.stdout)
print(error.stderr)
raise error
The output can be accessed via result.stdout, result.stderr, and result.returncode if needed.
The check parameter causes an exception to be raised if an error occurs. Note Python 3.7+ is required for the capture_output and text parameters, which simplify capturing and reading stdout/stderr.

Given that you are explicitly asking about testing for a command line application, I believe that you are aware of unit-testing tools in python and that you are actually looking for a tool to automate end-to-end tests of a command line tool. There are a couple of tools out there that are specifically designed for that. If you are looking for something that's pip-installable, I would recommend cram. It integrates well with the rest of the python environment (e.g. through a pytest extension) and it's quite easy to use:
Simply write the commands you want to run prepended with $ and the expected output prepended with . For example, the following would be a valid cram test:
$ echo Hello
Hello
By having four spaces in front of expected output and two in front of the test, you can actually use these tests to also write documentation. More on that on the website.

You can use standard unittest module:
# python -m unittest <test module>
or use nose as a testing framework. Just write classic unittest files in separate directory and run:
# nosetests <test modules directory>
Writing unittests is easy. Just follow online manual for unittesting

I would not test the program as a whole this is not a good test strategy and may not actually catch the actual spot of the error. The CLI interface is just front end to an API. You test the API via your unit tests and then when you make a change to a specific part you have a test case to exercise that change.
So, restructure your application so that you test the API and not the application it self. But, you can have a functional test that actually does run the full application and checks that the output is correct.
In short, yes testing the code is the same as testing any other code, but you must test the individual parts rather than their combination as a whole to ensure that your changes do not break everything.

How to use Python unit tests in self-contained program?

I want to use Python 3.3 with unit tests in small self-contained program, i.e. I don't want to split it up into a command line part and a "functional" part, which can be tested if it is started on itself on the command line.
So I have this little program:
import unittest
def stradd(a, b):
return a + b
class test_hello(unittest.TestCase):
def test_1(self):
self.assertEqual(stradd("a", "b"), "ab")
unittest.main()
print(stradd("Hello, ", "world"))
Unfortunately, the print() is never reached, since unittest.main() exits the program. And even if it would not exit, it would print all kinds of output to the screen that I don't want to see in normal operation.
Is there a way to run the tests silently, as long as there is no error? Of course, they should complain loudly if something doesn't work.
I've seen Run python unit tests as an option of the program, but that doesn't answer my question as well.

It is possible to achieve the effect you want with a plain unittest module. You just need to write your own simple test runner. Like this:
import unittest
def stradd(a, b):
return a + b
class test_hello(unittest.TestCase):
def test_1(self):
self.assertEqual(stradd("a", "b"), "ab")
def run_my_tests(test_case):
case = unittest.TestLoader().loadTestsFromTestCase(test_case)
result = unittest.TestResult()
case(result)
if result.wasSuccessful():
return True
else:
print("Some tests failed!")
for test, err in result.failures + result.errors:
print(test)
print(err)
return False
if run_my_tests(test_hello):
# All tests passed, so we can run our programm.
print(stradd("Hello, ", "world"))
run_my_tests function will return True if all tests pass successfully. But if there is a test failure, it will print all errors/failures to stdout. For example:
$ python myscript.py
Hello, world
$ # And now the test fails...
$ python myscript.py
Some tests failed!
test_1 (__main__.test_hello)
Traceback (most recent call last):
File "myscript.py", line 8, in test_1
self.assertEqual(stradd("a", "c"), "ab")
AssertionError: 'ac' != 'ab'

Just use python's nosetests or py.test. Then you can write the code exactly the way you want to - with nothing except test_ functions added to the program and run tests via
$ nosetests filename.py
or
$ py.test filename.py
Also yeah no need of classes then:
def test():
assert stradd("a", "b") == "ab"
Though it doesn't answer you "run silently" part. For me this + command line history works fine. For tiny programs ofc, basically snippets.

Other test frameworks won't help here because it's not test framework which is an issue here. Having said that py.test is the best one out there :).
The problem is unittest.main() function is designed specifically to run tests in a standard way and does not offer a way to customize this process in any way. This leaves us with two options
Use subprocess for running tests in separate process using unittest.main(), check the
output and continue with running our program if all tests passed
Leave high level unittest.main() alone and use other facilities
provided by unittest module
I'll write about both of these options as soon as I find some more free time.