We Keep Coding

NGINX/Apache and disabling assertions in Python [duplicate]

I'm running a python script from inside a different software (it provides a python interface to manipulate its data structures).
I'm optimizing my code for speed and would like to see what impact on performance my asserts have.
I'm unable to use python -O. What other options do I have, to programatically disable all asserts in python code? The variable __debug__ (which is cleared by -O flag) cannot be assigned to :(

The docs say,
The value for the built-in variable [__debug__] is determined when the
interpreter starts.
So, if you can not control how the python interpreter is started, then it looks like you can not disable assert.
Here then are some other options:
The safest way is to manually remove all the assert statements.
If all your assert statements occur on lines by themselves, then
perhaps you could remove them with
sed -i 's/assert /pass #assert /g' script.py
Note that this will mangle your code if other code comes after the assert. For example, the sed command above would comment-out the return in a line like this:
assert x; return True
which would change the logic of your program.
If you have code like this, it would probably be best to manually remove the asserts.
There might be a way to remove them programmatically by parsing your
script with the tokenize module, but writing such a program to
remove asserts may take more time than it would take to manually
remove the asserts, especially if this is a one-time job.
If the other piece of software accepts .pyc files, then there is a
dirty trick which seems to work on my machine, though note a Python
core developer warns against this (See Éric Araujo's comment on 2011-09-17). Suppose your script is called script.py.
Make a temporary script called, say, temp.py:
import script
Run python -O temp.py. This creates script.pyo.
Move script.py and script.pyc (if it exists) out of your PYTHONPATH
or whatever directory the other software is reading to find your
script.
Rename script.pyo --> script.pyc.
Now when the other software tries to import your script, it will
only find the pyc file, which has the asserts removed.
For example, if script.py looks like this:
assert False
print('Got here')
then running python temp.py will now print Got here instead of raising an AssertionError.

You may be able to do this with an environment variable, as described in this other answer. Setting PYTHONOPTIMIZE=1 is equivalent to starting Python with the -O option. As an example, this works in Blender 2.78, which embeds Python 3.5:
blender --python-expr 'assert False; print("foo")'
PYTHONOPTIMIZE=1 blender --python-expr 'assert False; print("foo")'
The first command prints a traceback, while the second just prints "foo".

As #unutbu describes, there is no official way of doing this. However, a simple strategy is to define a flag like _test somewhere (for example, as keyword argument to a function, or as a global variable in a module), then include this in your assert statements as follows:
def f(x, _test=True):
assert not _test or x > 0
...
Then you can disable asserts in that function if needed.
f(x, _test=False)

Python CLI program unit testing

I am working on a python Command-Line-Interface program, and I find it boring when doing testings, for example, here is the help information of the program:
usage: pyconv [-h] [-f ENCODING] [-t ENCODING] [-o file_path] file_path
Convert text file from one encoding to another.
positional arguments:
file_path
optional arguments:
-h, --help show this help message and exit
-f ENCODING, --from ENCODING
Encoding of source file
-t ENCODING, --to ENCODING
Encoding you want
-o file_path, --output file_path
Output file path
When I made changes on the program and want to test something, I must open a terminal,
type the command(with options and arguments), type enter, and see if any error occurs
while running. If error really occurs, I must go back to the editor and check the code
from top to end, guessing where the bug positions, make small changes, write print lines,
return to the terminal, run command again...
Recursively.
So my question is, what is the best way to do testing with CLI program, can it be as easy
as unit testing with normal python scripts?

I think it's perfectly fine to test functionally on a whole-program level. It's still possible to test one aspect/option per test. This way you can be sure that the program really works as a whole. Writing unit-tests usually means that you get to execute your tests quicker and that failures are usually easier to interpret/understand. But unit-tests are typically more tied to the program structure, requiring more refactoring effort when you internally change things.
Anyway, using py.test, here is a little example for testing a latin1 to utf8 conversion for pyconv::
# content of test_pyconv.py
import pytest
# we reuse a bit of pytest's own testing machinery, this should eventually come
# from a separatedly installable pytest-cli plugin.
pytest_plugins = ["pytester"]
#pytest.fixture
def run(testdir):
def do_run(*args):
args = ["pyconv"] + list(args)
return testdir._run(*args)
return do_run
def test_pyconv_latin1_to_utf8(tmpdir, run):
input = tmpdir.join("example.txt")
content = unicode("\xc3\xa4\xc3\xb6", "latin1")
with input.open("wb") as f:
f.write(content.encode("latin1"))
output = tmpdir.join("example.txt.utf8")
result = run("-flatin1", "-tutf8", input, "-o", output)
assert result.ret == 0
with output.open("rb") as f:
newcontent = f.read()
assert content.encode("utf8") == newcontent
After installing pytest ("pip install pytest") you can run it like this::
$ py.test test_pyconv.py
=========================== test session starts ============================
platform linux2 -- Python 2.7.3 -- pytest-2.4.5dev1
collected 1 items
test_pyconv.py .
========================= 1 passed in 0.40 seconds =========================
The example reuses some internal machinery of pytest's own testing by leveraging pytest's fixture mechanism, see http://pytest.org/latest/fixture.html. If you forget about the details for a moment, you can just work from the fact that "run" and "tmpdir" are provided for helping you to prepare and run tests. If you want to play, you can try to insert a failing assert-statement or simply "assert 0" and then look at the traceback or issue "py.test --pdb" to enter a python prompt.

Start from the user interface with functional tests and work down towards unit tests. It can feel difficult, especially when you use the argparse module or the click package, which take control of the application entry point.
The cli-test-helpers Python package has examples and helper functions (context managers) for a holistic approach on writing tests for your CLI. It's a simple idea, and one that works perfectly with TDD:
Start with functional tests (to ensure your user interface definition) and
Work towards unit tests (to ensure your implementation contracts)
Functional tests
NOTE: I assume you develop code that is deployed with a setup.py file or is run as a module (-m).
Is the entrypoint script installed? (tests the configuration in your setup.py)
Can this package be run as a Python module? (i.e. without having to be installed)
Is command XYZ available? etc. Cover your entire CLI usage here!
Those tests are simplistic: They run the shell command you would enter in the terminal, e.g.
def test_entrypoint():
exit_status = os.system('foobar --help')
assert exit_status == 0
Note the trick to use a non-destructive operation (e.g. --help or --version) as we can't mock anything with this approach.
Towards unit tests
To test single aspects inside the application you will need to mimic things like command line arguments and maybe environment variables. You will also need to catch the exiting of your script to avoid the tests to fail for SystemExit exceptions.
Example with ArgvContext to mimic command line arguments:
#patch('foobar.command.baz')
def test_cli_command(mock_command):
"""Is the correct code called when invoked via the CLI?"""
with ArgvContext('foobar', 'baz'), pytest.raises(SystemExit):
foobar.cli.main()
assert mock_command.called
Note that we mock the function that we want our CLI framework (click in this example) to call, and that we catch SystemExit that the framework naturally raises. The context managers are provided by cli-test-helpers and pytest.
Unit tests
The rest is business as usual. With the above two strategies we've overcome the control a CLI framework may have taken away from us. The rest is usual unit testing. TDD-style hopefully.
Disclosure: I am the author of the cli-test-helpers Python package.

So my question is, what is the best way to do testing with CLI program, can it be as easy as unit testing with normal python scripts?
The only difference is that when you run Python module as a script, its __name__ attribute is set to '__main__'. So generally, if you intend to run your script from command line it should have following form:
import sys
# function and class definitions, etc.
# ...
def foo(arg):
pass
def main():
"""Entry point to the script"""
# Do parsing of command line arguments and other stuff here. And then
# make calls to whatever functions and classes that are defined in your
# module. For example:
foo(sys.argv[1])
if __name__ == '__main__':
main()
Now there is no difference, how you would use it: as a script or as a module. So inside your unit-testing code you can just import foo function, call it and make any assertions you want.

Maybe too little too late,
but you can always use
import os.system
result = os.system(<'Insert your command with options here'>
assert(0 == result)
In that way, you can run your program as if it was from command line, and evaluate the exit code.
(Update after I studied pytest)
You can also use capsys.
(from running pytest --fixtures)
capsys
Enable text capturing of writes to sys.stdout and sys.stderr.
The captured output is made available via ``capsys.readouterr()`` method
calls, which return a ``(out, err)`` namedtuple.
``out`` and ``err`` will be ``text`` objects.

This isn't for Python specifically, but what I do to test command-line scripts is to run them with various predetermined inputs and options and store the correct output in a file. Then, to test them when I make changes, I simply run the new script and pipe the output into diff correct_output -. If the files are the same, it outputs nothing. If they're different, it shows you where. This will only work if you are on Linux or OS X; on Windows, you will have to get MSYS.
Example:
python mycliprogram --someoption "some input" | diff correct_output -
To make it even easier, you can add all these test runs to your 'make test' Makefile target, which I assume you already have. ;)
If you are running many of these at once, you could make it a little more obvious where each one ends by adding a fail tag:
python mycliprogram --someoption "some input" | diff correct_output - || tput setaf 1 && echo "FAILED"

The short answer is yes, you can use unit tests, and should. If your code is well structured, it should be quite easy to test each component separately, and if you need to to can always mock sys.argv to simulate running it with different arguments.

pytest-console-scripts is a Pytest plugin for testing python scripts installed via console_scripts entry point of setup.py.

For Python 3.5+, you can use the simpler subprocess.run to call your CLI command from your test.
Using pytest:
import subprocess
def test_command__works_properly():
try:
result = subprocess.run(['command', '--argument', 'value'], check=True, capture_output=True, text=True)
except subprocess.CalledProcessError as error:
print(error.stdout)
print(error.stderr)
raise error
The output can be accessed via result.stdout, result.stderr, and result.returncode if needed.
The check parameter causes an exception to be raised if an error occurs. Note Python 3.7+ is required for the capture_output and text parameters, which simplify capturing and reading stdout/stderr.

Given that you are explicitly asking about testing for a command line application, I believe that you are aware of unit-testing tools in python and that you are actually looking for a tool to automate end-to-end tests of a command line tool. There are a couple of tools out there that are specifically designed for that. If you are looking for something that's pip-installable, I would recommend cram. It integrates well with the rest of the python environment (e.g. through a pytest extension) and it's quite easy to use:
Simply write the commands you want to run prepended with $ and the expected output prepended with . For example, the following would be a valid cram test:
$ echo Hello
Hello
By having four spaces in front of expected output and two in front of the test, you can actually use these tests to also write documentation. More on that on the website.

You can use standard unittest module:
# python -m unittest <test module>
or use nose as a testing framework. Just write classic unittest files in separate directory and run:
# nosetests <test modules directory>
Writing unittests is easy. Just follow online manual for unittesting

I would not test the program as a whole this is not a good test strategy and may not actually catch the actual spot of the error. The CLI interface is just front end to an API. You test the API via your unit tests and then when you make a change to a specific part you have a test case to exercise that change.
So, restructure your application so that you test the API and not the application it self. But, you can have a functional test that actually does run the full application and checks that the output is correct.
In short, yes testing the code is the same as testing any other code, but you must test the individual parts rather than their combination as a whole to ensure that your changes do not break everything.

Disabling python's assert() without -0 flag

I'm running a python script from inside a different software (it provides a python interface to manipulate its data structures).
I'm optimizing my code for speed and would like to see what impact on performance my asserts have.
I'm unable to use python -O. What other options do I have, to programatically disable all asserts in python code? The variable __debug__ (which is cleared by -O flag) cannot be assigned to :(

The docs say,
The value for the built-in variable [__debug__] is determined when the
interpreter starts.
So, if you can not control how the python interpreter is started, then it looks like you can not disable assert.
Here then are some other options:
The safest way is to manually remove all the assert statements.
If all your assert statements occur on lines by themselves, then
perhaps you could remove them with
sed -i 's/assert /pass #assert /g' script.py
Note that this will mangle your code if other code comes after the assert. For example, the sed command above would comment-out the return in a line like this:
assert x; return True
which would change the logic of your program.
If you have code like this, it would probably be best to manually remove the asserts.
There might be a way to remove them programmatically by parsing your
script with the tokenize module, but writing such a program to
remove asserts may take more time than it would take to manually
remove the asserts, especially if this is a one-time job.
If the other piece of software accepts .pyc files, then there is a
dirty trick which seems to work on my machine, though note a Python
core developer warns against this (See Éric Araujo's comment on 2011-09-17). Suppose your script is called script.py.
Make a temporary script called, say, temp.py:
import script
Run python -O temp.py. This creates script.pyo.
Move script.py and script.pyc (if it exists) out of your PYTHONPATH
or whatever directory the other software is reading to find your
script.
Rename script.pyo --> script.pyc.
Now when the other software tries to import your script, it will
only find the pyc file, which has the asserts removed.
For example, if script.py looks like this:
assert False
print('Got here')
then running python temp.py will now print Got here instead of raising an AssertionError.

You may be able to do this with an environment variable, as described in this other answer. Setting PYTHONOPTIMIZE=1 is equivalent to starting Python with the -O option. As an example, this works in Blender 2.78, which embeds Python 3.5:
blender --python-expr 'assert False; print("foo")'
PYTHONOPTIMIZE=1 blender --python-expr 'assert False; print("foo")'
The first command prints a traceback, while the second just prints "foo".

As #unutbu describes, there is no official way of doing this. However, a simple strategy is to define a flag like _test somewhere (for example, as keyword argument to a function, or as a global variable in a module), then include this in your assert statements as follows:
def f(x, _test=True):
assert not _test or x > 0
...
Then you can disable asserts in that function if needed.
f(x, _test=False)

Running doctests through iPython and pseudo-consoles

I've got a fairly basic doctestable file:
class Foo():
"""
>>> 3+2
5
"""
if __name__ in ("__main__", "__console__"):
import doctest
doctest.testmod(verbose=True)
which works as expected when run directly through python.
However, in iPython, I get
1 items had no tests:
__main__
0 tests in 1 items.
0 passed and 0 failed.
Test passed.
Since this is part of a Django project and will need access to all of the appropriate variables and such that manage.py sets up, I can also run it through a modified command, which uses code.InteractiveConsole, one result of which is __name__ gets set to '__console__'.
With the code above, I get the same result as with iPython. I tried changing the last line to this:
this = __import__(__name__)
doctest.testmod(this, verbose=True)
and I get an ImportError on __console__, which makes sense, I guess. This has no effect on either python or ipython.
So, I'd like to be able to run doctests successfully through all three of these methods, especially the InteractiveConsole one, since I expect to be needing Django pony magic fairly soon.
Just for clarification, this is what I'm expecting:
Trying:
3+2
Expecting:
5
ok
1 items had no tests:
__main__
1 items passed all tests:
1 tests in __main__.Foo
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

The root problem is that ipython plays weird tricks with __main__ (through its own FakeModule module) so that, by the time doctest is introspecting that "alleged module" through its __dict__, Foo is NOT there -- so doctest doesn't recurse into it.
Here's one solution:
class Foo():
"""
>>> 3+2
5
"""
if __name__ in ("__main__", "__console__"):
import doctest, inspect, sys
m = sys.modules['__main__']
m.__test__ = dict((n,v) for (n,v) in globals().items()
if inspect.isclass(v))
doctest.testmod(verbose=True)
This DOES produce, as requested:
$ ipython dot.py
Trying:
3+2
Expecting:
5
ok
1 items had no tests:
__main__
1 items passed all tests:
1 tests in __main__.__test__.Foo
1 tests in 2 items.
1 passed and 0 failed.
Test passed.
Python 2.5.1 (r251:54863, Feb 6 2009, 19:02:12)
[[ snip snip ]]
In [1]:
Just setting global __test__ doesn't work, again because setting it as a global of what you're thinking of as __main__ does NOT actually place it in the __dict__ of the actual object that gets recovered by m = sys.modules['__main__'], and the latter is exactly the expression doctest is using internally (actually it uses sys.modules.get, but the extra precaution is not necessary here since we do know that __main__ exists in sys.modules... it's just NOT the object you expect it to be!-).
Also, just setting m.__test__ = globals() directly does not work either, for a different reason: doctest checks that the values in __test__ are strings, functions, classes, or modules, and without some selection you cannot guarantee that globals() will satisfy that condition (in fact it won't). Here I'm selecting just classes, if you also want functions or whatnot you can use an or in the if clause in the genexp within the dict call.
I don't know exactly how you're running a Django shell that's able to execute your script (as I believe python manage.py shell doesn't accept arguments, you must be doing something else, and I can't guess exactly what!-), but a similar approach should help (whether your Django shell is using ipython, the default when available, or plain Python): appropriately setting __test__ in the object you obtain as sys.modules['__main__'] (or __console__, if that's what you're then passing on to doctest.testmod, I guess) should work, as it mimics what doctest will then be doing internally to locate your test strings.
And, to conclude, a philosophical reflection on design, architecture, simplicity, transparency, and "black magic"...:
All of this effort is basically what's needed to defeat the "black magic" that ipython (and maybe Django, though it may be simply delegating that part to ipython) is doing on your behalf for your "convenience"... any time at which two frameworks (or more;-) are independently doing each its own brand of black magic, interoperability may suddenly require substantial effort and become anything BUT convenient;-).
I'm not saying that the same convenience could have been provided (by any one or more of ipython, django and/or doctests) without black magic, introspection, fake modules, and so on; the designers and maintainers of each of those frameworks are superb engineers, and I expect they've done their homework thoroughly, and are performing only the minimum amount of black magic that's indispensable to deliver the amount of user convenience they decided they needed. Nevertheless, even in such a situation, "black magic" suddenly turns from a dream of convenience to a nightmare of debugging as soon as you want to do something even marginally outside what the framework's author had conceived.
OK, maybe in this case not quite a nightmare, but I do notice that this question has been open a while and even with the lure of the bounty it didn't get many answers yet -- though you now do have two answers to pick from, mine using the __test__ special feature of doctest, #codeape's using the peculiar __IP.magic_run feature of ironpython. I prefer mine because it does not rely on anything internal or undocumented -- __test__ IS a documented feature of doctest, while __IP, with those two looming leading underscores, scream "deep internals, don't touch" to me;-)... if it breaks at the next point release I wouldn't be at all surprised. Still, matter of taste -- that answer may arguably be considered more "convenient".
But, this is exactly my point: convenience may come at an enormous price in terms of giving up simplicity, transparency, and/or avoidance of internal/undocumented/unstable features; so, as a lesson for all of us, the least black magic &c we can get away with (even at the price of giving up an epsilon of convenience here and there), the happier we'll all be in the long run (and the happier we'll make other developers that need to leverage our current efforts in the future).

The following works:
$ ipython
...
In [1]: %run file.py
Trying:
3+2
Expecting:
5
ok
1 items had no tests:
__main__
1 items passed all tests:
1 tests in __main__.Foo
1 tests in 2 items.
1 passed and 0 failed.
Test passed.
In [2]:
I have no idea why ipython file.py does not work. But the above is at least a workaround.
EDIT:
I found the reason why it does not work. It is quite simple:
If you do not specify the module to test in doctest.testmod(), it assumes that you want to test the __main__ module.
When IPython executes the file passed to it on the command line, the __main__ module is IPython's __main__, not your module. So doctest tries to execute doctests in IPython's entry script.
The following works, but feels a bit weird:
if __name__ == '__main__':
import doctest
import the_current_module
doctest.testmod(the_current_module)
So basically the module imports itself (that's the "feels a bit weird" part). But it works. Something I do not like abt. this approach is that every module needs to include its own name in the source.
EDIT 2:
The following script, ipython_doctest, makes ipython behave the way you want:
#! /usr/bin/env bash
echo "__IP.magic_run(\"$1\")" > __ipython_run.py
ipython __ipython_run.py
The script creates a python script that will execute %run argname in IPython.
Example:
$ ./ipython_doctest file.py
Trying:
3+2
Expecting:
5
ok
1 items had no tests:
__main__
1 items passed all tests:
1 tests in __main__.Foo
1 tests in 2 items.
1 passed and 0 failed.
Test passed.
Python 2.5 (r25:51908, Mar 7 2008, 03:27:42)
Type "copyright", "credits" or "license" for more information.
IPython 0.9.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
In [1]:

Using doctest "result parser" within unit-tests in Python?

I recently faced a problem about combining unit tests and doctests in Python. I worked around this problem in other way, but I still have question about it.
Python's doctest module parses docstrings in a module and run commands following ">>> " at the beginning of each line and compare the output of it and those in docstrings.
I wonder that I could use that comparison method implemented by doctest module when I want. I know that it's possible add doctest to test suite as a test case, but here I want to do it inside a single test case.
It is something like this:
class MyTest(TestCase):
def testIt(self):
# some codes like self.assertEqual(...)
output = StringIO()
with StdoutCollector(output):
# do something that uses stdout
# I want something like this:
doctest.compare_result(output.getvalue(), 'expected output')
# do more things
Because doctest uses some heuristics to compare the outputs like ellipsis.
Would somebody give an idea or suggestions?

See doctest.OutputChecker.check_output()