What is the best way to create a command for setuptools which generates a code coverage report using coverage.py?
What you are looking for is Extending Distutils capabilities through extensions and it is covered in the docs that I have linked. The basic idea is to give the command and the entr5y point for execution and the entry point should follow some of setuptools Component based terminology. I think, you are luck here, because someone has already tried successfully ( Adding Test Code Coverage Analysis to a Python Project's setup Command ) integrating it for his project and you should be able to adopt it for your purposes.
Here is one simple solution which uses subprocess calls to the coverage executable. I assume you have a Python package called mycoolpackage which contains the code for which you want to measure test coverage, and you have a mytests package which exposes a suite function which returns the test suite.
First, create a run-tests.py file:
import os.path, sys
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))
import unittest
from mytests import suite
unittest.main(defaultTest='suite')
then in setup.py create the following command and add it to the cmdclass argument of the setup() function.
class run_coverage(Command):
description = "Generate a test coverage report."
user_options = []
def initialize_options(self): pass
def finalize_options(self): pass
def run(self):
import subprocess
subprocess.call(['coverage', 'run', '--source=mycoolpackage', 'run-tests.py'])
subprocess.call(['coverage', 'html'])
Note that coverage.py does have an API which would allow you to accomplish this without using a subprocess call.
Related
I'm doing a simple script to run and test my code. How can i import dinamically and run my test classes?
This is the solution that I found to import and dynamically run my test classes.
import glob
import os
import imp
import unittest
def execute_all_tests(tests_folder):
test_file_strings = glob.glob(os.path.join(tests_folder, 'test_*.py'))
suites = []
for test in test_file_strings:
mod_name, file_ext = os.path.splitext(os.path.split(test)[-1])
py_mod = imp.load_source(mod_name, test)
suites.append(unittest.defaultTestLoader.loadTestsFromModule(py_mod))
text_runner = unittest.TextTestRunner().run(unittest.TestSuite(suites))
Install pytest, and run your tests with a command like:
py.test src
That's it. Py.test will load all test_*.py files, find all def test_* calls inside them, and run each one for you.
The board is having trouble answering your question because it's in "why is water wet?" territory; all test rigs come with runners that automatically do what your code snip does, so you only need read the tutorial for one to get started.
And major props for writing auto tests at all; they put you above 75% of all programmers.
This solution is too simple and perform what i want.
import unittest
def execute_all_tests(tests_folder):
suites = unittest.TestLoader().discover(tests_folder)
text_runner = unittest.TextTestRunner().run(suites)
I am using Python's unittest with simple code like so:
suite = unittest.TestSuite()
suite.addTest(unittest.defaultTestLoader.loadTestsFromTestCase(module1))
suite.addTest(unittest.defaultTestLoader.loadTestsFromTestCase(module2))
I want my test suite to automatically parse all the modules and searches for all the unit test cases files that we have written?
for e.g.
there are 5 files,
1). f1.py
2). f2.py
3). f3.py
4). f4.py
5). f5.py
we dont know which of this file is the unit test case file. I want a way through which each file will be parsed and only the name of the module that has unit test cases should be returned
NOTE:- I am using python 2.6.6 so could not really make use of unittest.TestLoaded.discover()
Consider using the nose tool, it completely changes your unit-testing life. You just run it in the source folder root like:
> nosetests
then it automatically finds all the test cases.
If you want also run all the doctests, use:
> nosetests --with-doctest
In case if you only want to find a list of modules programmatically, nose provides some API (unfortunately, not as convenient as TestLoader.discover()).
UPDATE: I've just discovered (pun intended) that there is a library called unittest2 that backports all the later unittest features to the earlier versions of Python. I'll keep the code below for the archaeologists, but I think, unittest2 is a better way to go.
import nose.loader
import nose.suite
import types
def _iter_modules(tests):
'''
Recursively find all the modules containing tests.
(Some may repeat)
'''
for item in tests:
if isinstance(item, nose.suite.ContextSuite):
for t in _iter_modules(item):
yield t
elif isinstance(item.context, types.ModuleType):
yield item.context.__name__
else:
yield item.context.__module__
def find_test_modules(basedir):
'''
Get a list of all the modules that contain tests.
'''
loader = nose.loader.TestLoader()
tests = loader.loadTestsFromDir(basedir)
modules = list(set(_iter_modules(tests))) # remove duplicates
return modules
Use the discovery feature of the unittest library:
$ python -m unittest discover --start-directory my_project
I have written a package (http://github.com/anntzer/parsedcmd) that runs with both Python2 and Python3. However, I had to write separate (py.test) unit tests for Python2 and Python3 (mainly because I want to test extra features of Python3, in particular keyword-only arguments), so I have a test_py2.py and a test_py3.py in a test subpackage. Now, if I run, say py.test2 mypkg, test_py2 passes, but test_py3 fails with a SyntaxError. Likewise, for py.test3 mypkg, test_py3 passes but test_py2 fails (I could make this one work though, it's just an issue of StringIO having moved to io).
I can design the test subpackage so that import mypkg.test only imports the proper version of the tests, but apparently py.test doesn't care -- it just sees two files matching test_* and grabs all tests in both of them, ignoring what __init__.py tells him to import.
So right now I have to do both py.test2 mypkg/test/test_py2.py and py.test3 mypkg/test/test_py3.py. Is there a way to set up the whole thing so that py.test2 mypkg and py.test3 mypkg would "just work"?
Thanks.
If you can then making your modules importable on all interpreters and skipping tests as appropriate is a common solution. Otherwise you can put the following as "conftest.py" into the test directory:
import sys
py3 = sys.version_info[0] >= 3
class DummyCollector(pytest.collect.File):
def collect(self):
return []
def pytest_pycollect_makemodule(path, parent):
bn = path.basename
if "py3" in bn and not py3 or ("py2" in bn and py3):
return DummyCollector(path, parent=parent)
This gets picked up a project-specific plugin and will properly ignore a test module with a filename containing a "py2" or "py3" substring on the wrong interpreter version. Of course you can refine it to rather have an explicit list directly in the conftest.py file instead of checking the filename etc.pp.
HTH, holger
You can put your tests in different packages and run only the tests in the appropriate package. Or you can load the appropriate test module in a script:
import sys, unittest
cur_version = sys.version_info
if cur_version[0] < 3:
import myApp.test.test_py2
unittest.TestLoader().loadTestsFromModule(myApp.test.test_py2).run()
else:
import myApp.test.test_py3
unittest.TestLoader().loadTestsFromModule(myApp.test.test_py3).run()
Alternatively, use a setup.py file so you can run:
python setup.py test
and put the versioning logic in there:
versionedTestSuite = "parsedcmd.test.test_py2" # do something as above here
setup(name='parsedcmd',
...
test_suite=versionedTestSuite,
)
A problem I continue to have it "bootstrapping" my tests.
The problem that I have is exactly what this guy has.
The top solution talks about creating a "boostrap" script. I presume that I must then enumerate all of the tests to be run, or use test manifests in the __init__.py files using the __all__ keyword. However, I noticed that the most recent Python documentation on unittest does not talk about __all__ anymore.
In 2.7, we have the python command called "discovery"
python -m unittest discover
That works even nicer. Because:
1) There's no need for Nose
2) There's no need for test manifests
But it doesn't seem to have a way to "bootstrap"
Do I need to use another test runner? One that allows bootstrapping AND discovery?
Do I need py.test?
http://pytest.org/
The reason that I need bootstrapping, is the problem that this guy has. Basically, my import statements don't work right if I run the test directly. I want to execute my suite of tests from the top of my project, just like the app would when it runs normally.
After all, import statements are always relative to their physical location. (BTW, I think this is a hindrance in Python)
Definition: What is Bootstrapping?
Bootstrapping means that I want to do some setup before running any tests at all in the entire project. This is sort of like me asking for a "test setup" at the whole project level.
Update
Here is another posting about the same thing. Using this 2.7 command, we can avoid Nose. But how does one add bootstrapping?
I got it!
Using this one script that I wrote and called it "runtests.py" and placed in my project root, I was able to "bootstrap" that is to run some initialization code AND use discovery. Woot!
In my case, the "bootstrap" code is the two lines that say:
import sys
sys.path.insert(0, 'lib.zip')
Thanks!
#!/usr/bin/python
import unittest
import sys
sys.path.insert(0, 'lib.zip')
if __name__ == "__main__":
all_tests = unittest.TestLoader().discover('.')
unittest.TextTestRunner().run(all_tests)
Here's what I do, and I think it works quite well. For a file/directory structure similar to this:
main_code.py
run_tests.py
/Modules
__init__.py
some_module1.py
some_module2.py
/Tests
__init__.py
test_module1.py
test_module2.py
It's fairly easy to organize your run_tests.py file to bootstrap the tests. First every file with test (test_module1.py, etc.) should implement a function that generates a test suite. Something like:
def suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(Test_Length))
suite.addTest(unittest.makeSuite(Test_Sum))
return suite
at the end of your test code. Then, in the run_tests.py file, you aggregate these into an additional test_suite, and run that:
import unittest
import Tests.test_module1 as test_module1
import Tests.test_module2 as test_module2
module1_test_suite = test_module1.suite()
module2_test_suite = test_module2.suite()
aggregate_suite = unittest.TestSuite()
aggregate_suite.addTest(module1_test_suite)
aggregate_suite.addTest(module2_test_suite)
unittest.TextTestsRunner(verbosity = 2).run(aggregate_suite
Then to run all of these tests, from the command line, simply run
python run_tests.py
Here's what I want to do: I want to build a test suite that's organized into packages like tests.ui, tests.text, tests.fileio, etc. In each __init__.py in these packages, I want to make a test suite consisting of all the tests in all the modules in that package. Of course, getting all the tests can be done with unittest.TestLoader, but it seems that I have to add each module individually. So supposing that test.ui has editor_window_test.py and preview_window_test.py, I want the __init__.py to import these two files and get a list of the two module objects. The idea is that I want to automate making the test suites so that I can't forget to include something in the test suite.
What's the best way to do this? It seems like it would be an easy thing to do, but I'm not finding anything.
I'm using Python 2.5 btw.
Good answers here, but the best thing to do would be to use a 3rd party test discovery and runner like:
Nose (my favourite)
Trial (pretty nice, especially when testing async stuff)
py.test (less good, in my opinion)
They are all compatible with plain unittest.TestCase and you won't have to modify your tests in any way, neither would you have to use the advanced features in any of them. Just use as a suite discovery.
Is there a specific reason you want to reinvent the nasty stuff in these libs?
Solution to exactly this problem from our django project:
"""Test loader for all module tests
"""
import unittest
import re, os, imp, sys
def find_modules(package):
files = [re.sub('\.py$', '', f) for f in os.listdir(os.path.dirname(package.__file__))
if f.endswith(".py")]
return [imp.load_module(file, *imp.find_module(file, package.__path__)) for file in files]
def suite(package=None):
"""Assemble test suite for Django default test loader"""
if not package: package = myapp.tests # Default argument required for Django test runner
return unittest.TestSuite([unittest.TestLoader().loadTestsFromModule(m)
for m in find_modules(package)])
if __name__ == '__main__':
unittest.TextTestRunner().run(suite(myapp.tests))
EDIT: The benefit compared to bialix's solution is that you can place this loader anytwhere in the project tree, there's no need to modify init.py in every test directory.
You can use os.listdir to find all files in the test.* directory and then filter out .py files:
# Place this code to your __init__.py in test.* directory
import os
modules = []
for name in os.listdir(os.path.dirname(os.path.abspath(__file__))):
m, ext = os.path.splitext()
if ext == '.py':
modules.append(__import__(m))
__all__ = modules
The magic variable __file__ contains filepath of the current module. Try
print __file__
to check.