We Keep Coding

`pytest`: Downloading a test file once, and using it for multiple tests

I am using pytest to run test case for a package I am developing. The tests use a small image file that I have saved as a github asset. The code below works just fine, but I think that pytest is downloading the image each time it runs a new test and that takes unnecessary time and resources. I was trying to figure out how I can download the file once, and then share it across test cases
Here is some sample code.
# -- in conftest.py --
import sys
import pytest
import os
import shutil
import requests
#pytest.fixture(scope="function")
def small_image(tmpdir):
url = 'https://github.com/.../sample_image_small.tif'
r = requests.get(url)
with open(os.path.join(str(tmpdir), 'sample_image_small.tif'), 'wb') as f:
f.write(r.content)
return os.path.join(str(tmpdir), 'sample_image_small.tif')
Then here are some very simple test cases that should be able to share the same image.
# -- test_package.py --
import pytest
import os
#pytest.mark.usefixtures('small_image')
def test_ispath(small_image, compression):
assert os.path.exists(small_image)
def test_isfile(small_image, compression):
assert os.path.isfile(small_image)
Now I believe that pytest will try and isolate each test by itself and so that is what causes the repeated downloads of files. I tried to set the #pytest.fixture(scope="module") instead of function but that was generating strange errors:
ScopeMismatch: You tried to access the 'function' scoped fixture 'tmpdir' with a 'module' scoped request object, involved factories
Is there a better way to setup the tests so that I don't keep download the file over and over?

First, a note beforehand: a better alternative to the old tmpdir/tmpdir_factory fixtures pair is tmp_path/tmp_path_factory which deals with pathlib objects instead of the deprecated py.path, see Temporary directories and files.
Second, if you want to handle files session-scoped (or module-scoped), tmp*_factory fixtures are meant for that. Example:
#pytest.fixture(scope='session')
def small_image(tmp_path_factory):
img = tmp_path_factory.getbasetemp() / 'sample_image_small.tif'
img.write_bytes(b'spam')
return img
The sample_image_small.tif will now be written once per test run.
Of course, there's nothing wrong with using tempfile as suggested by #MrBean Bremen in his answer, this is just an alternative doing the same, but using only standard pytest fixtures.

You can use the same code, just handle the tempfile yourself instead of using the tmpdir fixture (which cannot be used in module-scoped fixtures):
import os
import tempfile
import pytest
import requests
#pytest.fixture(scope="module")
def small_image():
url = 'https://github.com/.../sample_image_small.tif'
r = requests.get(url)
f = tempfile.NamedTemporaryFile(delete=False):
f.write(f.content)
yield f.name
os.remove(f.name)
This will create the file, return the file name, and delete the file after the tests are finished.
EDIT:
The answer by #hoefling shows a more standard way to do this, I'll leave this one for reference.

check if unit-test has passed

I developed a crawler and it's unit-tests (mainly to validate XPATHs). I want to run specific unit-tests before script execution in order to be sure that HTML structure has not changed and existing XPATHs still working. I don't want the output of unit-test, just a flag: passed or failed.
for example:
tests.py:
import unittest
class CrwTst(unittest.TestCase):
def test_1(self):
[..]
crawler.py
class Crawler(object):
def action_1(self):
[..]
and I want to work like:
if CrwTst.test_1() is True:
Crawler.action_1()

You could potentially do this:
crawler.py
import unittest
from tests import CrwTst
if unittest.TextTestRunner().run(CrwTst('test_1')).wasSuccessful():
Crawler.action_1()
Note however that you may run into an issue with circular imports, because your test presumably already depends on Crawler, and what you are looking to do will make the Crawler depend on the test. This will likely manifest itself as ImportError: cannot import name CrwTst.
To resolve that, you can dynamically import the CrwTst.
crawler.py
import unittest
def function_that_runs_crawler():
from tests import CrwTst # Dynamically import to resolve circular ref
if unittest.TextTestRunner().run(CrwTst('test_1')).wasSuccessful():
Crawler.action_1()

setuptools command for coverage.py

What is the best way to create a command for setuptools which generates a code coverage report using coverage.py?

What you are looking for is Extending Distutils capabilities through extensions and it is covered in the docs that I have linked. The basic idea is to give the command and the entr5y point for execution and the entry point should follow some of setuptools Component based terminology. I think, you are luck here, because someone has already tried successfully ( Adding Test Code Coverage Analysis to a Python Project's setup Command ) integrating it for his project and you should be able to adopt it for your purposes.

Here is one simple solution which uses subprocess calls to the coverage executable. I assume you have a Python package called mycoolpackage which contains the code for which you want to measure test coverage, and you have a mytests package which exposes a suite function which returns the test suite.
First, create a run-tests.py file:
import os.path, sys
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))
import unittest
from mytests import suite
unittest.main(defaultTest='suite')
then in setup.py create the following command and add it to the cmdclass argument of the setup() function.
class run_coverage(Command):
description = "Generate a test coverage report."
user_options = []
def initialize_options(self): pass
def finalize_options(self): pass
def run(self):
import subprocess
subprocess.call(['coverage', 'run', '--source=mycoolpackage', 'run-tests.py'])
subprocess.call(['coverage', 'html'])
Note that coverage.py does have an API which would allow you to accomplish this without using a subprocess call.

Bootstrapping tests and using Python test discovery

A problem I continue to have it "bootstrapping" my tests.
The problem that I have is exactly what this guy has.
The top solution talks about creating a "boostrap" script. I presume that I must then enumerate all of the tests to be run, or use test manifests in the __init__.py files using the __all__ keyword. However, I noticed that the most recent Python documentation on unittest does not talk about __all__ anymore.
In 2.7, we have the python command called "discovery"
python -m unittest discover
That works even nicer. Because:
1) There's no need for Nose
2) There's no need for test manifests
But it doesn't seem to have a way to "bootstrap"
Do I need to use another test runner? One that allows bootstrapping AND discovery?
Do I need py.test?
http://pytest.org/
The reason that I need bootstrapping, is the problem that this guy has. Basically, my import statements don't work right if I run the test directly. I want to execute my suite of tests from the top of my project, just like the app would when it runs normally.
After all, import statements are always relative to their physical location. (BTW, I think this is a hindrance in Python)
Definition: What is Bootstrapping?
Bootstrapping means that I want to do some setup before running any tests at all in the entire project. This is sort of like me asking for a "test setup" at the whole project level.
Update
Here is another posting about the same thing. Using this 2.7 command, we can avoid Nose. But how does one add bootstrapping?

I got it!
Using this one script that I wrote and called it "runtests.py" and placed in my project root, I was able to "bootstrap" that is to run some initialization code AND use discovery. Woot!
In my case, the "bootstrap" code is the two lines that say:
import sys
sys.path.insert(0, 'lib.zip')
Thanks!
#!/usr/bin/python
import unittest
import sys
sys.path.insert(0, 'lib.zip')
if __name__ == "__main__":
all_tests = unittest.TestLoader().discover('.')
unittest.TextTestRunner().run(all_tests)

Here's what I do, and I think it works quite well. For a file/directory structure similar to this:
main_code.py
run_tests.py
/Modules
__init__.py
some_module1.py
some_module2.py
/Tests
__init__.py
test_module1.py
test_module2.py
It's fairly easy to organize your run_tests.py file to bootstrap the tests. First every file with test (test_module1.py, etc.) should implement a function that generates a test suite. Something like:
def suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(Test_Length))
suite.addTest(unittest.makeSuite(Test_Sum))
return suite
at the end of your test code. Then, in the run_tests.py file, you aggregate these into an additional test_suite, and run that:
import unittest
import Tests.test_module1 as test_module1
import Tests.test_module2 as test_module2
module1_test_suite = test_module1.suite()
module2_test_suite = test_module2.suite()
aggregate_suite = unittest.TestSuite()
aggregate_suite.addTest(module1_test_suite)
aggregate_suite.addTest(module2_test_suite)
unittest.TextTestsRunner(verbosity = 2).run(aggregate_suite
Then to run all of these tests, from the command line, simply run
python run_tests.py

Using doctests from within unittests

I typically write both unittests and doctests in my modules. I'd like to automatically run all of my doctests when running the test suite. I think this is possible, but I'm having a hard time with the syntax.
I have the test suite
import unittest
class ts(unittest.TestCase):
def test_null(self): self.assertTrue(True)
if __name__ == '__main__': unittest.main()
I'd like to add to this suite all of the doctests in module module1. How can I do this? I've read the python docs, but I'm not any closer to success, here. Adding the lines
import doctest
import module1
suite = doctest.DocTestSuite(module1)
doesn't work. unittest.main() searches through the current file scope and runs every test case it finds, right? But DocTestSuite produces a test suite. How do I get unittest.main() to run the additional cases in the suite? Or am I just confused and deluded??
Once again, I'd be grateful for any help anyone can offer.

An update to this old question: since Python version 2.7 there is the load_tests protocol and there is no longer a need to write custom code. It allows you to add a function load_tests(), which a test loader will execute to update its collection of unit tests for the current module.
Put a function like this in your code module to package the module's own doctests into a test suite for unittest:
def load_tests(loader, tests, ignore):
tests.addTests(doctest.DocTestSuite())
return tests
Or, put a function like this into your unit test module to add the doctests from another module (for example, package.code_module) into the tests suite which is already there:
def load_tests(loader, tests, ignore):
tests.addTests(doctest.DocTestSuite(package.code_module))
return tests
When unittest.TestLoader methods loadTestsFromModule(), loadTestsFromName() or discover() are used unittest uses a test suite including both unit tests and doctests.

In this code i combined unittests and doctests from imported module
import unittest
class ts(unittest.TestCase):
def test_null(self):
self.assertTrue(True)
class ts1(unittest.TestCase):
def test_null(self):
self.assertTrue(True)
testSuite = unittest.TestSuite()
testSuite.addTests(unittest.makeSuite(ts))
testSuite.addTest(unittest.makeSuite(ts1))
import doctest
import my_module_with_doctests
testSuite.addTest(doctest.DocTestSuite(my_module_with_doctests))
unittest.TextTestRunner(verbosity = 2).run(testSuite)

I would recommend to use pytest --doctest-modules without any load_test protocol. You can simply add both the files or directories with your normal pytests and your modules with doctests to that pytest call.
pytest --doctest-modules path/to/pytest/unittests path/to/modules
It discovers and runs all doctests as well.
See https://docs.pytest.org/en/latest/doctest.html

This code will automatically run the doctests for all the modules in a package without needing to manually add a test suite for each module. This can be used with Tox.
import doctest
import glob
import os
import sys
if sys.version_info < (2,7,):
import unittest2 as unittest
else:
import unittest
import mypackage as source_package
def load_module_by_path(path):
"""Load a python module from its path.
Parameters
----------
path : str
Path to the module source file.
Returns
-------
mod : module
Loaded module.
"""
import imp
module_file_basename = os.path.basename(path)
module_name, ext = os.path.splitext(module_file_basename)
mod = imp.load_source(module_name, path)
return mod
def file_contains_doctests(path):
"""Scan a python source file to determine if it contains any doctest examples.
Parameters
----------
path : str
Path to the module source file.
Returns
-------
flag : bool
True if the module source code contains doctest examples.
"""
with open(path) as f:
for line in f:
if ">>>" in line:
return True
return False
def load_tests(loader, tests, pattern):
"""Run doctests for all modules"""
source_dir = os.path.dirname(source_package.__path__[0])
python_source_glob = os.path.join(source_dir, source_package.__name__, "*.py")
python_source_files = glob.glob(python_source_glob)
for python_source_file in python_source_files:
if not file_contains_doctests(python_source_file):
continue
module = load_module_by_path(python_source_file)
tests.addTests(doctest.DocTestSuite(module))
return tests

First I tried accepted answer from Andrey, but at least when running in Python 3.10 and python -m unittest discover it has led to running the test from unittest twice. Then I tried to simplify it and use load_tests and to my surprise it worked very well:
So just write both load_tests and normal unittest tests in a single file and it works!
import doctest
import unittest
import my_module_with_doctests
class ts(unittest.TestCase):
def test_null(self):
self.assertTrue(False)
# No need in any other extra code here
# Load doctests as unittest, see https://docs.python.org/3/library/doctest.html#unittest-api
def load_tests(loader, tests, ignore):
tests.addTests(doctest.DocTestSuite(my_module_with_doctests))
return tests

The zope.testing module provide such a functionality.
See
http://www.veit-schiele.de/dienstleistungen/schulungen/testen/doctests
for examples.