I'm trying to write a testing program to test many(identical) student assignments. I have a test written using the unittest library. The documentation seems to indicate that each test should be associated with one file. Instead, I'd like to have one test file and use command line arguments to point the test to the file it should test.
I know I can do this by using the argparse module in my unit tests, but is there a better way? It seems like this behavior should be supported in unittest, but I can't find anything in the documentation...
Create the Main test directory and add sub test packages. Have a test runner created for you pointing to the test directory. It could act as a suite. I have attached a piece of code that I have used for my test suite.
import os
import unittest
def main(test_path, test_pattern):
print(('Discovering tests in : {}'.format(test_path)))
suite = unittest.TestLoader().discover(test_path, test_pattern)
unittest.TextTestRunner(verbosity=2).run(suite)
if __name__ == '__main__':
root_path = os.path.abspath('.')
test_path = os.path.join(root_path, 'src/tests/')
test_pattern = 'test_*'
main(test_path, test_pattern)
Generally speaking, unittest is used to test module level python code, not interactions python code has with external programs. AFAIK, writing to stdout (ie. print) means you are either debugging or passing information to another program.
In your case, I don't think unittest is really necessary, unless you want to give assignments that are to "pass this unittest" (which is common in the wild).
Instead I would simply iterate over the directory that contains the assignments, check the stdout using subprocess, then write the results to a csv file:
import subprocess
import os
import csv
ASSIGNMENT_DIR = '/path/to/assignments'
expected_stdout = 'Hello World!'
def _determine_grade(stdout):
if stdout == expected_stdout:
return '100%'
return '0%'
grades = {}
for assignment in os.listdir(ASSIGNMENT_DIR):
filepath = os.path.join(ASSIGNMENT_DIR, assignment)
stdout = subprocesss.check_output(f'python3 {filepath}', shell=True)
grade = _determine_grade(stdout)
grades.append({'assignment':assignment, 'grade':grade})
with open('/path/to/grades.csv', 'w+') as f:
w = csv.DictWriter(f, ('assignment', 'grade'))
w.writeheader()
w.writerows(grades)
Related
TI am quite new to Python Programming and have a question on testing using Pytest. In a high-level, I have a program that takes 3 pieces of user input and generates a text file in the end. For my tests, I want to basically compare the files my program outputted, with what it should be.
Now, I am not sure how to go about testing. The program itself takes no arguments, but just relies on 3 pieces of user input, which I'll use monkeypatch to simulate. Do I create a new python file called program_test.py and have methods in here that call the original program? I have tried this, but I'm having trouble actually calling the original program and sending in the simulated inputs. Or, do I have tests in the original program (which doesn't make much sense to me).
I want something like this:
import my_program
def test_1():
inputs = iter(['input1', 'input2', 'input3'])
monkeypatch.setattr('builtins.input', lambda x: next(inputs))
my_program
# now do some assertion with some file comparison
# pseudocode
assert filecompare.cmp(expectedfile, actualfile)
This just seems to be running the original program and I think its to do with the import statement i.e. it is never running test_1(), probably because I never call it? Any help would be appreciated!
Without providing your my_program code it's hard to tell what's going on.
Since you are mentioning import problems, I guess your not defining main() and if __name__ == "__main__".
Here's a little example of how you can test that.
First, structure your my_program to have main function which contains the code and then add if __name__ == "__main__" which will allow you to run main function if the my_program is executed directly but also to import my_program as module to other files (without running it, for more information please see: What does if name == "main": do?).
my_program:
def main():
x = input()
y = input()
z = input()
with open("test", "w") as f_out:
f_out.write(f"{x}-{y}-{z}")
if __name__ == "__main__":
main()
Now you can create a test.py file and test the main function of my_program:
import os
import filecmp
import my_program
def test_success(monkeypatch):
inputs = ["input1", "input2", "input3"]
monkeypatch.setattr("builtins.input", lambda: next(iter(inputs)))
my_program.main()
with open("expected", "w") as f_out:
f_out.write("-".join(inputs))
assert filecmp.cmp("expected", "test")
os.remove("test")
os.remove("expected")
def test_fail(monkeypatch):
inputs = ["input1", "input2", "input3"]
monkeypatch.setattr("builtins.input", lambda: next(iter(inputs)))
my_program.main()
with open("expected", "w") as f_out:
f_out.write("something-else-test")
assert not filecmp.cmp("expected", "test")
os.remove("test")
os.remove("expected")
This is an example so I used os.remove to delete the files. Ideally you would define fixtures in your tests to use tempfile and generate random temporary files which will be automatically deleted after your tests.
I have two tests in which I'm trying to import things from the same myconfig.py file. But I need some variables in myconfig.py to be changed after every test so the next can use a new one.
Is there any possible way to do it?
I already have function that re-writes info and it works correctly, but both tests import the very first version of myconfig.py.
I tried to use fixtures, but it didn't work. I also tried to run/exec myconfig.py before every launch, didn't help either.
I use two same small tests, run them as a package called test_package.
from myconfig import url, file_name, line_index
from test_package.functions_filee import upgrade_number
from test_package.pages.my_page import MyPageObject
from file_where_info_should be_rewritten import phone_number
def test_rewrite_number(self, browser):
my_page = MyPageObject(browser, url)
my_page.go_to_site()
my_page.input_phone_number(phone_number)
upgrade_number(file_name, line_index, phone_number)
upgrade_number function code:
def upgrade_number(file_name, line_index, phone_number):
with open(file_name, 'r+') as f:
lines = f.readlines()
new_number = str(int(phone_number) + 1)
new_str = f"phone_number = '{new_number}'\n"
lines[line_index] = new_str
f.seek(0)
f.writelines(lines)
File, where info should be rewritten consists of only a single line:
phone_number = "1111111111"
So basically I want the first test to import 1111111111 as a phone number, then increase it to 1111111112 so that the second test can use 1111111112 as its test data.
EDIT: I did it like this
import sys
import importlib
importlib.reload(sys.modules['dynamic_vars'])
from dynamic_vars import *
Eventually I gave up on this method of making new phone number for every test, but maybe it will be helpful for anyone
The classical pattern for this is called Test setup. In pytest you have setup_method which you can implement to reload your config and it will be done before each test is executed.
Similarly, you can use teardown_method to run code after each test. Your test can also set a class variable so each invocation of setup_method or teardown_method knows which value to use.
BUT
You should not do what you explain here. This makes tests dependent on each other which is a very problematic antipattern. Instead, make each test setup what it needs explicitly at the beginning. So, if a second test needs 1111111112 as its input data, specify this at the beginning of the test - by calling an utility function that will prepare the config appropriately.
This not only makes the tests independent - it also makes them more readable, as it's clear at the beginning of the test what the scenario is.
A global variable can be easily mocked following these answers. Great. However, this does not work when trying to mock a variable in a script that you call with subprocess.call() in a test with Pytest.
Here is my simplified script in a file called so_script.py:
import argparse
INCREMENTOR = 4
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('input_nr', type=int, help='An int to increment')
args = parser.parse_args()
with open('test.txt', 'w+') as f:
f.write(str(args.input_nr + INCREMENTOR))
Now, say I want to mock the value of INCREMENTOR in my tests to be 1. If I do this:
from subprocess import call
from unittest import mock
def test_increments_with_1():
with mock.patch('so_script.INCREMENTOR', 1):
call(['python', 'so_script.py', '3'])
with open('test.txt', 'r+') as f:
assert f.read() == '4'
The test will fail, because the value of INCREMENTOR remains 4, even though I tried to patch it to 1. So what gets written to the file is 7 instead of 4.
So my question is: how do I mock the INCREMENTOR global variable in my so_script.py file so that, when calling subprocess.call() on it, it remains mocked?
Because the so_script.py script and pytest are executed in different processes, one cannot mock objects in so_script.py while the latter is being called as a different process in tests.
The best solution I found was to put everything from the if __name__ == '__main__: block in a function and test that function with Pytest, mocking whatever I needed to mock. And, to have 100% test coverage (which was my initial intent with calling the script as a subprocess), I applied this solution.
So I dropped using subprocess.call() in my tests and wrote an init() function checking if __name__ == '__main__:, and then mocked __name__ in the tests to test the function, just as the article advises to do. This got me 100% test coverage and full mocking capabilities.
I am writing a test_examples.py to test the execution of a folder of python examples. Currently I use glob to parse the folder and then use subprocess to execute each python file. The issue is that some of these files are plots and they open a Figure window that halts until the window is closed.
A lot of the questions on this issue offer solutions from within the file, but how could I suppress the output whilst running the file externally without any modification?
What I have done so far is:
import subprocess as sb
import glob
from nose import with_setup
def test_execute():
files = glob.glob("../*.py")
files.sort()
for fl in files:
try:
sb.call(["ipython", "--matplotlib=Qt4", fl])
except:
assert False, "File: %s ran with some errors\n" % (fl)
This kind of works, in that it suppresses the Figures, but it doesn't throw any exceptions (even if the program has an error). I am also not 100% sure what it is doing. Is it appending all of the figures to Qt4 or will the Figure be removed from memory when that script has finished?
Ideally I would like to ideally run each .py file and capture its stdout and stderr, then use the exit condition to report the stderr and fail the tests. Then when I run nosetests it will run the examples folder of programs and check that they all run.
You could force matplotlib to use the Agg backend (which won't open any windows) by inserting the following lines at the top of each source file:
import matplotlib
matplotlib.use('Agg')
Here's a one-liner shell command that will dynamically insert these lines at the top of my_script.py (without modifying the file on disk) before piping the output to the Python interpreter for execution:
~$ sed "1i import matplotlib\nmatplotlib.use('Agg')\n" my_script.py | python
You should be able to make the equivalent call using subprocess, like this:
p1 = sb.Popen(["sed", "1i import matplotlib\nmatplotlib.use('Agg')\n", fl],
stdout=sb.PIPE)
exit_cond = sb.call(["python"], stdin=p1.stdout)
You could capture the stderr and stdout from your scripts by passing the stdout= and stderr= arguments to sb.call(). This would, of course, only work in Unix environments that have the sed utility.
Update
This is actually quite an interesting problem. I thought about it a bit more, and I think this is a more elegant solution (although still a bit of a hack):
#!/usr/bin/python
import sys
import os
import glob
from contextlib import contextmanager
import traceback
set_backend = "import matplotlib\nmatplotlib.use('Agg')\n"
#contextmanager
def redirected_output(new_stdout=None, new_stderr=None):
save_stdout = sys.stdout
save_stderr = sys.stderr
if new_stdout is not None:
sys.stdout = new_stdout
if new_stderr is not None:
sys.stderr = new_stderr
try:
yield None
finally:
sys.stdout = save_stdout
sys.stderr = save_stderr
def run_exectests(test_dir, log_path='exectests.log'):
test_files = glob.glob(os.path.join(test_dir, '*.py'))
test_files.sort()
passed = []
failed = []
with open(log_path, 'w') as f:
with redirected_output(new_stdout=f, new_stderr=f):
for fname in test_files:
print(">> Executing '%s'" % fname)
try:
code = compile(set_backend + open(fname, 'r').read(),
fname, 'exec')
exec(code, {'__name__':'__main__'}, {})
passed.append(fname)
except:
traceback.print_exc()
failed.append(fname)
pass
print ">> Passed %i/%i tests: " %(len(passed), len(test_files))
print "Passed: " + ', '.join(passed)
print "Failed: " + ', '.join(failed)
print "See %s for details" % log_path
return passed, failed
if __name__ == '__main__':
run_exectests(*sys.argv[1:])
Conceptually this is very similar to my previous solution - it works by reading in the test scripts as strings, and prepending them with a couple of lines that will import matplotlib and set the backend to a non-interactive one. The string is then compiled to Python bytecode, then executed. The main advantage is that it this ought to be platform-independent, since sed is not required.
The {'__name__':'__main__'} trick with the globals is necessary if, like me, you tend to write your scripts like this:
def run_me():
...
if __name__ == '__main__':
run_me()
A few points to consider:
If you try to run this function from within an ipython session where you've already imported matplotlib and set an interactive backend, the set_backend trick won't work and you'll still get figures popping up. The easiest way is to run it directly from the shell (~$ python exectests.py testdir/ logfile.log), or from an (i)python session where you haven't set an interactive backend for matplotlib. It should also work if you run it in a different subprocess from within your ipython session.
I'm using the contextmanager trick from this answer to redirect stdin and stdout to a log file. Note that this isn't threadsafe, but I think it's pretty unusual for scripts to open subprocesses.
Coming to this late, but I am trying to figure something similar out myself, and this is what I have come up with so far. Basically, if your plots are calling, for example, matplotlib.pyplot.show to show the plot, you can mock that method out using a patch decorator. Something like:
from unittest.mock import patch
#patch('matplotlib.pyplot.show') # passes a mock object to the decorated function
def test_execute(mock_show):
assert mock_show() == None # shouldn't do anything
files = glob.glob("../*.py")
files.sort()
for fl in files:
try:
sb.call(["ipython", fl])
except:
assert False, "File: %s ran with some errors\n" % (fl)
Basically the patch decorator should replace any call to matplotlib.pyplot.show within the decorated function with a mock object that doesn't do anything. At least that's how it's supposed to work in theory. In my application, my terminal is still trying to open plots and this is resulting in errors. I hope it works better for you, and I will update if I figure out something wrong in the above that is leading to my issue.
Edit: for completeness, you might be generating figures with a call to matplotlib.pyplot.figure() or matplotlib.pyplot.subplots(), in which case these are what you would mock out instead of matplotlib.pyplot.show(). Same syntax as above, you would just use:
#patch('matplotlib.pyplot.figure')
or:
#patch('matplotlib.pyplot.subplots')
Say I've got a test suite like this:
class SafeTests(unittest.TestCase):
# snip 20 test functions
class BombTests(unittest.TestCase):
# snip 10 different test cases
I am currently doing the following:
suite = unittest.TestSuite()
loader = unittest.TestLoader()
safetests = loader.loadTestsFromTestCase(SafeTests)
suite.addTests(safetests)
if TARGET != 'prod':
unsafetests = loader.loadTestsFromTestCase(BombTests)
suite.addTests(unsafetests)
unittest.TextTestRunner().run(suite)
I have major problem, and one interesting point
I would like to be using nose or py.test (doestn't really matter which)
I have a large number of different applications that are exposing these test
suites via entry points.
I would like to be able to aggregate these custom tests across all installed
applications so I can't just use a clever naming convention. I don't
particularly care about these being exposed through entry points, but I
do care about being able to run tests across applications in
site-packages. (Without just importing... every module.)
I do not care about maintaining the current dependency on
unittest.TestCase, trashing that dependency is practically a goal.
EDIT This is to confirm that #Oleksiy's point about passing args to
nose.run does in fact work with some caveats.
Things that do not work:
passing all the files that one wants to execute (which, weird)
passing all the modules that one wants to execute. (This either executes
nothing, the wrong thing, or too many things. Interesting case of 0, 1 or
many, perhaps?)
Passing in the modules before the directories: the directories have to come
first, or else you will get duplicate tests.
This fragility is absurd, if you've got ideas for improving it I welcome
comments, or I set up
a github repo with my
experiments trying to get this to work.
All that aside, The following works, including picking up multiple projects
installed into site-packages:
#!python
import importlib, os, sys
import nose
def runtests():
modnames = []
dirs = set()
for modname in sys.argv[1:]:
modnames.append(modname)
mod = importlib.import_module(modname)
fname = mod.__file__
dirs.add(os.path.dirname(fname))
modnames = list(dirs) + modnames
nose.run(argv=modnames)
if __name__ == '__main__':
runtests()
which, if saved into a runtests.py file, does the right thing when run as:
runtests.py project.tests otherproject.tests
For nose you can have both tests in place and select which one to run using attribute plugin, which is great for selecting which tests to run. I would keep both tests and assign attributes to them:
from nose.plugins.attrib import attr
#attr("safe")
class SafeTests(unittest.TestCase):
# snip 20 test functions
class BombTests(unittest.TestCase):
# snip 10 different test cases
For you production code I would just call nose with nosetests -a safe, or setting NOSE_ATTR=safe in your os production test environment, or call run method on nose object to run it natively in python with -a command line options based on your TARGET:
import sys
import nose
if __name__ == '__main__':
module_name = sys.modules[__name__].__file__
argv = [sys.argv[0], module_name]
if TARGET == 'prod':
argv.append('-a slow')
result = nose.run(argv=argv)
Finally, if for some reason your tests are not discovered you can explicitly mark them as test with #istest attribute (from nose.tools import istest)
This turned out to be a mess: Nose pretty much exclusively uses the
TestLoader.load_tests_from_names function (it's the only function tested in
unit_tests/test_loader)
so since I wanted to actually load things from an arbitrary python object I
seemed to need to write my own figure out what kind of load function to use.
Then, in addition, to correctly get things to work like the nosetests script
I needed to import a large number of things. I'm not at all certain that this
is the best way to do things, not even kind of. But this is a stripped down
example (no error checking, less verbosity) that is working for me:
import sys
import types
import unittest
from nose.config import Config, all_config_files
from nose.core import run
from nose.loader import TestLoader
from nose.suite import ContextSuite
from nose.plugins.manager import PluginManager
from myapp import find_test_objects
def load_tests(config, obj):
"""Load tests from an object
Requires an already configured nose.config.Config object.
Returns a nose.suite.ContextSuite so that nose can actually give
formatted output.
"""
loader = TestLoader()
kinds = [
(unittest.TestCase, loader.loadTestsFromTestCase),
(types.ModuleType, loader.loadTestsFromModule),
(object, loader.loadTestsFromTestClass),
]
tests = None
for kind, load in kinds.items():
if isinstance(obj, kind) or issubclass(obj, kind):
log.debug("found tests for %s as %s", obj, kind)
tests = load(obj)
break
suite = ContextSuite(tests=tests, context=obj, config=config)
def main():
"Actually configure the nose config object and run the tests"
config = Config(files=all_config_files(), plugins=PluginManager())
config.configure(argv=sys.argv)
tests = []
for group in find_test_objects():
tests.append(load_tests(config, group))
run(suite=tests)
If your question is, "How do I get pytest to 'see' a test?", you'll need to prepend 'test_' to each test file and each test case (i.e. function). Then, just pass the directories you want to search on the pytest command line and it will recursively search for files that match 'test_XXX.py', collect the 'test_XXX' functions from them and run them.
As for the docs, you can try starting here.
If you don't like the default pytest test collection method, you can customize it using the directions here.
If you are willing to change your code to generate a py.test "suite" (my definition) instead of a unittest suite (tech term), you may do so easily. Create a file called conftest.py like the following stub
import pytest
def pytest_collect_file(parent, path):
if path.basename == "foo":
return MyFile(path, parent)
class MyFile(pytest.File):
def collect(self):
myname="foo"
yield MyItem(myname, self)
yield MyItem(myname, self)
class MyItem(pytest.Item):
SUCCEEDED=False
def __init__(self, name, parent):
super(MyItem, self).__init__(name, parent)
def runtest(self):
if not MyItem.SUCCEEDED:
MyItem.SUCCEEDED = True
print "good job, buddy"
return
else:
print "you sucker, buddy"
raise Exception()
def repr_failure(self, excinfo):
return ""
Where you will be generating/adding your code into your MyFile and MyItem classes (as opposed to the unittest.TestSuite and unittest.TestCase). I kept the naming convention of MyFile class that way, because it is intended to represent something that you read from a file, but of course you can basically decouple it (as I've done here). See here for an official example of that. The only limit is that in the way I've written this foo must exist as a file, but you can decouple that too, e.g. by using conftest.py or whatever other file name exist in your tree (and only once, otherwise everything will run for each files that matches -- and if you don't do the if path.basename test for every file that exists in your tree!!!)
You can run this from command line with
py.test -whatever -options
or programmactically from any code you with
import pytest
pytest.main("-whatever -options")
The nice thing with py.test is that you unlock many very powerful plugings such as html report