How to detect which tests pytest need to run?
Up to now I run all tests in CI. This is slow.
Isn't there a way to automatically detect which tests need to run, and only execute a fraction of the whole test suite?
Background: I am reading the Google SE book and read about their gigantic monorepo and that their tool Blaze can detect which tests need to run. Sounds very cool.
I had good experience with pytest-testmon which determines the subset of tests to execute on code change based on test coverage. The details on that are explained well in Determining affected tests section of testmon's docs, but the general advice is to maintain high line/branch coverage for testmon to be effective. This also means that it will fail where coverage fails.
If you want to use it in the CI, you need to cache the .testmondata database file between test jobs and ensure the --testmon arg is used when invoking pytest. It also depends on your CI provider, but most popular ones out there offer caching mechanisms (Github, Gitlab, CircleCI, Appveyor etc).
testmon can also be effectively combined with pytest-watch watchdog to keep a daemon for test execution running. An example command that reruns subset of tests on code changes and sends system notifications on success or failure:
$ pytest-watch --runner="pytest --testmon" \
--onfail="notify-send --urgency=low -i error 'Tests Failed'" \
--onpass="notify-send --urgency=low -i info 'Tests passed'"
You can also use Bazel for building and testing as well; however, IMO it is an overkill for a pure Python project and is not worth it for just the incremental testing purposes.
You can use the pytest-incremental plugin for this which uses AST to analyse the import structure of a project and only runs tests for which the source code has changed.
pip install pytest-incremental
python -m pytest --inc
Analyzing the graph is easy to see that a change in app would cause
only test_app to be execute. And a change in util would cause all
tests to be executed.
from the docs
You can add #pytest.mark to the tests
#pytest.mark.SomeFeature
#pytest.mark.Regression
def test_example1(self):
pass
#pytest.mark.AnotherFeature
#pytest.mark.Regression
def test_example2(self):
pass
And use it when triggering the tests
pytest TestsFolder -m SomeFeature
This will run only test_example1, -m Regression will run both tests.
What's the project about: a C++ library on top of which we would like to add a runtime prediction-module which is composed by python plugins (e.g. a plugin for the modeling phase, another one for the prediction, another one for features clustering, etc.). The destination OS is GNU/Linux.
A maybe naive idea is about running a shell script which inside is calling the main python plugin script: so from the main application we would fork into a child process which will be assigned with the shell script file execution. Despite being an easy-to-use solution, it seems to be missing a bit of communication potential.
Do you have any suggestions for a more elegant and robust solution? Or even some reference projects with a similar structure.
I have a third party software which is able to run some python scripts using something like:
software.exe -script pythonscript.py
My company is heavily dependent on this software as well as on the scripts we develop for it. Currently we have some QA that checks the output of the scripts, but we really want to start unit testing the scripts to make it easier to find bugs and make the test system more complete.
My problem is how is it possible to run "embedded" unit tests? We use pydev+eclipse and I tried to use it's remote debbuging to make it work with the unit tests, but I cannot really make it work. How can I make the server connection "feed" the unit test?
The other idea would be to parse the stdout of the software, but that would not really be a unit test... And the added complexity it seems to bring makes this approach less interesting.
I would expect that something like this has already been done somewhere else and I tried googling for it, but maybe I am just not using the correct keywords. Could anyone give me a starting point?
Thank you
A bit more info would be helpful. Are you using a testing framework (e.g. unittest or nose), or if not, how are the tests structured? What is software.exe?
In python, unit tests are really nothing more than a collection of functions which raise an exception on failure, so they can be called from a script like any other function. In theory, therefore, you can simply create a test runner (if you're not already using one such as nose), and run it as software.exe -script runtests.py. In pydev, you can set up software.exe as a customised python interpreter.
If the problem is that software.exe hides stdout, then simply write the results to a log file instead. You could also create a environment which mocks that provided by software.exe and run the tests using python.exe.
If unit tests are for your code and not for the functionality provided by software.exe then you could run the tests using a standalone python mocking software.exe parts where necessary. As an intermediate step you could try to run unittest-based scripts using `software.exe'
Well, generally speaking, testing software shall be done by a Continuous Integration suite (And Jenkins is your friend).
Now, I think you'll have to test your scripts pythonscript.py by setting a test() function inside the python script that will emulate the possible environments you'll give to the entry point of your script. And you'll be able to use unittest to execute the test functions of all your scripts. You can also embed tests in doctests, but I personally don't like that.
And then, in your software.exe, you'll be able to execute tests by emulating all the environment combinations. But as you don't say much about software.exe I won't be able to help you more... (what language ? is software.exe already unit tested ?)
nose
is a test runner which extends PyUnit. Is it possible to write e.g
$ nosetests --with-shell myTest.py -myargs test
If not, then is there a plugin, or do i need to develop it myself.
Any suggestions ?
Nose is not a general test harness. It's specifically a Python harness which runs Python unit tests.
So, while you can write extensions for it to execute scripts and mark them as successes or failures based on the exit status or an output string, I think it's an attempt to shoehorn the harness into doing something it's not really meant to do.
You should package your tests as Python functions or classes and then have them use a library to run external scripts the output or behaviour of which is translated into something that nose can interpret rather than extend nose to directly run scripts.
Also, I've experimented with nose a bit and found it's extension mechanism quite clumsy compared to py.test. You might want to give that a shot.
Basically, growl notifications (or other callbacks) when tests break or pass. Does anything like this exist?
If not, it should be pretty easy to write.. Easiest way would be to..
run python-autotest myfile1.py myfile2.py etc.py
Check if files-to-be-monitored have been modified (possibly just if they've been saved).
Run any tests in those files.
If a test fails, but in the previous run it passed, generate a growl alert. Same with tests that fail then pass.
Wait, and repeat steps 2-5.
The problem I can see there is if the tests are in a different file. The simple solution would be to run all the tests after each save.. but with slower tests, this might take longer than the time between saves, and/or could use a lot of CPU power etc..
The best way to do it would be to actually see what bits of code have changed, if function abc() has changed, only run tests that interact with this.. While this would be great, I think it'd be extremely complex to implement?
To summarise:
Is there anything like the Ruby tool autotest (part of the ZenTest package), but for Python code?
How do you check which functions have changed between two revisions of a script?
Is it possible to determine which functions a command will call? (Somewhat like a reverse traceback)
I found autonose to be pretty unreliable but sniffer seems to work very well.
$ pip install sniffer
$ cd myproject
Then instead of running "nosetests", you run:
$ sniffer
Or instead of nosetests --verbose --with-doctest, you run:
$ sniffer -x--verbose -x--with-doctest
As described in the readme, it's a good idea to install one of the platform-specific filesystem-watching libraries, pyinotify, pywin32 or MacFSEvents (all installable via pip etc)
autonose created by gfxmonk:
Autonose is an autotest-like tool for python, using the excellent nosetest library.
autotest tracks filesystem changes and automatically re-run any changed tests or dependencies whenever a file is added, removed or updated. A file counts as changed if it has iself been modified, or if any file it imports has changed.
...
Autonose currently has a native GUI for OSX and GTK. If neither of those are available to you, you can instead run the console version (with the --console option).
I just found this: http://www.metareal.org/p/modipyd/
I'm currently using thumb.py, but as my current project transitions from a small project to a medium sized one, I've been looking for something that can do a bit more thorough dependency analysis, and with a few tweaks, I got modipyd up and running pretty quickly.
Guard is an excellent tool that monitors for file changes and triggers tasks automatically. It's written in Ruby, but it can be used as a standalone tool for any task like this. There's a guard-nosetests plugin to run Python tests via nose.
Guard supports cross-platform notifications (Linux, OSX, Windows), including Growl, as well as many other great features. One of my can't-live-without dev tools.
One very useful tool that can make your life easier is entr. Written in C, and uses kqueue or inotify under the hood.
Following command runs your test suite if any *.py file in your project is changed.
ls */**.py | entr python -m unittest discover -s test
Works for BSD, Mac OS, and Linux. You can get entr from Homebrew.
Maybe buildbot would be useful http://buildbot.net/trac
For your third question, maybe the trace module is what you need:
>>> def y(a): return a*a
>>> def x(a): return y(a)
>>> import trace
>>> tracer = trace.Trace(countfuncs = 1)
>>> tracer.runfunc(x, 2)
4
>>> res = tracer.results()
>>> res.calledfuncs
{('<stdin>', '<stdin>', 'y'): 1, ('<stdin>', '<stdin>', 'x'): 1}
res.calledfuncs contains the functions that were called. If you specify countcallers = 1 when creating the tracer, you can get caller/callee relationships. See the docs of the trace module for more information.
You can also try to get the calls via static analysis, but this can be dangerous due to the dynamic nature of Python.
Django's development server has a file change monitor that watches for modifications and automatically reloads itself. You could re-use this code to launch unit tests on file modification.
Maybe Nose http://somethingaboutorange.com/mrl/projects/nose/ has a plugin http://somethingaboutorange.com/mrl/projects/nose/doc/writing_plugins.html
Found this: http://jeffwinkler.net/2006/04/27/keeping-your-nose-green/
You can use nodemon for the task, by watching .py files and execute manage.py test. The command will be: nodemon --ext py --exec "python manage.py test".
nodemon is an npm package however, I assume you have node installed.
Check out pytddmon. Here is a video demonstration of how to use it:
http://pytddmon.org/?page_id=33