I'm having a little issue with the nose testing framework.
Scenario:
I want to test my software with devices I have connected to my computer. Each device is tested with different set of configurations. Luckily, as the testing code does not change, I can simply create files containing the configurations for each device.
For now, I have used normal test methods inside my class. Meaning that inside of my test method I'm loading the configuration file for the device I want to test and then iterate over all configurations in that file and perform the test. Unfortunately, this only gives one test result per device, not one per configuration the devices was tested with.
So I stumbled over nose own test generators. The old tests were modified so they are using a generator and everything worked fine so far, I got a result for each configuration, everything worked great.
My little Issue:
However, I've seen to hit a wall now. When using --collect-only to show the available tests, I get one of two possible outcomes:
A configuration file is loaded and the test generator generates the tests according to that configuration file. This means that nose displays the test with each possible parameter configuration. However, as the configuration file may not be the right one for the device I want to test later on, I get false result.
I found out that the plug-in that offers the --collect-only functionality bypasses the fixtures of the tests. So I moved the loading of the configuration into a fixture to avoid having the generated tests spam the list of available tests. Unfortunately, this resulted in no test being generated and hence not test being displayed as the generator didn't generate anything
What I tried so far:
So, I have tried a few things to solve this issue:
Using a flag to determine if --collect-only is running. As the collect plug-in bypasses the fixtures, I have set a flag in the generator with a default value of true and set it to false in the fixture of that generator. I hoped that checking the flag and simply ignoring the test generation when --collect-only was running would solve my issue. That was when I learned that nose checks if a test method is a generator and expects it to give test method.
As my first idea failed because nose knows that the function is a generator and expects at least one generated function from it, my second idea was to call the function with an empty set of parameters. As my configuration is stored in a dict(), I simply passed an empty dict. Luckily, this was enough to generate a test. However, apparently, nose makes a little check if the test is executable and if it fails, the test is once again ignored.
After the second fail I tried the other direction. I read the source code of nose a bit, trying to figure out how it works internally. That was when I stumbled over the "isgenerator()" check. So I thought, I would scan my directories for the tests on my own, add all static tests I find to a list and when I stumble across a generator, I will not generate tests, but add the name of the generator to the lists of tests. Well, this one fails so far as I have no real experience on how nose works internally. Meaning, I find all the static tests, however not the generators. For the moment, this is my code to find the tests:
k
from nose.loader import TestLoader
from nose.util import isgenerator
import os
folder = os.getcwd()
testLoader = TestLoader()
tests = testLoader.loadTestsFromDir(folder)
for testModule in tests:
print ("Found test module %s" % testModule.id())
for testFile in testModule:
print ("Found test file %s" % testFile.id())
for test in testFile:
print("Found test %s" % test.id())
if not isgenerator(test):
for x in test:
print("Found x %s" % x.id())
else:
print ("GENERATOR FOUND")
I ran out of ideas on what to try. Maybe I ran in some sort of X-Y problem, or maybe I'm simply blind to see the obvious solution.
Ok, so, after jumping through the debugger a bit more, I came up with the following solution and will post it here as an answer because it seems to work. However, it is neither really elegant looking nor do I know how stable it is exactly.
foundTests = dict()
for testPackage in loadedTests:
#print ("Found test module %s" % testModule.id())
for testModule in testPackage:
#print ("Found test file %s" % testFile.id())
for testCase in testModule:
#print("Found test %s" % test.id())
for test in testCase:
if isinstance(test, Test):
key, value = readTestInformation(test)
foundTests[key] = value
elif hasattr(test, "test_generator"):
factory = test.factory
for suite in factory.suites:
try:
if inspect.ismethod(suite):
key, value = readGeneratedTestInformation(suite)
foundTests[key]=value
except:
pass
This function simply iterates over all packages found, inside them iterates over all modules found, inside the modules it iterates over all classes and at last iterates over all test methods inside a class.
If a test method is a generator it seems to have the "test_generator" attribute, so I check for that and then iterate over all suites inside the factory, checking each time if it is a function. If it is, I got the test generator function.
So, if anyone comes up with a better solution, I'd be happy to see it. This way seems kind of hacky.
Related
I am looking to automate the process where:
I run some python code,
then run a set of tests using pytest
then, if all tests are validated, start the process again with new data.
I am thinking of writing a script executing the python code, then calling pytest using pytest.main(), check with the help of the exit code that all tests passed and in case of success start again.
The issue is that it is stated in pytest docs (https://docs.pytest.org/en/stable/usage.html) that it is not recommended to make multiple calls to pytest.main():
Note from pytest docs:
"Calling pytest.main() will result in importing your tests and any modules that they import. Due to the caching mechanism of python’s import system, making subsequent calls to pytest.main() from the same process will not reflect changes to those files between the calls. For this reason, making multiple calls to pytest.main() from the same process (in order to re-run tests, for example) is not recommended."
I was woundering if it was ok to call pytest.main() the way I intend to or if there was any better way to achieve what I am looking for?
I've made a simple example to make it problem more clear:
A = [0]
def some_action(x):
x[0] += 1
if __name__ == '__main__':
print('Initial value of A: {}'.format(A))
for i in range(10):
if i == 5:
# one test in test_mock2 that fails
test_dir = "./tests/functional_tests/test_mock2.py"
else:
# two tests in test_mock that pass
test_dir = "./tests/functional_tests/test_mock.py"
some_action(A)
check_tests = int(pytest.main(["-q", "--tb=no", test_dir]))
if check_tests != 0:
print('Interrupted at i={} because of tests failures'.format(i))
break
if i > 5:
print('All tests validated, final value of A: {}'.format(A))
else:
print('final value of A: {}'.format(A))
In this example some_action is executed until i reaches 5, at which point the tests fail and the process of executing/testing is interrupted. It seems to work fine, I'm only concerned because of the comments in pytest docs as stated above
The warning applies to the following sequence of events:
Run pytest.main on some folder which imports a.py, directly or indirectly.
Modify a.py (manually or programatically).
Attempt to rerun pytest.main on the same directory in the same python process as #1
The second run in step #3 will not not see the changes you made to a.py in step #2. That is because python does not import a file twice. Instead, it will check if the file has an entry in sys.modules, and use that instead. It's what lets you import large libraries multiple times without incurring a huge penalty every time.
Modifying the values in imported modules is fine. Python binds names to references, so if you bind something (like a new integer value) to the right name, everyone will be able to see it. Your some_action function is a good example of this. Future tests will run with the modified value if they import your script as a module.
The reason that the caveat is there is that pytest is usually used to test code after it has been modified. The warning is simply telling you that if you modify your code, you need to start pytest.main in a new python process to see the changes.
Since you do not appear to be modifying the code of the files in your test and expecting the changes to show up, the caveat you cite does not apply to you. Keep doing what you are doing.
The decorator #unittest.SkipTest prevents a unittest from being executed automatically when running the unit tests in a test class. Unfortunately, it also makes
the individual execution in PyCharm (by right-clicking on the function and
selecting Run/Debug for this test method) fail with a TypeError: don't know how to make test from: <function .... at 0x7fbc8e1234c0>
Is there a way to disable a unit test from automatic (bulk) execution
when running the complete test class, but leaving it executable by hand (preferably in PyCharm) such that it can be run without the need to make any changes in the test file?
I thought
#unittest.skipUnless(condition, reason)
might possibly come in handy, but could not come up with a condition that is satisfied only when the test is
launched by hand. Any suggestions appreciated!
Have you tried including a parameter that is set when run through CI/CD? You could have the default value set to false, and then CI/CD sets it to true. This would give you the condition you are looking for.
As far as I'm aware, the only way to differentiate between CI/CD runs and IDE runs is through some intermediary parameter that you must set.
Edit:
Try setting a custom build configuration in PyCharm for that specific test. You could have that build configuration pass in a parameter to your testsuite. At that point you would have a proper condition to have this test not be skipped when you run tests using the command line vs PyCharm's integrated test runner.
For simplicity, you'll want the default value of the parameter to be to skip, and only set the Boolean value to not skip by passing in True to that param in the special build config in PyCharm.
See: https://www.jetbrains.com/help/idea/build-configuration.html
I would try to control this with a parameter that is set only in PyCharm or another IDE.
That is, you could use skipUnless(...) with a condition relating to an environment variable defined in your PyCharm test configuration.
Here is a complete example:
import os
import unittest
def is_prime(n):
""" Return whether `n` is prime. """
return n == 2 or not (n % 2 == 0 or any(n % i == 0 for i in range(3, n, 2)))
class IsPrimeTest(unittest.TestCase):
#unittest.skipUnless(os.getenv('MYPROJECT_DEVELOPMENT_TEST'), reason="Lengthy test")
def test_is_prime(self):
self.assertTrue(is_prime(2))
self.assertTrue(is_prime(3))
self.assertFalse(is_prime(4))
self.assertTrue(is_prime(5))
When run from the shell, the tests are indeed skipped:
$ python3 -m unittest test_example.py
s
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK (skipped=1)
If you select "Edit Configurations...", you can set the specific environment variable in the test configuration, as shown in the screenshot below. With this change, the test in question is actually run in PyCharm (not shown).
Addendum from my comment below:
One helpful option may be to create multiple test "Targets" in PyCharm, using the "+" button (the upper left corner of my screenshot). Then you could have one target that includes the test, and one target that does not, naming them "Standard unit tests" and "All unit tests", for example. To switch between them, you could choose each in a certain dropdown next to a green "play" button (toolbar in my main IDE window). This seems pretty ergonomic and simple.
When running tests that target a specific method which uses reflection, I encounter the problem that the output of tests is dependent on whether I run them with PTVS ('run all tests' in Test Explorer) or with the command line Python tool (both on Windows and Linux systems):
$ python -m unittest
I assumed from the start that it has something to do with differences in how the test runners work in PTVS and Python's unittest framework (because I've noticed other differences, too).
# method to be tested
# written in Python 3
def create_line(self):
problems = []
for creator in LineCreator.__subclasses__():
item = creator(self.structure)
cls = item.get_subtype()
line = cls(self.text)
try:
line.parse()
return line
except ParseException as exc:
problems.append(exc)
raise ParseException("parsing did not succeed", problems)
""" subclasses of LineCreator are defined in separate files.
They implement get_subtype() and return the class objects of the actual types they must instantiate.
"""
I have noticed that the subclasses found in this way will vary, depending on which modules have been loaded in the code that calls this method. This is exactly what I want (for now). Given this knowledge, I am always careful to only have access to one subclass of LineCreator in any given test module, class, or method.
However, when I run the tests from the Python command line, it is clear from the ParseException.problems attribute that both are loaded at all times. It is also easy to reproduce: inserting the following code makes all tests fail on the command line, yet they succeed on PTVS.
if len(LineCreator.__subclasses__()) > 1:
raise ImportError()
I know that my tests should run independently from each other and from any contextual factors. That is actually what I'm trying to achieve here.
In case I wasn't clear, my question is why behaviors are different, and which one is correct. And if you're feeling really generous, how to change my code to make tests succeed on all platforms.
I'm wondering if there is a way I can specify the list of tests to run in a file.
I already know that you can do it in the command like so:
nosetests myfile.py:test1 myfile.py:test3 myfile.py:test20
but this will get out of hand easily.
I also know that there is a
loadTestsFromNames(names, module=None)
method as part of python unittest which can probably be used to load test names parsed from a yaml file but then if nose does not have such a thing implemented, it's quite useless to me.
Despite valid concerns in the comments to your question, it is perfectly valid to wish to run a select, limited number of testcases. For example: all testcases might be expensive, while a select set can indicate acceptance for more testing. Or you might need to control the "ordering" of tests - which is bad practice for unit tests (but with very slow integration tests, might be something you're forced to accept).
Two suggestions:
1) test tagging:
http://nose.readthedocs.io/en/latest/plugins/attrib.html
2) You can also write a small Nose plugin which selects tests based on patterns, or returns a fixed list of tests.
I use test tagging (nosetests -a "tagname") and it's flexible. The Nose plugin will work also; haven't tried it myself.
This one creates many xunit.xml-s which is something I wanted for jenkins.
list_of_tests = ["test_1", "test_2", "test_3"]
for atest in list_of_tests:
test = Popen(['nosetests', '--nologcapture', 'myfile.py:%s'%atest, '-v', '-s', '--with-xunit', '--xunit-file=%s.xml' % atest], stdout=PIPE, stderr=STDOUT)
for line in iter(test.stdout.readline, ''):
print line.strip()
I'm doing TDD using Python and the unittest module. In NUnit you can Assert.Inconclusive("This test hasn't been written yet").
So far I haven't been able to find anything similar in Python to indicate that "These tests are just placeholders, I need to come back and actually put the code in them."
Is there a Pythonic pattern for this?
With the new and updated unittest module you can skip tests:
#skip("skip this test")
def testSomething(self):
pass # TODO
def testBar(self):
self.skipTest('We need a test here, really')
def testFoo(self):
raise SkipTest('TODO: Write a test here too')
When the test runner runs these, they are counted separately ("skipped: (n)").
I would not let them pass or show OK, because you will not find them easily back.
Maybe just let them fail and the reason (not written yet), which seems logical because you have a test that is not finished.
I often use self.fail as a todo list
def test_some_edge_case(self):
self.fail('Need to check for wibbles')
Works well for me when I'm doing tdd.