We are trying to implement an automated testing framework using nose. The intent is to add a few command line options to pass into the tests, for example a hostname. We run these tests against a web app as integration tests.
So, we've created a simple plugin that adds a option to the parser:
import os
from nose.plugins import Plugin
class test_args(Plugin):
"""
Attempting to add command line parameters.
"""
name = 'test_args'
enabled = True
def options(self, parser, env=os.environ):
super(test_args, self).options(parser, env)
parser.add_option("--hostname",
action="store",
type="str",
help="The hostname of the server")
def configure(self, options, conf):
self.hostname = options.hostname
The option is available now when we run nosetests...but I can't figure out how to use it within a test case? Is this possible? I can't find any documentation on how to access options or the configuration within a test case.
The adding of the command line arguments is purely for development/debugging code purposes. We plan to use config files for our automated runs in bamboo. However, when developing integration tests and also debugging issues, it is nice to change the config on the fly. But we need to figure out how to actually use the options first...I feel like I'm just missing something basic, or I'm blind...
Ideally we could extend the testconfig plugin to make passing in config arguments from this:
--tc=key:value
to:
--key=value
If there is a better way to do this then I'm all ears.
One shortcut is to access import sys; sys.argv within the test - it will have the list of parameters passed to the nose executable, including the plugin ones. Alternatively your plugin can add attributes to your tests, and you can refer to those attributes - but it requires more heavy lifting - similar to this answer.
So I've found out how to make this work:
import os
from nose.plugins import Plugin
case_options = None
class test_args(Plugin):
"""
Attempting to add command line parameters.
"""
name = 'test_args'
enabled = True
def options(self, parser, env=os.environ):
super(test_args, self).options(parser, env)
parser.add_option("--hostname",
action="store",
type="str",
help="The hostname of the server")
def configure(self, options, conf):
global case_options
case_options = options
Using this you can do this in your test case to get the options:
from test_args import case_options
To solve the different config file issues, I've found you can use a setup.cfg file written like an INI file to pass in default command line parameters. You can also pass in a -c config_file.cfg to pick a different config. This should work nicely for what we need.
Related
My codebase has multiple scripts, say sc1.py and sc2.py. Their code looks like this (just replace 1 for 2 to imagine what the other is like):
#click.command()
#click.option(--op1, default="def1", required=True)
def sc1(op1):
pass
if __name__ == "__main__":
sc1()
What I want to do is write a wrapper that handles concerns like loading the config file (and handling missing ones appropriately), configuring the logger, etc.
The logger, as an example, requires some options, e.g. the log level, and this option doesn't appear in any of the existing scripts. I'd like to handle it at the wrapper level (since the wrapper will deal with common concerns like log configuration).
That is what the wrapper might look like:
#click.command()
#click.option(--level, default=INFO, required=False)
def wrapper(level, wrapped_main_function):
# use level to setup logger basic config.
wrapped_main_function()
Then I would modify the "main" in the scripts (sc1.py, sc2.py etc):
from wrap_module import wrapper
#click.command()
#click.option(--op1, default="def1", required=True)
def sc1(op1):
pass
if __name__ == "__main__":
wrapper(???, sc1)
Put into words I am trying to split the options between two functions, say wrapper and sc1, and have wrapper deal with its own dedicated options and then call sc1, forwarding the remaining command line options. All while wrapper is not aware of what options those might be, since the whole point is to write something generic that can be used for all the scripts and having wrapper deal with the commonalities.
The command line should look something like:
python sc.py --op1 something --level debug
I can't figure out the right syntax to do what I want.
I think you can get close to what you want with a slightly different approach. First, we have the following in common.py:
import click
def add_common_options(func):
func = click.option("--level")(func)
return func
def handle_common_options(level):
print("log level is", level)
And then in sc1.py, we have:
import click
from common import add_common_options, handle_common_options
#click.command()
#click.option("--op1")
#add_common_options
def sc1(op1, **common_options):
handle_common_options(**common_options)
print("op1 is", op1)
if __name__ == "__main__":
sc1()
(With sc2.py implemented in a similar fashion.)
This gets you the behavior you want. The help output for sc1.py
looks like:
Usage: sc1.py [OPTIONS]
Options:
--op1 TEXT
--level TEXT
--help Show this message and exit.
And you can run a command line like this:
$ python sc1.py --op1 foo --level info
log level is info
op1 is foo
I have trace32 installed at C drive and have mentioned that directory in my code. Suppose if some other user run this code in their system, the code does not work because the user has installed application in different location. How can I make this directory generic and dynamic and make it work for all users?
You have multiple possibilities. Bevor explaining them some generic tips:
Make the TRACE32 system path configurable, not a path inside the installation. In your case this would be r"C:\T32". This path is called t32sys or T32SYS.
Make sure you use os.path.join to concatenate your strings, so it works on the users operating system: os.path.join(r"C:\T32", "bin/windows64")
Command line arguments using argparse. This is the simplest solution which requires the user to start the Python script like this: python script.py --t32sys="C:\t32".
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--t32sys", help="TRACE32 system directory.")
args = parser.parse_args()
t32sys = args["t32sys"]
Instead of command line parameters you could also use a configuration file. For this you can use the built-in configparser module. This has the advantage that the user doesn't need to specify the directory as a command line argument, but the disadvantage that the user needs to be aware of the configuration file.
Configuration file (example.ini):
[DEFAULT]
t32sys = C:\T32
import configparser
parser = configparser.ConfigParser()
parser.read("example.ini")
args = parser["DEFAULT"]
t32sys = args["t32sys"]
Environment variables using os.environ. T32SYS is a environment variable often used for this, but it's not ensured to be set. So you have to tell users that they have to set the variable before using your tool. This approach has the advantage to work in the background, but also in my opinion a little bit obfuscating. I'd only use this in combination with argparse or configparse to override.
import os
t32sys = os.environ.get('T32SYS')
You can of course combine multiple ways with fallbacks / overrides.
I wonder, is it possible to pass behave arguments, eg. "-D environment". by default they are taken from the behave file.
Maybe there is some way to keep each configuration in a different file? Maybe many behave files? or Such as: behave "path to file with arguments"?
At this point i figured out that I could put bash scripts containing various configuration " #!/bin/bash behave ..."
I asking because i want easily manage my configurations when I run "behave -..." without editing many arguments
I think you could take advantage of using custom json configuration files described in behave Advanced Cases section: https://behave.readthedocs.io/en/latest/new_and_noteworthy_v1.2.5.html#advanced-cases
from documentation:
# -- FILE: features/environment.py
import json
import os.path
def before_all(context):
"""Load and update userdata from JSON configuration file."""
userdata = context.config.userdata
configfile = userdata.get("configfile", "userconfig.json")
if os.path.exists(configfile):
assert configfile.endswith(".json")
more_userdata = json.load(open(configfile))
context.config.update_userdata(more_userdata)
# -- NOTE: Reapplies userdata_defines from command-line, too.
So, if you like to use custom config by specifying it at the command line, you could run it as for example:
behave -D conffile=myconfig.json
Then I would parametrize this line to something like:
myconfigfile = context.config.userdata["conffile"]
configfile = userdata.get("configfile", myconfigfile)
I am new to pytest framework and I wonder if there is a standard approach my pytests to receive parameters from other script. I guess the most common practice is to write #pytest.fixture that looks like:
#pytest.fixture
def param(request):
device = request.config.getoption("--parameter")
if parameter == 'A':
return value_a
else:
return default_value
And running my test with command py.test --parameter=value_a(this command is executed by some other script)
But if my test needs many parameters, lets say for example 10, that would be one long command. So I am asking what is the standard approach in situation like that - do I provide some kind of xml file or serialized dictionary with parameters and my fixture to take the parameters from them.
Also how my other script will know what kind of parameters to provide to my test - should there be test_parameters configuration file or some hardcoded data in my conftest.py, that holds the information for the parameters, or should I use get them by reading the signature of the tests with from inspect import signature.
EDITED
Here is a sample of my tests
class TestH5Client:
# runs before everything else in the class
def setup_class(cls, ip="11.111.111.111",
browserType="Chrome",
port="4444",
client_url="https://somelink.com/",
username="some_username",
hpassword="some_pass"):
cls.driver = get_remote_webdriver(ip, port, browserType)
cls.driver.implicitly_wait(60)
cls.client_url = client_url
cls.username = username
cls.password = password
def teardown_class(cls):
cls.driver.quit()
def test_login_logout(self):
# opening web_client
self.driver.set_page_load_timeout(60)
self.driver.get(self.h5_client_url)
# opening web_client log-in window
self.driver.set_page_load_timeout(60)
self.driver.find_element_by_css_selector("div.gettingStarted p:nth-child(4) a:nth-child(1)").click()
# log in into the client
self.driver.find_element_by_id("username").send_keys(self.h5_username)
self.driver.find_element_by_id("password").send_keys(self.h5_password)
self.driver.set_page_load_timeout(60)
self.driver.find_element_by_id("submit").click()
# clicking on the app_menu so the logout link appears
self.driver.implicitly_wait(60)
self.driver.find_element_by_id("action-userMenu").click()
# clicking on the logout link
self.driver.implicitly_wait(60)
self.driver.find_element_by_css_selector("#vui-actions-menu li:nth-child(3)").click()
assert "Login" in self.driver.title
def test_open_welcome_page(self):
"""fast selenium test for local testing"""
self.driver.set_page_load_timeout(20)
self.driver.get(self.h5_client_url)
assert "Welcome" in self.driver.title
def test_selenium_fail(self):
"""quick test of selenium failure for local testing"""
self.driver.set_page_load_timeout(20)
self.driver.get(self.h5_client_url)
assert "NotInHere" in self.driver.title
I need all those parameters to be provided by an outside python providing test parameters framework. And I need to know how this framework should get the name of the parameters and how to run those tests with these parameters.
Given your answers in the comments, the data changes weekly.
So I'd suggest passing in a single parameter, the path to a file specifying the rest of your info.
Use whatever parsing mechanism you're already using elsewhere - XML, Json, whatever - or use something simple like a config file reader.
Create a fixture with session scope (like this) and either give it some sensible defaults, or have it fail violently if it doesn't get a valid parameter file.
Say I've got a test suite like this:
class SafeTests(unittest.TestCase):
# snip 20 test functions
class BombTests(unittest.TestCase):
# snip 10 different test cases
I am currently doing the following:
suite = unittest.TestSuite()
loader = unittest.TestLoader()
safetests = loader.loadTestsFromTestCase(SafeTests)
suite.addTests(safetests)
if TARGET != 'prod':
unsafetests = loader.loadTestsFromTestCase(BombTests)
suite.addTests(unsafetests)
unittest.TextTestRunner().run(suite)
I have major problem, and one interesting point
I would like to be using nose or py.test (doestn't really matter which)
I have a large number of different applications that are exposing these test
suites via entry points.
I would like to be able to aggregate these custom tests across all installed
applications so I can't just use a clever naming convention. I don't
particularly care about these being exposed through entry points, but I
do care about being able to run tests across applications in
site-packages. (Without just importing... every module.)
I do not care about maintaining the current dependency on
unittest.TestCase, trashing that dependency is practically a goal.
EDIT This is to confirm that #Oleksiy's point about passing args to
nose.run does in fact work with some caveats.
Things that do not work:
passing all the files that one wants to execute (which, weird)
passing all the modules that one wants to execute. (This either executes
nothing, the wrong thing, or too many things. Interesting case of 0, 1 or
many, perhaps?)
Passing in the modules before the directories: the directories have to come
first, or else you will get duplicate tests.
This fragility is absurd, if you've got ideas for improving it I welcome
comments, or I set up
a github repo with my
experiments trying to get this to work.
All that aside, The following works, including picking up multiple projects
installed into site-packages:
#!python
import importlib, os, sys
import nose
def runtests():
modnames = []
dirs = set()
for modname in sys.argv[1:]:
modnames.append(modname)
mod = importlib.import_module(modname)
fname = mod.__file__
dirs.add(os.path.dirname(fname))
modnames = list(dirs) + modnames
nose.run(argv=modnames)
if __name__ == '__main__':
runtests()
which, if saved into a runtests.py file, does the right thing when run as:
runtests.py project.tests otherproject.tests
For nose you can have both tests in place and select which one to run using attribute plugin, which is great for selecting which tests to run. I would keep both tests and assign attributes to them:
from nose.plugins.attrib import attr
#attr("safe")
class SafeTests(unittest.TestCase):
# snip 20 test functions
class BombTests(unittest.TestCase):
# snip 10 different test cases
For you production code I would just call nose with nosetests -a safe, or setting NOSE_ATTR=safe in your os production test environment, or call run method on nose object to run it natively in python with -a command line options based on your TARGET:
import sys
import nose
if __name__ == '__main__':
module_name = sys.modules[__name__].__file__
argv = [sys.argv[0], module_name]
if TARGET == 'prod':
argv.append('-a slow')
result = nose.run(argv=argv)
Finally, if for some reason your tests are not discovered you can explicitly mark them as test with #istest attribute (from nose.tools import istest)
This turned out to be a mess: Nose pretty much exclusively uses the
TestLoader.load_tests_from_names function (it's the only function tested in
unit_tests/test_loader)
so since I wanted to actually load things from an arbitrary python object I
seemed to need to write my own figure out what kind of load function to use.
Then, in addition, to correctly get things to work like the nosetests script
I needed to import a large number of things. I'm not at all certain that this
is the best way to do things, not even kind of. But this is a stripped down
example (no error checking, less verbosity) that is working for me:
import sys
import types
import unittest
from nose.config import Config, all_config_files
from nose.core import run
from nose.loader import TestLoader
from nose.suite import ContextSuite
from nose.plugins.manager import PluginManager
from myapp import find_test_objects
def load_tests(config, obj):
"""Load tests from an object
Requires an already configured nose.config.Config object.
Returns a nose.suite.ContextSuite so that nose can actually give
formatted output.
"""
loader = TestLoader()
kinds = [
(unittest.TestCase, loader.loadTestsFromTestCase),
(types.ModuleType, loader.loadTestsFromModule),
(object, loader.loadTestsFromTestClass),
]
tests = None
for kind, load in kinds.items():
if isinstance(obj, kind) or issubclass(obj, kind):
log.debug("found tests for %s as %s", obj, kind)
tests = load(obj)
break
suite = ContextSuite(tests=tests, context=obj, config=config)
def main():
"Actually configure the nose config object and run the tests"
config = Config(files=all_config_files(), plugins=PluginManager())
config.configure(argv=sys.argv)
tests = []
for group in find_test_objects():
tests.append(load_tests(config, group))
run(suite=tests)
If your question is, "How do I get pytest to 'see' a test?", you'll need to prepend 'test_' to each test file and each test case (i.e. function). Then, just pass the directories you want to search on the pytest command line and it will recursively search for files that match 'test_XXX.py', collect the 'test_XXX' functions from them and run them.
As for the docs, you can try starting here.
If you don't like the default pytest test collection method, you can customize it using the directions here.
If you are willing to change your code to generate a py.test "suite" (my definition) instead of a unittest suite (tech term), you may do so easily. Create a file called conftest.py like the following stub
import pytest
def pytest_collect_file(parent, path):
if path.basename == "foo":
return MyFile(path, parent)
class MyFile(pytest.File):
def collect(self):
myname="foo"
yield MyItem(myname, self)
yield MyItem(myname, self)
class MyItem(pytest.Item):
SUCCEEDED=False
def __init__(self, name, parent):
super(MyItem, self).__init__(name, parent)
def runtest(self):
if not MyItem.SUCCEEDED:
MyItem.SUCCEEDED = True
print "good job, buddy"
return
else:
print "you sucker, buddy"
raise Exception()
def repr_failure(self, excinfo):
return ""
Where you will be generating/adding your code into your MyFile and MyItem classes (as opposed to the unittest.TestSuite and unittest.TestCase). I kept the naming convention of MyFile class that way, because it is intended to represent something that you read from a file, but of course you can basically decouple it (as I've done here). See here for an official example of that. The only limit is that in the way I've written this foo must exist as a file, but you can decouple that too, e.g. by using conftest.py or whatever other file name exist in your tree (and only once, otherwise everything will run for each files that matches -- and if you don't do the if path.basename test for every file that exists in your tree!!!)
You can run this from command line with
py.test -whatever -options
or programmactically from any code you with
import pytest
pytest.main("-whatever -options")
The nice thing with py.test is that you unlock many very powerful plugings such as html report