I have a main function specified as entry point in my package's setup.py which uses the argparse package in order to pass command line arguments (see discussion here):
# file with main routine specified as entry point in setup.py
import argparse
def main():
parser = argparse.ArgumentParser()
parser.add_argument('a', type=str, help='mandatory argument a')
args = parser.parse_args()
Ideally, I would like to use the same main function in the package's tests as suggested here. In the latter context, I would like to call the main function from within the test class and set (some of) the command line arguments prior to the function call (which otherwise will fail, due to missing arguments).
# file in the tests folder calling the above main function
class TestConsole(TestCase):
def test_basic(self):
set_value_of_a()
main()
Is that possible?
The argparse module actually reads input variables from special variable, which is called ARGV (short from ARGument Vector). This variable is usually accessed by reading sys.argv from sys module.
This variable is a ordinary list, so you can append your command-line parameters to it like this:
import sys
sys.argv.extend(['-a', SOME_VALUE])
main()
However, messing with sys.argv at runtime is not a good way of testing.
A much more cleaner way to replace the sys.argv for some limited scope is using unittest.mock.patch context manager, like this:
with unittest.mock.patch('sys.argv'. ['-a', SOME_VALUE]):
main()
Read more about unittest.mock.patch in documentation
Also, check this SO question:
How do I set sys.argv so I can unit test it?
#William Fernandes: Just for the sake of completeness, I'll post the full solution in the way that was suggested by (checking for an empty dict not kwargs is None):
def main(**kwargs):
a = None
if not kwargs:
parser = argparse.ArgumentParser()
parser.add_argument('a', type=str, help='mandatory argument a')
args = parser.parse_args()
a = args.a
else:
a = kwargs.get('a')
print(a)
From within the test class the main function can then be called with arguments:
# file in the tests folder calling the above main function
class TestConsole(TestCase):
def test_basic(self):
main(a=42)
The call from the command line without kwargs then requires the specification of the command line argument a=....
Add kwargs to main and if they're None, you set them to the parse_args.
Related
The standard mantra to pass a configuration to hydra is to decorate with #hydra.main then to define your main(cfg), and then in the main block call main with no arguments. But suppose I want to also accept non-hydra command line args, so want to pass both args and the config? Is this viewed as morally impure and discouraged? Here is an example (which does not work) of the sort of thing I mean (it errors out claiming it is missing a cfg argument).
from omegaconf import DictConfig, OmegaConf
import hydra
#hydra.main(version_base=None)
def my_app(foo, cfg: DictConfig) -> None:
print(foo)
print(OmegaConf.to_yaml(cfg))
if __name__ == "__main__":
my_app(5)
This is not supported.
In principle, you have nothing to base different values for your 5 parameter before the config is composed.
If you want to have additional logic before composing the config, look at the Compose API as an alternative to #hydra.main() (or in addition).
I'm coding a script that I need to execute with the following line (this is a compulsory request for an assignment).
python3 my_program.py method_name
Where method_name is the name of the algorithm in the script that I want to run in this execution. To do this I'm using argparse like this:
parser = argparse.ArgumentParser(description=None)
parser.add_argument('--method', type=str, help='Method')
args = parser.parse_args([])
If I print the args this is the result:
print(args)
Namespace(method=None)
So apparently args is empty even it seems like it read the execution line correctly, because I did not get any error message. Why is my code not reading the name of the method? Should I be using a different method than argparse to read the method_name with the requested line? Or am I using argparse wrong?
Thank you! :)
Because you're explicitly passing an empty list to .parse_args(). Just use:
args = parser.parse_args()
to capture the passed --method into the argparse.Namespace.
Detail: if you look at the argparse.py code in CPython, you'll see the following:
def parse_known_args(self, args=None, namespace=None):
if args is None:
# args default to the system args
args = _sys.argv[1:]
else:
# make sure that args are mutable
args = list(args)
You want the if args is None branch (that's the implicit with parse_args()), which will actually capture sys.argv[:1], e.g. the options passed on the command line.
This question already has answers here:
Calling functions by array index in Python
(5 answers)
Closed 1 year ago.
I am new to python and I am exploring python I have many different functions in 1 file want to expose those functions to client.
This is my app.py file
import sys
def get_data(e_id, t_id):
#some code
def get_2_data(e_id, t_id)
#some code
def get_3_data(t_id)
#some code
if __name__ == '__main__':
get_data(sys.argv[1], sys.argv[2])
Here I want to get specific function data. Currently I am running python app.py 1234 1.
The function which is defined under main gets called.
But I want get_3_data() data. How to call the particular function or someone wants to fetch get_2_data(). How to expose those functions. Any suggestion. I dont want any HTTP call or API. I want to call by method name.
The clean way to do this is by using the module argparse.
Here is how to do this:
import argparse
def get_data(args):
e_id = args.e_id
t_id = args.t_id
print(f'Function get_data. e_id: {e_id}. t_id: {t_id}')
def get_data_2(args):
e_id = args.e_id
t_id = args.t_id
print(f'Function get_data_2. e_id: {e_id}. t_id: {t_id}')
def get_data_3(args):
t_id = args.t_id
print(f'Function get_data_3. t_id: {t_id}')
if __name__ == '__main__':
# Create the arguments parser
argparser = argparse.ArgumentParser()
# Create the subparsers
subparsers = argparser.add_subparsers()
# Add a parser for the first function
get_data_parser = subparsers.add_parser('get_data')
# Set the function name
get_data_parser.set_defaults(func=get_data)
# Add its arguments
get_data_parser.add_argument('e_id')
get_data_parser.add_argument('t_id')
# Add a parser for the second function
get_data_2_parser = subparsers.add_parser('get_data_2')
# Set the function name
get_data_2_parser.set_defaults(func=get_data_2)
# Add its arguments
get_data_2_parser.add_argument('e_id')
get_data_2_parser.add_argument('t_id')
# Add a parser for the third function
get_data_3_parser = subparsers.add_parser('get_data_3')
# Set the function name
get_data_3_parser.set_defaults(func=get_data_3)
# Add its arguments
get_data_3_parser.add_argument('t_id')
# Get the arguments from the comand line
args = argparser.parse_args()
# Call the selected function
args.func(args)
As showed in this example, you will have to change your functions a little bit:
They will take only one argument called args (created in the main function with args = argparser.parse_args())
And then, in each function, you will get the needed parameters by their names (the ones added with add_argument, like in get_data_3_parser.add_argument('t_id'). So, to get the argument called t_id, you will write t_id = args.t_id.
argparse documentation: https://docs.python.org/3/library/argparse.html
argparse tutorial: https://docs.python.org/3/howto/argparse.html
From the documentation:
Many programs split up their functionality into a number of
sub-commands, for example, the svn program can invoke sub-commands
like svn checkout, svn update, and svn commit. Splitting up
functionality this way can be a particularly good idea when a program
performs several different functions which require different kinds of
command-line arguments. ArgumentParser supports the creation of such
sub-commands with the add_subparsers() method.
The below paste contains relevant snippets from three separate Python files. The first is a script called from the command line which instantiates CIPuller given certain arguments. What happens is that the script gets called with something like:
script.py ci (other args to be swallowed by argparse).
The second is part of a subclass called Puller. The third is part of a subclass of Puller called CIPuller.
This works wonderfully, as the correct subclass is called, and any user using the wrong other args gets to see the correct args for their given subclass, plus the generic arguments from the superclass. (Although I was made aware offline that perhaps I should use argparse sub-commands for this.)
I'm stuck trying to write tests for these classes. Currently, I need an ArgumentParser to instantiate the classes, but in testing I'm not instantiating things from the command line, hence my ArgumentParser is useless.
I tried creating an ArgumentParser in the test harness to pass to CIPuller's constructor in the test code, but if I use add_argument there, argparse understandably complains about double (duplicate) arguments when it calls add_argument in the CIPuller constructor.
What would be a suitable design to test these classes with arguments?
#!/usr/bin/env python
from ci_puller import CIPuller
import argparse
import sys
# Using sys.argv[1] for the argument here, as we don't want to pass that onto
# the subclasses, which should receive a vanilla ArgumentParser
puller_type = sys.argv.pop(1)
parser = argparse.ArgumentParser(
description='Throw data into Elasticsearch.'
)
if puller_type == 'ci':
puller = CIPuller(parser, 'single')
else:
raise ValueError("First parameter must be a supported puller. Exiting.")
puller.run()
class Puller(object):
def __init__(self, parser, insert_type):
self.add_arguments(parser)
self.args = parser.parse_args()
self.insert_type = insert_type
def add_arguments(self,parser):
parser.add_argument(
"-d", "--debug",
help="print debug info to stdout",
action="store_true"
)
parser.add_argument(
"--dontsend",
help="don't actually send anything to Elasticsearch",
action="store_true"
)
parser.add_argument(
"--host",
help="override the default host that the data is sent to",
action='store',
default='kibana.munged.tld'
)
class CIPuller(Puller):
def __init__(self, parser, insert_type):
self.add_arguments(parser)
self.index_prefix = "code"
self.doc_type = "cirun"
self.build_url = ""
self.json_url = ""
self.result = []
super(CIPuller, self).__init__(parser, insert_type)
def add_arguments(self, parser):
parser.add_argument(
'--buildnumber',
help='CI build number',
action='store',
required=True
)
parser.add_argument(
'--testtype',
help='Job type per CI e.g. minitest / feature',
choices=['minitest', 'feature'],
required=True
)
parser.add_argument(
'--app',
help='App e.g. sapi / stats',
choices=['sapi', 'stats'],
required=True
)
Unittesting for argparse is tricky. There is a test/test_argparse.py file that is run as part of the overall Python unittest. But it has a complicated custom testing harness to handle most cases.
There are three basic issues, 1) calling parse_args with test values, 2) testing the resulting args, 3) testing for errors.
Testing the resulting args is relatively easy. And the argparse.Namespace class has simple __eq__ method so you can test one namespace against another.
There are two ways of testing inputs. One is to modify the sys.argv. Initially sys.argv has strings meant for the tester.
self.args = parser.parse_args()
tests sys.argv[1:] as a default. So if you change sys.argv you can test custom values.
But you can also give parse_args a custom list. The argparse docs uses this in most of its examples.
self.args = parser.parse_args(argv=myargv)
If myarg is None it uses sys.argv[1:]. Otherwise it uses that custom list.
Testing errors requires either a custom parse.error method (see docs) or wrapping the parse_args in a try/except block that can catch a sys.exit exception.
How do you write tests for the argparse portion of a python module?
python unittest for argparse
Argparse unit tests: Suppress the help message
Unittest with command-line arguments
Using unittest to test argparse - exit errors
I want to use nose and coverage in my project. When I run nose with --with-coverage argument, my programs argument-parsing module goes nuts because "--with-coverage" isn't a real argument according to it.
How do I turn the argparse off, but during testing only? Nose says all my tests fail because of the bad argument.
I actually just ran into this issue myself the other day. You don't need to "disable" your parsing module or anything. What you can do is change the module that uses argparse to ignore those arguments it receives that it doesn't recognize. That way they can still be used by other scripts (for example if your command-line call passes secondary arguments to another program execution).
Without your code, I'll assume you're using the standard parse_args() method on your argparse.ArgumentParser instance. Replace it with parse_known_args() instead.
Then, whenever you subsequently reference the parsed-arguments Namespace object, you'll need to specify and element, specifically 0. While parse_args() returns the args object alone, parse_known_args() returns tuple: the first element is the parsed known arguments, and the latter element contains the ignored unrecognized arguments (which you can later use/pass in your Python code, if necessary).
Here's the example change from my own project:
class RunArgs(object):
'''
A placeholder for processing arguments passed to program execution.
'''
def __init__(self):
self.getArgs()
#self.pause = self.args.pause # old assignment
self.pause = self.args[0].pause # new assignment
#...
def __repr__(self):
return "<RunArgs(t=%s, #=%s, v=%s)>" % (str(x) for x in (self.pause,self.numreads,self.verbose))
def getArgs(self):
global PAUSE_TIME
global NUM_READS
parser = argparse.ArgumentParser()
parser.add_argument('-p', '--pause', required=False,
type=self.checkPauseArg, action='store', default=PAUSE_TIME)
parser.add_argument('-n', '--numreads', required=False,
type=self.checkNumArg, action='store', default=NUM_READS)
parser.add_argument('-v', '--verbose', required=False,
action='store_true')
#self.args = parser.parse_args() # old parse call
self.args = parser.parse_known_args() # new parse call
#...
I've read that you can use nose-testconfig, or otherwise use mock to replace the call (not test it). Though I'd agree with #Ned Batchelder, it begs questioning the structure of the problem.
As a workaround, instead of running nose with command-line arguments, you can have a .noserc or nose.cfg in the current working directory:
[nosetests]
verbosity=3
with-coverage=1
Though, I agree that parse_known_args() is a better solution.
It sounds like you have tests that run your code, and then your code uses argparse which implicitly pulls arguments from sys.argv. This is a bad way to structure your code. Your code under test should be getting arguments passed to it some other way so that you can control what arguments it sees.
This is an example of why global variables are bad. sys.argv is a global, shared by the entire process. You've limited the modularity, and therefore the testability, of your code by relying on that global.