Does argparse provide built-in facilities for having it parse groups or parsers into their own namespaces? I feel like I must be missing an option somewhere.
Edit: This example is probably not exactly what I should be doing to structure the parser to meet my goal, but it was what I worked out so far. My specific goal is to be able to give subparsers groups of options that are parsed into namespace fields. The idea I had with parent was simply to use common options for this same purpose.
Example:
import argparse
# Main parser
main_parser = argparse.ArgumentParser()
main_parser.add_argument("-common")
# filter parser
filter_parser = argparse.ArgumentParser(add_help=False)
filter_parser.add_argument("-filter1")
filter_parser.add_argument("-filter2")
# sub commands
subparsers = main_parser.add_subparsers(help='sub-command help')
parser_a = subparsers.add_parser('command_a', help="command_a help", parents=[filter_parser])
parser_a.add_argument("-foo")
parser_a.add_argument("-bar")
parser_b = subparsers.add_parser('command_b', help="command_b help", parents=[filter_parser])
parser_b.add_argument("-biz")
parser_b.add_argument("-baz")
# parse
namespace = main_parser.parse_args()
print namespace
This is what I get, obviously:
$ python test.py command_a -foo bar -filter1 val
Namespace(bar=None, common=None, filter1='val', filter2=None, foo='bar')
But this is what I am really after:
Namespace(bar=None, common=None, foo='bar',
filter=Namespace(filter1='val', filter2=None))
And then even more groups of options already parsed into namespaces:
Namespace(common=None,
foo='bar', bar=None,
filter=Namespace(filter1='val', filter2=None),
anotherGroup=Namespace(bazers='val'),
anotherGroup2=Namespace(fooers='val'),
)
I've found a related question here but it involves some custom parsing and seems to only covers a really specific circumstance.
Is there an option somewhere to tell argparse to parse certain groups into namespaced fields?
If the focus is on just putting selected arguments in their own namespace, and the use of subparsers (and parents) is incidental to the issue, this custom action might do the trick.
class GroupedAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
group,dest = self.dest.split('.',2)
groupspace = getattr(namespace, group, argparse.Namespace())
setattr(groupspace, dest, values)
setattr(namespace, group, groupspace)
There are various ways of specifying the group name. It could be passed as an argument when defining the Action. It could be added as parameter. Here I chose to parse it from the dest (so namespace.filter.filter1 can get the value of filter.filter1.
# Main parser
main_parser = argparse.ArgumentParser()
main_parser.add_argument("-common")
filter_parser = argparse.ArgumentParser(add_help=False)
filter_parser.add_argument("--filter1", action=GroupedAction, dest='filter.filter1', default=argparse.SUPPRESS)
filter_parser.add_argument("--filter2", action=GroupedAction, dest='filter.filter2', default=argparse.SUPPRESS)
subparsers = main_parser.add_subparsers(help='sub-command help')
parser_a = subparsers.add_parser('command_a', help="command_a help", parents=[filter_parser])
parser_a.add_argument("--foo")
parser_a.add_argument("--bar")
parser_a.add_argument("--bazers", action=GroupedAction, dest='anotherGroup.bazers', default=argparse.SUPPRESS)
...
namespace = main_parser.parse_args()
print namespace
I had to add default=argparse.SUPPRESS so a bazers=None entry does not appear in the main namespace.
Result:
>>> python PROG command_a --foo bar --filter1 val --bazers val
Namespace(anotherGroup=Namespace(bazers='val'),
bar=None, common=None,
filter=Namespace(filter1='val'),
foo='bar')
If you need default entries in the nested namespaces, you could define the namespace before hand:
filter_namespace = argparse.Namespace(filter1=None, filter2=None)
namespace = argparse.Namespace(filter=filter_namespace)
namespace = main_parser.parse_args(namespace=namespace)
result as before, except for:
filter=Namespace(filter1='val', filter2=None)
I'm not entirely sure what you're asking, but I think what you want is for an argument group or sub-command to put its arguments into a sub-namespace.
As far as I know, argparse does not do this out of the box. But it really isn't hard to do by postprocessing the result, as long as you're willing to dig under the covers a bit. (I'm guessing it's even easier to do it by subclassing ArgumentParser, but you explicitly said you don't want to do that, so I didn't try that.)
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
breakfast = parser.add_argument_group('breakfast')
breakfast.add_argument('--spam')
breakfast.add_argument('--eggs')
args = parser.parse_args()
Now, the list of all destinations for breakfast options is:
[action.dest for action in breakfast._group_actions]
And the key-value pairs in args is:
args._get_kwargs()
So, all we have to to is move the ones that match. It'll be a little easier if we construct dictionaries to create the namespaces from:
breakfast_options = [action.dest for action in breakfast._group_actions]
top_names = {name: value for (name, value) in args._get_kwargs()
if name not in breakfast_options}
breakfast_names = {name: value for (name, value) in args._get_kwargs()
if name in breakfast_options}
top_names['breakfast'] = argparse.Namespace(**breakfast_names)
top_namespace = argparse.Namespace(**top_names)
And that's it; top_namespace looks like:
Namespace(breakfast=Namespace(eggs=None, spam='7'), foo='bar')
Of course in this case, we've got one static group. What if you wanted a more general solution? Easy. parser._action_groups is a list of all groups, but the first two are the global positional and keyword groups. So, just iterate over parser._action_groups[2:], and do the same thing for each that you did for breakfast above.
What about sub-commands instead of groups? Similar, but the details are different. If you've kept around each subparser object, it's just whole other ArgumentParser. If not, but you did keep the subparsers object, it's a special type of Action, whose choices is a dict whose keys are the subparser names and whose values are the subparsers themselves. If you kept neither… start at parser._subparsers and figure it out from there.
At any rate, once you know how to find the names you want to move and where you want to move them, it's the same as with groups.
If you've got, in addition to global args and/or groups and subparser-specific args and/or groups, some groups that are shared by multiple subparsers… then conceptually it gets tricky, because each subparser ends up with references to the same group, and you can't move it to al of them. But fortunately, you're only dealing with exactly one subparser (or none), so you can just ignore the other subparsers and move any shared group under the selected subparser (and any group that doesn't exist in the selected subparser, either leave at the top, or throw away, or pick one subparser arbitrarily).
Nesting with Action subclasses is fine for one type of Action, but is a nuisance if you need to subclass several types (store, store true, append, etc). Here's another idea - subclass Namespace. Do the same sort of name split and setattr, but do it in the Namespace rather than the Action. Then just create an instance of the new class, and pass it to parse_args.
class Nestedspace(argparse.Namespace):
def __setattr__(self, name, value):
if '.' in name:
group,name = name.split('.',1)
ns = getattr(self, group, Nestedspace())
setattr(ns, name, value)
self.__dict__[group] = ns
else:
self.__dict__[name] = value
p = argparse.ArgumentParser()
p.add_argument('--foo')
p.add_argument('--bar', dest='test.bar')
print(p.parse_args('--foo test --bar baz'.split()))
ns = Nestedspace()
print(p.parse_args('--foo test --bar baz'.split(), ns))
p.add_argument('--deep', dest='test.doo.deep')
args = p.parse_args('--foo test --bar baz --deep doodod'.split(), Nestedspace())
print(args)
print(args.test.doo)
print(args.test.doo.deep)
producing:
Namespace(foo='test', test.bar='baz')
Nestedspace(foo='test', test=Nestedspace(bar='baz'))
Nestedspace(foo='test', test=Nestedspace(bar='baz', doo=Nestedspace(deep='doodod')))
Nestedspace(deep='doodod')
doodod
The __getattr__ for this namespace (needed for actions like count and append) could be:
def __getattr__(self, name):
if '.' in name:
group,name = name.split('.',1)
try:
ns = self.__dict__[group]
except KeyError:
raise AttributeError
return getattr(ns, name)
else:
raise AttributeError
I've proposed several other options, but like this the best. It puts the storage details where they belong, in the Namespace, not the parser.
In this script I have modified the __call__ method of the argparse._SubParsersAction. Instead of passing the namespace on to the subparser, it passes a new one. It then adds that to the main namespace. I only change 3 lines of __call__.
import argparse
def mycall(self, parser, namespace, values, option_string=None):
parser_name = values[0]
arg_strings = values[1:]
# set the parser name if requested
if self.dest is not argparse.SUPPRESS:
setattr(namespace, self.dest, parser_name)
# select the parser
try:
parser = self._name_parser_map[parser_name]
except KeyError:
args = {'parser_name': parser_name,
'choices': ', '.join(self._name_parser_map)}
msg = _('unknown parser %(parser_name)r (choices: %(choices)s)') % args
raise argparse.ArgumentError(self, msg)
# CHANGES
# parse all the remaining options into a new namespace
# store any unrecognized options on the main namespace, so that the top
# level parser can decide what to do with them
newspace = argparse.Namespace()
newspace, arg_strings = parser.parse_known_args(arg_strings, newspace)
setattr(namespace, 'subspace', newspace) # is there a better 'dest'?
if arg_strings:
vars(namespace).setdefault(argparse._UNRECOGNIZED_ARGS_ATTR, [])
getattr(namespace, argparse._UNRECOGNIZED_ARGS_ATTR).extend(arg_strings)
argparse._SubParsersAction.__call__ = mycall
# Main parser
main_parser = argparse.ArgumentParser()
main_parser.add_argument("--common")
# sub commands
subparsers = main_parser.add_subparsers(dest='command')
parser_a = subparsers.add_parser('command_a')
parser_a.add_argument("--foo")
parser_a.add_argument("--bar")
parser_b = subparsers.add_parser('command_b')
parser_b.add_argument("--biz")
parser_b.add_argument("--baz")
# parse
input = 'command_a --foo bar --bar val --filter extra'.split()
namespace = main_parser.parse_known_args(input)
print namespace
input = '--common test command_b --biz bar --baz val'.split()
namespace = main_parser.parse_args(input)
print namespace
This produces:
(Namespace(command='command_a', common=None,
subspace=Namespace(bar='val', foo='bar')),
['--filter', 'extra'])
Namespace(command='command_b', common='test',
subspace=Namespace(baz='val', biz='bar'))
I used parse_known_args to test how extra strings are passed back to the main parser.
I dropped the parents stuff because it does not add anything to this namespace change. it is just a convenient way of defining a set of arguments that several subparsers use. argparse does not keep a record of which arguments were added via parents, and which were added directly. It is not a grouping tool
argument_groups don't help much either. They are used by the Help formatter, but not by parse_args.
I could subclass _SubParsersAction (instead of reassigning __call__), but then I'd have change the main_parse.register.
Starting from abarnert's answer, I put together the following MWE++ ;-) that handles multiple configuration groups with similar option names.
#!/usr/bin/env python2
import argparse, re
cmdl_skel = {
'description' : 'An example of multi-level argparse usage.',
'opts' : {
'--foo' : {
'type' : int,
'default' : 0,
'help' : 'foo help main',
},
'--bar' : {
'type' : str,
'default' : 'quux',
'help' : 'bar help main',
},
},
# Assume your program uses sub-programs with their options. Argparse will
# first digest *all* defs, so opts with the same name across groups are
# forbidden. The trick is to use the module name (=> group.title) as
# pseudo namespace which is stripped off at group parsing
'groups' : [
{ 'module' : 'mod1',
'description' : 'mod1 description',
'opts' : {
'--mod1-foo, --mod1.foo' : {
'type' : int,
'default' : 0,
'help' : 'foo help for mod1'
},
},
},
{ 'module' : 'mod2',
'description' : 'mod2 description',
'opts' : {
'--mod2-foo, --mod2.foo' : {
'type' : int,
'default' : 1,
'help' : 'foo help for mod2'
},
},
},
],
'args' : {
'arg1' : {
'type' : str,
'help' : 'arg1 help',
},
'arg2' : {
'type' : str,
'help' : 'arg2 help',
},
}
}
def parse_args ():
def _parse_group (parser, opt, **optd):
# digest variants
optv = re.split('\s*,\s*', opt)
# this may rise exceptions...
parser.add_argument(*optv, **optd)
errors = {}
parser = argparse.ArgumentParser(description=cmdl_skel['description'],
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
# it'd be nice to loop in a single run over zipped lists, but they have
# different lenghts...
for opt in cmdl_skel['opts'].keys():
_parse_group(parser, opt, **cmdl_skel['opts'][opt])
for arg in cmdl_skel['args'].keys():
_parse_group(parser, arg, **cmdl_skel['args'][arg])
for grp in cmdl_skel['groups']:
group = parser.add_argument_group(grp['module'], grp['description'])
for mopt in grp['opts'].keys():
_parse_group(group, mopt, **grp['opts'][mopt])
args = parser.parse_args()
all_group_opts = []
all_group_names = {}
for group in parser._action_groups[2:]:
gtitle = group.title
group_opts = [action.dest for action in group._group_actions]
all_group_opts += group_opts
group_names = {
# remove the leading pseudo-namespace
re.sub("^%s_" % gtitle, '', name) : value
for (name, value) in args._get_kwargs()
if name in group_opts
}
# build group namespace
all_group_names[gtitle] = argparse.Namespace(**group_names)
# rebuild top namespace
top_names = {
name: value for (name, value) in args._get_kwargs()
if name not in all_group_opts
}
top_names.update(**all_group_names)
top_namespace = argparse.Namespace(**top_names)
return top_namespace
def main():
args = parse_args()
print(str(args))
print(args.bar)
print(args.mod1.foo)
if __name__ == '__main__':
main()
Then you can call it like this (mnemonic: --mod1-... are options for "mod1", etc.):
$ ./argparse_example.py one two --bar=three --mod1-foo=11231 --mod2.foo=46546
Namespace(arg1='one', arg2='two', bar='three', foo=0, mod1=Namespace(foo=11231), mod2=Namespace(foo=46546))
three
11231
Based on the answer by #abarnert, I wrote a simple function that does what the OP wants:
from argparse import Namespace, ArgumentParser
def parse_args(parser):
assert isinstance(parser, ArgumentParser)
args = parser.parse_args()
# the first two argument groups are 'positional_arguments' and 'optional_arguments'
pos_group, optional_group = parser._action_groups[0], parser._action_groups[1]
args_dict = args._get_kwargs()
pos_optional_arg_names = [arg.dest for arg in pos_group._group_actions] + [arg.dest for arg in optional_group._group_actions]
pos_optional_args = {name: value for name, value in args_dict if name in pos_optional_arg_names}
other_group_args = dict()
# If there are additional argument groups, add them as nested namespaces
if len(parser._action_groups) > 2:
for group in parser._action_groups[2:]:
group_arg_names = [arg.dest for arg in group._group_actions]
other_group_args[group.title] = Namespace(**{name: value for name, value in args_dict if name in group_arg_names})
# combine the positiona/optional args and the group args
combined_args = pos_optional_args
combined_args.update(other_group_args)
return Namespace(**combined_args)
You just give it the ArgumentParser instance and it returns a nested NameSpace according to the group structure of the arguments.
Please check out the argpext module on PyPi, it may help you!
Related
I normally use Python for scripts but I am trying to write a unit test and am having a lot of issues. I would like to test a method that creates a parameter --users. The value is how many occurred.
count_users(df, args.metrics)
It is a spark dataframe and the metrics are set like so:
if __name__ == "__main__":
parser = argparse.ArgumentParser("Processing args")
parser.add_argument("--metrics", required=True)
main(parser.parse_args())
The method looks like this:
def count_users(df, metrics):
users = df.where(df.users > 0).count()
temp_df = df.withColumn("user_count_values", F.lit(users))
temp_df.write.json(metrics)
Now I am trying to write my test, and this is where I am not sure about:
def test_count_users(self):
df = (
SparkSession.builder.appName("test")
.getOrCreate()
.createDataFrame(
data=[
(Decimal(0),),
(Decimal(22),),
],
schema=StructType(
[
StructField("users", DecimalType(38, 4), True),
]
),
)
)
ap = argparse.ArgumentParser("Test args")
ap.add_argument("metrics")
args = {_.dest: _ for _ in ap._actions if isinstance(_, _StoreAction)}
assert args.keys() == {"metrics"}
count_users(df, args.metrics)
self.assertTrue(args["metrics"], 1)
Right now I get an error that reads
count_users(df, args.metrics)
AttributeError: 'dict' object has no attribute 'metrics'
It's unclear what you are trying to achieve with the args = { ... line, or the two asserts. Remove them, use something standard like
import argparse
parser = argparse.ArgumentParser("Test args")
parser.add_argument("--metrics", required=True)
args = parser.parse_args(["--metrics", "output.json"])
count_users(df, args.metrics)
Your args variable won't have the appropriate attribute(s) until you parse the arguments. Of course, normally you'd call
args = parser.parse_args()
and let the user provide the --metrics outputfilename.json arguments to the script. The above is more for example or test use cases.
I wasn't sure how to search for this but I am trying to make a script that dynamically launches programs. I will have a couple of JSON files and I want to be able to do a search replace sort of thing.
So I'll setup an example:
config.json
{
"global_vars": {
"BASEDIR": "/app",
"CONFIG_DIR": "{BASEDIR}/config",
"LOG_DIR": "{BASEDIR}/log",
"CONFIG_ARCHIVE_DIR": "{CONFIG_DIR}/archive"
}
}
Then process.json
{
"name": "Dummy_Process",
"binary": "java",
"executable": "DummyProcess-0.1.0.jar",
"launch_args": "-Dspring.config.location={CONFIG_DIR}/application.yml -Dlogging.config={CONFIG_DIR}/logback-spring.xml -jar {executable}",
"startup_log": "{LOG_DIR}/startup_{name}.out"
}
Now I want to be able to load both of these JSON objects and be able to use the values there to update. So like "CONFIG_ARCHIVE_DIR": "{CONFIG_DIR}/archive" will become CONFIG_ARCHIVE_DIR": "/app/config/archive"
Does anyone know a good way to do this recursively because I'm running into issues when I'm trying to use something like CONFIG_DIR which requires BASEDIR first.
I have this function that loads all the data:
#Recursive function, loops and loads all values into data
def _load_data(data,obj):
for i in obj.keys():
if isinstance(obj[i],str):
data[i]=obj[i]
if isinstance(obj[i],dict):
data=_load_data(data,obj[i])
return data
Then I have this function:
def _update_data(data,data_str=""):
if not data_str:
data_str=json.dumps(data)
for i in data.keys():
if isinstance(data[i],str):
data_str=data_str.replace("{"+i+"}",data[i])
if isinstance(data[i],dict):
data=_update_data(data,data_str)
return json.loads(data_str)
So this works for one level but I don't know if this is the best way to do it. It stops working when I hit a case like the CONFIG_DIR because it would need to loop over the data multiple times. First it needs to update the BASEDIR then once more to update CONFIG_DIR. suggestion welcome.
The end goal of this script is to create a start/stop/status script to manage all of our binaries. They all use different binaries to start and I want one Processes file for multiple servers. Each process will have a servers array to tell the start/stop script what to run on given server. Maybe there's something like this already out there so if there is, please point me in the direction.
I will be running on Linux and prefer to use Python. I want something smart and easy for someone else to pickup and use/modify.
I made something that works with the example files you provided. Note that I didn't handle multiple keys or non-dictionaries in the data. This function accepts a list of the dictionaries obtained after JSON parsing your input files. It uses the fact that re.sub can accept a function for the replacement value and calls that function with each match. I am sure there are plenty of improvements that could be made to this, but it should get you started at least.
def make_config(configs):
replacements = {}
def find_defs(config):
# Find leaf nodes of the dictionary.
defs = {}
for k, v in config.items():
if isinstance(v, dict):
# Nested dictionary so recurse.
defs.update(find_defs(v))
else:
defs[k] = v
return defs
for config in configs:
replacements.update(find_defs(config))
def make_replacement(m):
# Construct the replacement string.
name = m.group(0).strip('{}')
if name in replacements:
# Replace replacement strings in the replacement string.
new = re.sub('\{[^}]+\}', make_replacement, replacements[name])
# Cache result
replacements[name] = new
return new
raise Exception('Replacement string for {} not found'.format(name))
finalconfig = {}
for name, value in replacements.items():
finalconfig[name] = re.sub('\{[^}]+\}', make_replacement, value)
return finalconfig
With this input:
[
{
"global_vars": {
"BASEDIR": "/app",
"CONFIG_DIR": "{BASEDIR}/config",
"LOG_DIR": "{BASEDIR}/log",
"CONFIG_ARCHIVE_DIR": "{CONFIG_DIR}/archive"
}
},
{
"name": "Dummy_Process",
"binary": "java",
"executable": "DummyProcess-0.1.0.jar",
"launch_args": "-Dspring.config.location={CONFIG_DIR}/application.yml -Dlogging.config={CONFIG_DIR}/logback-spring.xml -jar {executable}",
"startup_log": "{LOG_DIR}/startup_{name}.out"
}
]
It gives this output:
{
'BASEDIR': '/app',
'CONFIG_ARCHIVE_DIR': '/app/config/archive',
'CONFIG_DIR': '/app/config',
'LOG_DIR': '/app/log',
'binary': 'java',
'executable': 'DummyProcess-0.1.0.jar',
'launch_args': '-Dspring.config.location=/app/config/application.yml -Dlogging.config=/app/config/logback-spring.xml -jar DummyProcess-0.1.0.jar',
'name': 'Dummy_Process',
'startup_log': '/app/log/startup_Dummy_Process.out'
}
As an alternative to the answer by #FamousJameous and if you don't mind changing to ini format, you can also use the python built-in configparser which already has support to expand variables.
I implemented a solution with a class (Config) with a couple of functions:
_load: simply convert from JSON to a Python object;
_extract_params: loop over the document (output of _load) and add them to a class object (self.params);
_loop: loop over the object returned from _extract_params and, if the values contains any {param}, call the _transform method;
_transform: replace the {param} in the values with the correct values, if there is any '{' in the value linked to the param that needs to be replaced, call again the function
I hope I was clear enough, here is the code:
import json
import re
config = """{
"global_vars": {
"BASEDIR": "/app",
"CONFIG_DIR": "{BASEDIR}/config",
"LOG_DIR": "{BASEDIR}/log",
"CONFIG_ARCHIVE_DIR": "{CONFIG_DIR}/archive"
}
}"""
process = """{
"name": "Dummy_Process",
"binary": "java",
"executable": "DummyProcess-0.1.0.jar",
"launch_args": "-Dspring.config.location={CONFIG_DIR}/application.yml -Dlogging.config={CONFIG_DIR}/logback-spring.xml -jar {executable}",
"startup_log": "{LOG_DIR}/startup_{name}.out"
}
"""
class Config(object):
def __init__(self, documents):
self.documents = documents
self.params = {}
self.output = {}
# Loads JSON to dictionary
def _load(self, document):
obj = json.loads(document)
return obj
# Extracts the config parameters in a dictionary
def _extract_params(self, document):
for k, v in document.items():
if isinstance(v, dict):
# Recursion for inner dictionaries
self._extract_params(v)
else:
# if not a dict set params[k] as v
self.params[k] = v
return self.params
# Loop on the configs dictionary
def _loop(self, params):
for key, value in params.items():
# if there is any parameter inside the value
if len(re.findall(r'{([^}]*)\}', value)) > 0:
findings = re.findall(r'{([^}]*)\}', value)
# call the transform function
self._transform(params, key, findings)
return self.output
# Replace all the findings with the correct value
def _transform(self, object, key, findings):
# Iterate over the found params
for finding in findings:
# if { -> recursion to set all the needed values right
if '{' in object[finding]:
self._transform(object, finding, re.findall(r'{([^}]*)\}', object[finding]))
# Do de actual replace
object[key] = object[key].replace('{'+finding+'}', object[finding])
self.output = object
return self.output
# Entry point
def process_document(self):
params = {}
# _load the documents and extract the params
for document in self.documents:
params.update(self._extract_params(self._load(document)))
# _loop over the params
return self._loop(params)
# return self.output
if __name__ == '__main__':
config = Config([config, process])
print(config.process_document())
I am sure there are many other better ways to reach your goal, but I still hope this can bu useful to you.
I want to receive a dict(str -> str) argument from the command line. Does argparse.ArgumentParser provide it? Or any other library?
For the command line:
program.py --dict d --key key1 --value val1 --key key2 --value val2
I expect the following dictionary:
d = {"key1": "val1", "key2": "val2"}
Here's another solution using a custom action, if you want to specify dict key pairs together comma-separated --
import argparse
import sys
parser = argparse.ArgumentParser(description='parse key pairs into a dictionary')
class StoreDictKeyPair(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
my_dict = {}
for kv in values.split(","):
k,v = kv.split("=")
my_dict[k] = v
setattr(namespace, self.dest, my_dict)
parser.add_argument("--key_pairs", dest="my_dict", action=StoreDictKeyPair, metavar="KEY1=VAL1,KEY2=VAL2...")
args = parser.parse_args(sys.argv[1:])
print args
Running:
python parse_kv.py --key_pairs 1=2,a=bbb,c=4 --key_pairs test=7,foo=bar
Output:
Namespace(my_dict={'1': '2', 'a': 'bbb', 'c': '4', 'test': '7', 'foo': 'bar'})
If you want to use nargs instead of comma-separated values in string:
class StoreDictKeyPair(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, **kwargs):
self._nargs = nargs
super(StoreDictKeyPair, self).__init__(option_strings, dest, nargs=nargs, **kwargs)
def __call__(self, parser, namespace, values, option_string=None):
my_dict = {}
print "values: {}".format(values)
for kv in values:
k,v = kv.split("=")
my_dict[k] = v
setattr(namespace, self.dest, my_dict)
parser.add_argument("--key_pairs", dest="my_dict", action=StoreDictKeyPair, nargs="+", metavar="KEY=VAL")
args = parser.parse_args(sys.argv[1:])
print args
Running
python arg_test4.py --key_pairs 1=2 a=bbb c=4 test=7 foo=bar
Outputs:
values: ['1=2', 'a=bbb', 'c=4', 'test=7', 'foo=bar']
Namespace(my_dict={'1': '2', 'a': 'bbb', 'c': '4', 'test': '7', 'foo': 'bar'})
I would use something like this:
p = argparse.ArgumentParser()
p.add_argument("--keyvalue", action='append',
type=lambda kv: kv.split("="), dest='keyvalues')
args = p.parse_args("--keyvalue foo=6 --keyvalue bar=baz".split())
d = dict(args.keyvalues)
You could create a custom action which would "append" a parsed key-value pair directly into a dictionary, rather than simply accumulating a list of (key, value) tuples. (Which I see is what skyline75489 did; my answer differs in using a single --keyvalue option with a custom type instead of separate --key and --value options to specify pairs.)
just another easy way:
parser = argparse.ArgumentParser()
parser.add_argument('--key1')
parser.add_argument('--key2')
args = parser.parse_args()
my_dict = args.__dict__
use vars
d = vars(parser.parse_args())
Python receives arguments in the form of an array argv. You can use this to create the dictionary in the program itself.
import sys
my_dict = {}
for arg in sys.argv[1:]:
key, val=arg.split(':')[0], arg.split(':')[1]
my_dict[key]=val
print my_dict
For command line:
python program.py key1:val1 key2:val2 key3:val3
Output:
my_dict = {'key3': 'val3', 'key2': 'val2', 'key1': 'val1'}
Note: args will be in string, so you will have to convert them to store numeric values.
I hope it helps.
Python one-line argparse dictionary arguments argparse_dictionary.py
# $ python argparse_dictionary.py --arg_dict=1=11,2=22;3=33 --arg_dict=a=,b,c=cc,=dd,=ee=,
# Namespace(arg_dict={'1': '11', '2': '22', '3': '33', 'a': '', 'c': 'cc', '': 'dd'})
import argparse
arg_parser = argparse.ArgumentParser()
arg_parser.add_argument(
'--arg_dict',
action=type(
'', (argparse.Action, ),
dict(__call__=lambda self, parser, namespace, values, option_string: getattr(
namespace, self.dest).update(
dict([
v.split('=') for v in values.replace(';', ',').split(',')
if len(v.split('=')) == 2
])))),
default={},
metavar='KEY1=VAL1,KEY2=VAL2;KEY3=VAL3...',
)
print(arg_parser.parse_args())
A straight forward way of parsing an input like:
program.py --dict d --key key1 --value val1 --key key2 --value val2
is:
parser=argparse.ArgumentParser()
parser.add_argument('--dict')
parser.add_argument('--key', action='append')
parser.add_argument('--value', action='append')
args = parser.parse_args()
which should produce (if my mental parser is correct)
args = Namespace(dict='d', key=['key1','key2'], value=['value1','value2'])
You should be able construct a dictionary from that with:
adict = {k:v for k, v in zip(args.key, args.value)}
Using args.dict to assign this to a variable with that name requires some un-python trickery. The best would be to create a element in another dictionary with this name.
another_dict = {args.dict: adict}
This solution doesn't perform much error checking. For example, it doesn't make sure that there are the same number of keys and values. It also wouldn't let you create multiple dictionaries (i.e. repeated --dict arguments). It doesn't require any special order. --dict could occur after a --key key1 pair. Several --value arguments could be together.
Tying the key=value together as chepner does gets around a number of those problems.
There is a simple solution in Python 3.6 if you're simply trying to convert argparse input to a dictionary. An example is as follows:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-i', '--input', help='the path to the input file')
parser.add_argument('-o', '--output', help='the path to the output file')
args = parser.parse_args()
arguments = dict(args._get_kwargs())
for k, v in arguments.items():
print(k, v)
Given command line input such as python3 script_name.py --input 'input.txt' --output 'output.txt' the code would output to terminal:
input input.txt
output output.txt
As for the current libraries like argparse, docopt and click, none of them support using dict args. The best solution I can think of is to make a custom argparse.Action to support it youself:
import argparse
class MyAction(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, **kwargs):
super(MyAction, self).__init__(option_strings, dest, nargs, **kwargs)
def __call__(self, parser, namespace, values, option_string=None):
print '%r %r %r' % (namespace, values, option_string)
value_dict = {}
values.reverse()
while len(values) > 0:
v = eval(values.pop()).lstrip('--') # This is like crazy hack, I know.
k = eval(values.pop())
value_dict[k] = v
setattr(namespace, self.dest, value_dict)
parser = argparse.ArgumentParser()
parser.add_argument('-d', action=MyAction, nargs='*')
args = parser.parse_args('-d "--key" "key1" "--value" "val1" "--key" "key2" "--value" "val2"'.split())
print(args)
I'd like to know if I can have a parameter list with keywords given inside a string that I can pass into a function? Basically, the parameter list may or may not have keywords, so the parameter list would have variable 'types'. Here's an example of what I'm trying to do:
from bs4 import BeautifulSoup
import urllib.request as urlreq
import my_parameters # can have variable values
# my_parameters.useful_token_concept = ["h1", "class_ = some_class"]
# I want to pass these above parameters into a function; "class_" is
# a keyword, but it's wrapped in a string => gives me problems
url = my_parameters.url
page = urlreq.urlope(url)
pageHtml = page.read()
page.close()
soup = BeautifulSoup(pageHtml)
# something like the following line works:
# params = soup.find("h1", class_ = "some_class")
params = soup.find(*my_parameters.useful_token_concept)
# params = soup.find(my_parameters.useful_token_concept[0],\
# my_parameters.useful_token_concept[1])
# I don't know how long the list of attributes/parameter-list to
# BeautifulSoup's find() function will be, nor do I know what keywords,
# if any, will be passed into find(), as given by a user to my_parameters.
print(params) # should print the html the user wants to scrape.
Why not just use a better representation? I.e, instead of
my_parameters.useful_token_concept = ["h1", "class_ = some_class"]
use
my_parameters.useful_token_concept = ["h1", {"class_": "some_class"}]
Since these values' representation is up to you, using a dict to represent keyword parameters is much simpler than encoding them into a string and then having to parse that string back!
You need to split your token list into a dictionary of keyword arguments, and a list of positional arguments.
kwargs = {}
args = []
for i in my_parameters.useful_token_concept:
bits = i.split('=')
if len(bits) > 1:
kwargs[bits[0].strip()] = bits[1].strip()
else:
args.append(bits[0].strip())
params = soup.find(*args, **kwargs)
You could create a string representation of how all the arguments would be passed and use eval() to turn them into something you could actually use in a real function call:
my_parameters.useful_token_concept = ["h1", "class_ = some_class"]
def func_proxy(*args, **kwargs):
" Just return all positional and keyword arguments. "
return args, kwargs
calling_seq = ', '.join(my_parameters.useful_token_concept)
args, kwargs = eval('func_proxy({})'.format(calling_seq))
print('args:', args) # -> args: (<Header1 object>,)
print('kwargs:', kwargs) # -> kwargs: {'class_': <class '__main__.some_class'>}
parms = soup.find(*args, **kwargs)
We have an api that consumes a around 50 to 100 query params. Currently the handler takes all of the params and sets them as attributes in a Meta object. Something like this
meta = Meta()
meta.param1 = param.get('param1', 'somedefault')
meta.param2 = param.get('param2', 'someotherdefault')
and so on. My question is, is there a better way to handle this than just a loooong list of assigns in the handler? My current idea is to just break it out into a helper function.
meta = self.get_meta(param)
Any other ideas?
(updated my example)
PARAMETERS = [
'param1',
'param2',
# ...
]
meta = Meta()
for name in PARAMETERS:
setattr(meta, name, param[name])
Based on your comment...
DEFAULTS = {
'param1': 1,
'param2': 'something',
}
meta = Meta()
for name, value in DEFAULTS.items():
setattr(meta, name, param.get(name, value))
This seems like something you should do with a mapping instead. Unless you need to filter the parameters, this sounds like a bad idea.
So you'll have something like this:
class Meta(IterableUserDict):
pass
meta = Meta()
meta.update(param)
Expanding on whats been said including default values
PARAMETERS = [
('param1', "default"),
('param2', "default2"),
# ...
]
meta = Meta()
for name, default in PARAMETERS:
setattr(meta, name, param.get(name, default))