I have to run the main file that depends on the rest of a relatively large project. Project structure can be seen as such
main.py
opts.py
--models \
--model1.py
--model2.py
...
--schedulers
--scheduler1.py
--scheduler2.py
...
...
The problem is when I have to pass arguments to each component (using argparse). A simple way would be to define all parameters in a single place for each component. This worked so far for me (in opts.py), but I would like to make something more elegant. In my parse function for each component parse_models or parse_scheduler I would like to iterate through each submodule of models and schedulers and let them define their own arguments by calling a function define_arguments that each of them has where they create their own sub parser.
All in all, how do I iterate through the submodules and call their define_arguments function from within opts.py?
You can iterate over all the python files using the glob module. You can find the correct path with the parent module's __path__ attribute. Import the modules using importlib.import_module. The imported module then contains the define_arguments function that you can pass a parser per submodule to define the arguments on:
from glob import glob
import os, importlib
def load_submodule_parsers(parent_module, parser, help=None):
if help is None:
help = parent_module.__name__ + " modules"
modules = glob(os.path.join(parent_module.__path__, "*.py"))
subparsers = parser.add_subparsers(help=help)
for module_file in modules:
module_name = os.path.basename(module_file)[:-3]
if module == "__init__":
continue
module = importlib.import_module(module_name, package=parent_module.__name__)
if "define_arguments" not in module.__dict__:
raise ImportError(parent_module.__name__ + " submodule '" + module_name + "' does not have required 'define_arguments' function.")
parser = subparsers.add_parser(module_name)
module.define_arguments(parser)
Pass the function the parent module object:
import argparse, models, schedulers
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
models_parser = subparsers.add_parser("models")
load_submodule_parsers(models, models_parser)
schedulers_parser = subparsers.add_parser("schedulers")
load_submodule_parsers(schedulers, schedulers_parser)
Untested code but I think you can refine it from here
Is it allowed to group custom Django commands to separate folders inside the same Django app?
I have a lot of them and wanted to group them logically by purpose. Created folders but Django can't find them.
Maybe I'm trying to run them wrong. Tried:
python manage.py process_A_related_data
the same plus imported all commands in __init__.py
python manage.py folderA process_A_related_data
python manage.py folderA.process_A_related_data
python manage.py folderA/process_A_related_data
Got following error:
Unknown command: 'folderA/process_A_related_data'
Type 'manage.py help' for usage.
I think you can create a basic custom command which will run other commands from relevent folders. Here is an approach you can take:
First make a folder structure like this:
management/
commands/
folder_a/
process_A_related_data.py
folder_b/
process_A_related_data.py
process_data.py
Then inside process_data.py, update the command like this:
from django.core import management
from django.core.management.base import BaseCommand
import importlib
class Command(BaseCommand):
help = 'Folder Process Commands'
def add_arguments(self, parser):
parser.add_argument('-u', '--use', type=str, nargs='?', default='folder_a.process_A_related_data')
def handle(self, *args, **options):
try:
folder_file_module = options['use'] if options['use'].startswith('.') else '.' + options['use']
command = importlib.import_module(folder_file_module, package='your_app.management.commands')
management.call_command(command.Command())
except ModuleNotFoundError:
self.stderr.write(f"No relevent folder found: {e.name}")
Here I am using call_command method to call other managment commands.
Then run commands like this:
python manage.py process_data --use folder_a.process_A_related_data
Finally, if you want to run commands like python manage.py folder_a.process_A_related_data, then probably you need to change in manage.py. Like this:
import re
...
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
if re.search('folder_[a-z].*', sys.argv[-1]):
new_arguments = sys.argv[:-1] + ['process_data','--use', sys.argv[-1]]
execute_from_command_line(new_arguments)
else:
execute_from_command_line(sys.argv)
You should be able to partition the code by using mixins (I have not tried this in this context, though)
A standard management command looks like
from django.core.management.base import BaseCommand
class Command(BaseCommand):
help = 'FIXME A helpful comment goes here'
def add_arguments(self, parser):
parser.add_argument( 'name', ...)
# more argument definitions
def handle(self, *args, **options):
# do stuff
Which can probably be replaced by a "stub" in app/management/commands:
from wherever.commands import FooCommandMixin
from django.core.management.base import BaseCommand
class Command(FooCommandMixin, BaseCommand):
# autogenerated -- do not put any code in here!
pass
and in wherever/commands
class FooCommandMixin( object):
help = 'FIXME A helpful comment goes here'
def add_arguments(self, parser):
parser.add_argument( 'name', ...)
# more argument definitions
def handle(self, *args, **options):
# do the work
It would not be hard to write a script to go through a list of file names or paths (using glob.glob) using re.findall to identify appropriate class declarations, and to (re)generate a matching stub for each in the app's management/commands folder.
Also/instead Python's argparse allows for the definition of sub-commands. So you should be able to define a command that works like
./manage.py foo bar --aa --bb something --cc and
./manage.py foo baz --bazzy a b c
where the syntax after foo is determined by the next word (bar or baz or ...). Again I have no experience of using subcommands in this context.
I found no mention of support for this feature in the release notes. It looks to be that this is still not supported as of version Django 3.0. I would suggest that you use meaningful names for your files that help you specify. You could always come up w/ a naming convention!
A workaround could be: create a specific Django "satellite" app for each group of management commands.
In recent version of Django, the requirements for a Python module to be an app are minimal: you won't need to provide any fake models.py or other specific files as happened in the old days.
While far from perfect from a stylistic point of view, you still gain a few advantages:
no need to hack the framework at all
python manage.py will list the commands grouped by app
you can control the grouping by providing suitable names to the apps
you can use these satellite apps as container for specific unit tests
I always try to avoid fighting against the framework, even when this means to compromise, and sometimes accept it's occasional design limitations.
Hopefully this will translate into an elegant solution, but i am unable to figure it out myself. I have been reading huge amounts of examples and explanations but I cant seem to get it to work.
I am writing a program that needs the following options:
myprogram:
-h help
--DEBUG debugging on
Commands are: dashboard / promote / deploy / auto-tag / dbquery
Some commands have sub-commands.
Dashboard prosesing:
dashboard <ALL | DEBUG | AAA [sub1-aaa | ...] >
Promoting:
promote <environment> [to environment] <system> [version]
Deployment
deploy <system> [version] <in environment>
Auto-tagging
auto-tag <repo> <project> [--days-back]
Database queries (--tab is optional)
(system or env or both are required)
dbquery lock set <system | environment> [--tab]
dbquery lock clear <system | environment> [--tab]
dbquery lock show <system | environment | before-date | after-date> [--tab]
dbquery promote list <system| version| environment | before-date | after-date> [--tab]
dbquery deploy list <system| version| environment | before-date | after-date> [--tab]
If a command is used, some sub-commands or options are required.
I have a hard time getting this done using the argparse library. I tried to use add_argument_group, subparsers etc. But I think I am missing something basic here. All examples i found getting close are about svn, but they seem to only go 1 level after svn. And i need more, or a different approach.
If possible I would like to make all parameters after dbquery deploy list optional, with at least 1 option required. But to distinguish between the name of a system and an environemnt might become tricky so it might be better to change this:
dbquery lock set <system | environment>
into
dbquery lock set <system=system | env=environment>
p.s. Options enclosed between [] are optional, options between <> are required.
Thanks in advance.
In response to the comment of providing my code, lets focus on the dbquery, because the rest might be a repetition:
import argparse
parser = argparse.ArgumentParser(description="Main cli tool for processing in the CD pipeline (%s)" % VERSION)
subparsers = parser.add_subparsers(help='additional help',title='subcommands')
dbq_parser=subparsers.add_parser("dbqueue", aliases=['dbq'])
dbq_group_lock = dbq_parser.add_argument_group('lock', 'lock desc')
dbq_group_promote =dbq_parser.add_argument_group('promote')
dbq_group_deploy = dbq_parser.add_argument_group('deploy','Deployment description')
dbq_group_lock.add_argument('set', help="Sets lock")
dbq_group_lock.add_argument('clear', help='Clears lock')
dbq_group_lock.add_argument('show', help='Show lock status')
dbq_group_deploy.add_argument('name system etc')
Executing results in:
# python3 main.py -h
usage: main.py [-h] [--debug] {dbqueue,dbq} ...
Main cli tool for processing in the CD pipeline (cdv3, Jun 2019, F.IJskes)
optional arguments:
-h, --help show this help message and exit
--debug Generate debug output and keep temp directories
subcommands:
{dbqueue,dbq} additional help
This looks okay, but:
#python3 main.py dbq -h
usage: main.py dbqueue [-h] set clear show name system etc
optional arguments:
-h, --help show this help message and exit
lock:
lock desc
set Sets lock
clear Clears lock
show Show lock status
shows that the expected parameters are not lock, promote or deploy.
Okay, the feedback helped my understanding. I now understand that parsers can get subparsers and those can get parsers. So there might be no limit in the depth one can go.
This new insight got me to this: (a partial copy from my working example)
import argparse
main_parser = argparse.ArgumentParser(description="Main cli tool for processing in the CD pipeline (%s)" % VERSION)
main_subparsers = main_parser.add_subparsers(help='',title='Possible commnds')
dashbrd_subparser = main_subparsers.add_parser('dashboard', help="Proces specified dashboards", allow_abbrev=True)
dashbrd_subparser.add_argument('who?',help='Proces dashboard for ALL, Supplier or DEBUG what should happen.')
dashbrd_subparser.add_argument('-subsystem', help='Select subsystem to proces for given supplier')
dbq_main=main_subparsers.add_parser("dbquery", help="Query database for locks,deployments or promotes")
dbq_main_sub=dbq_main.add_subparsers(help="additions help", title='sub commands for dbquery')
dbq_lock=dbq_main_sub.add_parser('lock', help='query db for lock(s)')
dbq_lock_sub=dbq_lock.add_subparsers(help='', title='subcommands for lock')
dbq_lock_sub_set=dbq_lock_sub.add_parser('set', help='sets a lock')
dbq_lock_sub_set.add_argument('-env', required=True)
dbq_lock_sub_set.add_argument('--tab',required=False, action="store_true" )
dbq_lock_sub_clear=dbq_lock_sub.add_parser('clear', help='clears a lock')
# dbq_lock_sub_set.add_argument('-env', required=True)
# dbq_lock_sub_set.add_argument('--tab', required=False)
dbq_lock_sub_show=dbq_lock_sub.add_parser('show', help='shows a lock/locks')
# dbq_lock_sub_set.add_argument('-env', required=True)
# dbq_lock_sub_set.add_argument('--tab', required=False)
print( vars(main_parser.parse_args()))
exit(1)
I now only seem to have issues using parameters as '-env' and '-subsystem' across the different sub-commands. Because there is a conflict when i add them to another parser.
I also have no data to use about what options where chosen. This is something that is also needed.
Almost there, and what I now have is usable to a great extent, so I'll post this so other users might benefit from my work, which has benefited by the comments of hpaulj which led me to other documents that could clarify other parts.
# see: https://pymotw.com/3/argparse/
# BLOCK FOR SHARED options used at more places,
# see also: https://stackoverflow.com/questions/7498595/python-argparse-add-argument-to-multiple-subparsers
env_shared = argparse.ArgumentParser(add_help=False)
env_shared.add_argument('-env', action="store", help='name of environment <tst|acc|acc2|prod|...>', metavar='<environment>', required=True)
ds_shared = argparse.ArgumentParser(add_help=False)
ds_shared.add_argument('-ds', action="store", help='name of subsystem to use <ivs-gui|ivs-vpo|...>', metavar='<system name>', required=True)
version_shared = argparse.ArgumentParser(add_help=False)
version_shared.add_argument('-version', action="store", help='version to use. If used, specify at least sprint as in 1.61', metavar='<version>')
tab_shared=argparse.ArgumentParser(add_help=False)
tab_shared.add_argument('--tab', required=False, action="store_true", help='Use nice tabulation for output')
# END OF SHARED options
# MAIN
main_p = argparse.ArgumentParser(description="Main cli tool for use in the CD v3 pipeline (%s)" % VERSION, epilog='Defaults are taken from the configuration file, which is pulled from git when needed.', )
# Change formatting of help output for main-parser
main_p.formatter_class = lambda prog: argparse.HelpFormatter(prog, max_help_position=30, width=80)
main_p.add_argument("--debug", action="store_true", help="Generate debug output and keep temp directories")
# MAIN-SUB
main_s_p = main_p.add_subparsers(title='Commands', dest='main_cmd')
# MAIN-SUB-DASHBOARD
dashbrd_p = main_s_p.add_parser('dashboard', aliases=['db'], help="Proces specified dashboards", parents=[ds_shared])
dashbrd_p.formatter_class = lambda prog: argparse.HelpFormatter(prog, max_help_position=40, width=80)
dashbrd_p.add_argument('which', help='Proces dashboard for <ALL|Supplier|DEBUG what would happen>. If supplier is specified an optional subsystem to be processed can be passed using -ds')
# MAIN-SUB-QUERY
query_p = main_s_p.add_parser("query", aliases=['qry'], help="Query database for locks,deployments or promotes")
query_p.formatter_class = lambda prog: argparse.HelpFormatter(prog, max_help_position=40, width=80)
# MAIN-SUB-QUERY-SUB
query_s_p = query_p.add_subparsers(help="additions help", title='sub commands for query', dest='query_cmd')
# MAIN-SUB-QUERY-LOCK
q_lock_p = query_s_p.add_parser('lock', help='query db for lock(s)', )
# MAIN-SUB-QUERY-LOCK-SUB
q_lock_sub = q_lock_p.add_subparsers(help='', title='subcommands for dbquery lock', dest='lock_cmd')
# MAIN-SUB-QUERY-LOCK-SUB-SET
q_lock_set_p = q_lock_sub.add_parser('set', help='sets a lock', parents=[env_shared, ds_shared])
# MAIN-SUB-QUERY-LOCK-SUB-CLEAR
q_lock_clear_p = q_lock_sub.add_parser('clear', help='clears a lock', parents=[env_shared, ds_shared], )
# MAIN-SUB-QUERY-LOCK-SUB-SHOW
q_lock_show_p = q_lock_sub.add_parser('show', help='shows a lock/locks', parents=[env_shared, tab_shared])
# MAIN-SUB-QUERY-PROMOTE
q_promote_p = query_s_p.add_parser('promote', help='query db for promotions', parents=[env_shared,ds_shared] )
# MAIN-SUB-QUERY-DEPLOY
q_deploy_p = query_s_p.add_parser('deploy', help='query db for deployments')
# MAIN-SUB-PROMOTE
promote_p = main_s_p.add_parser('promote', help='Do a promotion.', parents=[env_shared, ds_shared, version_shared])
promote_p.add_argument('-dest', required=False, action="store", help='If used specifies the destination, otherwise defaults are used. (-dest is optional, -env is not)', metavar='<dest_env>')
# MAIN-SUB-DEPLOY
deploy_p = main_s_p.add_parser('deploy', help='Deploy software', parents=[ds_shared, env_shared])
# MAIN-SUB-AUTOTAG
autotag_p = main_s_p.add_parser('autotag', help="Autotags specified repository", parents=[ds_shared])
print("------------arguments-----------")
print(vars(main_p.parse_args()))
This creates a lot of the functionality i intended.
I am using pytest to run tests in multiple environments and I wanted to include that information (ideally) in an ini style config file. I would also like to override parts or all of the configuration at the command line as well. I tried using the hook pytest_addoption in my conftest.py like so:
def pytest_addoption(parser):
parser.addoption("--hostname", action="store", help="The host")
parser.addoption("--port", action="store", help="The port")
#pytest.fixture
def hostname(request):
return request.config.getoption("--hostname")
#pytest.fixture
def port(request):
return request.config.getoption("--port")
Using this I can add the configuration info at the command line, but not in a config file. I also tried adding
[pytest]
addopts = --hostname host --port 311
to my pytest.ini file, but that didn't work. Is there a way to do this without building my own plugin? Thanks for your time.
The parser object does have an addini method as well that you can use to specify configuration options through an ini file.
Here is the documentation for it: https://pytest.org/latest/writing_plugins.html?highlight=addini#_pytest.config.Parser.addini
addini(name, help, type=None, default=None)[source]
registers an ini-file option.
Name: name of the ini-variable
Type: type of the variable, can be pathlist, args, linelist or bool.
Default: default value if no ini-file option exists but is queried.
The value of ini-variables can be retrieved via a call to config.getini(name).
So I'm very noob in dealing with nose plugins.
I've been searching a lot but docs regarding nose plugins seem scarce.
I read and tried what's in the following links to try to write a simple nose plugin
and run it with nosetests, without success:
https://nose.readthedocs.org/en/latest/doc_tests/test_init_plugin/init_plugin.html
https://nose.readthedocs.org/en/latest/plugins/writing.html
I don't want to write my own test-runner or run the tests from any other script (via run(argv=argv, suite=suite(), ...)),
like they do in the first link.
I wrote a file myplugin.py with a class like this:
import os
from nose.plugins import Plugin
class MyCustomPlugin(Plugin):
name = 'myplugin'
def options(self, parser, env=os.environ):
parser.add_option('--custom-path', action='store',
dest='custom_path', default=None,
help='Specify path to widget config file')
def configure(self, options, conf):
if options.custom_path:
self.make_some_configs(options.custom_path)
self.enabled = True
def make_some_configs(self, path):
# do some stuff based on the given path
def begin(self):
print 'Maybe print some useful stuff...'
# do some more stuff
and added a setup.py like this:
try:
from setuptools import setup, find_packages
except ImportError:
import distribute_setup
distribute_setup.use_setuptools()
from setuptools import setup, find_packages
setup(
name='mypackage',
...
install_requires=['nose==1.3.0'],
py_modules=['myplugin'],
entry_points={
'nose.plugins.1.3.0': [
'myplugin = myplugin:MyCustomPlugin'
]
}
)
Both files are in the same directory.
Every time I run nosetests --custom-path [path], I get:
nosetests: error: no such option: --custom-path
From the links mentioned above, I thought that's all that was required to register and enable a custom plugin.
But it seems that, either I'm doing something really wrong, or nose's docs are outdated.
Can someone please point me the correct way to register and enable a plugin, that I can use with nosetests?
Thanks a lot!! :)
You don't want the nose version in entry_points in setup.py. Just use nose.plugins.0.10 as the docs say. The dotted version in the entry point name is not so much a nose version as a plugin API version.