I am adding validation using schema for CLI that uses docopt, but I cannot seem to get optional to work. I want to validate that:
the input file exists
valid options are used
if the PATH is added that the directory exists.
Here is app so far
"""DVget
Usage:
DVget [-s] FILE [PATH]
Process a file, return data based on selection
and write results to PATH/output-file
Arguments:
FILE specify input file
PATH specify output directory (default: ./)
Options:
-s returns sections
-p returns name-sets
-m returns modules
"""
import os
from docopt import docopt
from schema import Schema, And, Use, Optional, SchemaError
# START OF SCRIPT
if __name__ == "__main__":
arguments = docopt(__doc__, version="0.1")
#print(arguments)
schema = Schema({
'FILE': [Use(open, error='FILE should be readable')],
Optional('PATH'): And(os.path.exists, error='PATH should exist'),
'-': And(str, lambda s: s in ('s', 'p', 'm'))})
try:
arguments = schema.validate(arguments)
# process(arguments)
except SchemaError as e:
exit(e)
running DVget -s "c:\test.txt" gives me the error message 'PATH should exist' even when using Optional in schema and docopt. Any suggestions?
Related
i'm writing a test of this function
def create_folder_if_not_exists(
sdk: looker_sdk,
folder_name: str,
parent_folder_name: str) -> dict:
folder = sdk.search_folders(name=folder_name)[0]
try:
parent_id = sdk.search_folders(name=parent_folder_name)[0].id
logger.info(f'Creating folder "{folder_name}"')
folder = sdk.create_folder(
body=models.CreateFolder(
name=folder_name,
parent_id=parent_id
)
)
return folder
except looker_sdk.error.SDKError as err:
logger.error(err.args[0])
return folder
This is my current test, using the python pytest library, but i keep getting this for my test Failed: DID NOT RAISE <class 'looker_sdk.error.SDKError'>
def test_create_folder_if_not_exists_parent1(mocker):
# Tests if a folder has parent id of 1 we raise an exception
sdk = fake_methods_data.MockSDK()
sf_data = fake_methods_data.MockSearchFolder(
name='goog', parent_id=1, id=3)
mocker.patch.object(sdk, "search_folders")
mocker.patch.object(sdk, "create_folder",
side_effect=[looker_sdk.error.SDKError])
sdk.search_folders.return_value = [sf_data]
with pytest.raises(looker_sdk.error.SDKError) as err:
test = fc.create_folder_if_not_exists(
sdk=sdk, folder_name='googn', parent_folder_name='1')
assert str(err.value) == 'test'
assert test.parent_id == 1
assert test.name == 'googn'
Does anyone know how to force a function to return a class error using pytest ? I've been looking at this [stackoverflow] (Mocking a function to raise an Exception to test an except block) but am struggling to get it to work. Hoping for some other thoughts.
This sounds like something I have done for work (open-source software dev stuff). In my case, I needed to test an except block raised when an executable file could not be run on a particular OS version. In our testing framework we use pytest and monkeypatch to test things. I've included the relevant bits of code below, along with some explanation about what is happening. I think this is probably what you mean by 'patch the sdk error', and I believe that is probably what you need to do. If anything is unclear, or you have more questions, let me know.
In conftest.py I define pytest fixtures that get used for tests in more than one test file. Here, I mock the scenario I want to test, using monkeypatch to fake the results I want from the parts of the get_version() function I'm not trying to test.
# conftest.py
import subprocess
import shutil
import os
import re
import platform
from pathlib import Path
import pytest
#pytest.fixture
def executable_incompatible_with_os(monkeypatch):
"""
Mocks an executable file that is incompatible with the OS.
(This situation likely only applies to blastall.)
"""
def mock_which(*args, **kwargs):
"""Mock an absolute file path."""
return args[0]
def mock_isfile(*args, **kwargs):
"""Mock a call to `os.path.isfile()`."""
return True
def mock_access(*args, **kwargs):
"""Mock a call to `os.access()`."""
return True
def mock_subprocess(*args, **kwargs):
"""Mock a call to `subprocess.run()` with an incompatible program."""
raise OSError
# Replace calls to existing methods with my mocked versions
monkeypatch.setattr(shutil, "which", mock_which)
monkeypatch.setattr(Path, "is_file", mock_isfile)
monkeypatch.setattr(os.path, "isfile", mock_isfile)
monkeypatch.setattr(os, "access", mock_access)
monkeypatch.setattr(subprocess, "run", mock_subprocess)
In test_aniblastall.py I test parts of aniblastall.py. In this case, I'm testing the behaviour when an OSError is raised; the code that raises the error in the test is in conftest.py. The entire pytest fixture I defined there is passed as a parameter to the test.
# test_aniblastall.py
from pathlib import Path
import unittest
# Test case 4: there is an executable file, but it will not run on the OS
def test_get_version_os_incompatible(executable_incompatible_with_os):
"""Test behaviour when the program can't run on the operating system.
This will happen with newer versions of MacOS."""
test_file_4 = Path("/os/incompatible/blastall")
assert (
aniblastall.get_version(test_file_4)
== f"blastall exists at {test_file_4} but could not be executed"
)
aniblastall.py contains the function the error should be raised from.
# aniblastall.py
import logging
import os
import platform
import re
import shutil
import subprocess
from pathlib import Path
def get_version(blast_exe: Path = pyani_config.BLASTALL_DEFAULT) -> str:
"""
The following circumstances are explicitly reported as strings
- no executable at passed path
- non-executable file at passed path (this includes cases where the user doesn't have execute permissions on the file)
- no version info returned
- executable cannot be run on this OS
"""
logger = logging.getLogger(__name__)
try:
blastall_path = Path(shutil.which(blast_exe)) # type:ignore
except TypeError:
return f"{blast_exe} is not found in $PATH"
if not blastall_path.is_file(): # no executable
return f"No blastall at {blastall_path}"
# This should catch cases when the file can't be executed by the user
if not os.access(blastall_path, os.X_OK): # file exists but not executable
return f"blastall exists at {blastall_path} but not executable"
if platform.system() == "Darwin":
cmdline = [blast_exe, "-version"]
else:
cmdline = [blast_exe]
try:
result = subprocess.run(
cmdline, # type: ignore
shell=False,
stdout=subprocess.PIPE, # type: ignore
stderr=subprocess.PIPE,
check=False, # blastall doesn't return 0
)
except OSError:
logger.warning("blastall executable will not run", exc_info=True)
return f"blastall exists at {blastall_path} but could not be executed"
version = re.search( # type: ignore
r"(?<=blastall\s)[0-9\.]*", str(result.stderr, "utf-8")
).group()
if 0 == len(version.strip()):
return f"blastall exists at {blastall_path} but could not retrieve version"
return f"{platform.system()}_{version} ({blastall_path})"
This is super valuable #baileythegreen, however my problem was far simpler. I had an if/else and the else had the try/catch error code piece. I was so focused on that I didn't check the simple part of if it was even getting to the else. :(
I am trying to implement hostname like module and my target machine in an amazon-ec2. But When I am running the script its giving me below error:
[ansible-user#ansible-master ~]$ ansible node1 -m edit_hostname.py -a node2
ERROR! this task 'edit_hostname.py' has extra params, which is only allowed in the following modules: meta, group_by, add_host, include_tasks, import_role, raw, set_fact, command, win_shell, import_tasks, script, shell, include_vars, include_role, include, win_command
My module is like this:
#!/usr/bin/python
from ansible.module_utils.basic import *
try:
import json
except ImportError:
import simplejson as json
def write_to_file(module, hostname, hostname_file):
try:
with open(hostname_file, 'w+') as f:
try:
f.write("%s\n" %hostname)
finally:
f.close()
except Exception:
err = get_exception()
module.fail_json(msg="failed to write to the /etc/hostname file")
def main():
hostname_file = '/etc/hostname'
module = AnsibleModule(argument_spec=dict(name=dict(required=True, type=str)))
name = module.params['name']
write_to _file(module, name, hostname_file)
module.exit_json(changed=True, meta=name)
if __name__ == "__main__":
main()
I don't know where I am making the mistake. Any help will be greatly appreciated. Thank you.
When developing a new module, I would recommend to use the boilerplate described in the documentation. This also shows that you'll need to use AnsibleModule to define your arguments.
In your main, you should add something like the following:
def main():
# define available arguments/parameters a user can pass to the module
module_args = dict(
name=dict(type='str', required=True)
)
# seed the result dict in the object
# we primarily care about changed and state
# change is if this module effectively modified the target
# state will include any data that you want your module to pass back
# for consumption, for example, in a subsequent task
result = dict(
changed=False,
original_hostname='',
hostname=''
)
module = AnsibleModule(
argument_spec=module_args
supports_check_mode=False
)
# manipulate or modify the state as needed (this is going to be the
# part where your module will do what it needs to do)
result['original_hostname'] = module.params['name']
result['hostname'] = 'goodbye'
# use whatever logic you need to determine whether or not this module
# made any modifications to your target
result['changed'] = True
# in the event of a successful module execution, you will want to
# simple AnsibleModule.exit_json(), passing the key/value results
module.exit_json(**result)
Then, you can call the module like so:
ansible node1 -m mymodule.py -a "name=myname"
ERROR! this task 'edit_hostname.py' has extra params, which is only allowed in the following modules: meta, group_by, add_host, include_tasks, import_role, raw, set_fact, command, win_shell, import_tasks, script, shell, include_vars, include_role, include, win_command
As explained by your error message, an anonymous default parameter is only supported by a limited number of modules. In your custom module, the paramter you created is called name. Moreover, you should not include the .py extension in the module name. You have to call your module like so as an ad-hoc command:
$ ansible node1 -m edit_hostname -a name=node2
I did not test your module code so you may have further errors to fix.
Meanwhile, I still strongly suggest you use the default boilerplate from the ansible documentation as proposed in #Simon's answer.
Given the following program:
#!/usr/bin/env python
import click
#click.command()
#click.argument("arg")
#click.option("--opt")
#click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):
print("arg: {}".format(arg))
print("opt: {}".format(opt))
print("config_file: {}".format(config_file))
return
if __name__ == "__main__":
main()
I can run it with the arguments and options provided through command line.
$ ./click_test.py my_arg --config_file my_config_file
arg: my_arg
opt: None
config_file: my_config_file
How do I provide a configuration file (in ini? yaml? py? json?) to --config_file and accept the content as the value for the arguments and options?
For instance, I want my_config_file to contain
opt: my_opt
and have the output of the program show:
$ ./click_test.py my_arg --config_file my_config_file
arg: my_arg
opt: my_opt
config_file: my_config_file
I've found the callback function which looked to be useful but I couldn't find a way to modify the sibling arguments/options to the same function.
This can be done by over riding the click.Command.invoke() method like:
Custom Class:
def CommandWithConfigFile(config_file_param_name):
class CustomCommandClass(click.Command):
def invoke(self, ctx):
config_file = ctx.params[config_file_param_name]
if config_file is not None:
with open(config_file) as f:
config_data = yaml.safe_load(f)
for param, value in ctx.params.items():
if value is None and param in config_data:
ctx.params[param] = config_data[param]
return super(CustomCommandClass, self).invoke(ctx)
return CustomCommandClass
Using Custom Class:
Then to use the custom class, pass it as the cls argument to the command decorator like:
#click.command(cls=CommandWithConfigFile('config_file'))
#click.argument("arg")
#click.option("--opt")
#click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):
Test Code:
# !/usr/bin/env python
import click
import yaml
#click.command(cls=CommandWithConfigFile('config_file'))
#click.argument("arg")
#click.option("--opt")
#click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):
print("arg: {}".format(arg))
print("opt: {}".format(opt))
print("config_file: {}".format(config_file))
main('my_arg --config_file config_file'.split())
Test Results:
arg: my_arg
opt: my_opt
config_file: config_file
I realize that this is way old, but since Click 2.0, there's a more simple solution. The following is a slight modification of the example from the docs.
This example takes explicit --port args, it'll take an environment variable, or a config file (with that precedence).
Command Groups
Our code:
import os
import click
from yaml import load
try:
from yaml import CLoader as Loader
except ImportError:
from yaml import Loader
#click.group(context_settings={'auto_envvar_prefix': 'FOOP'}) # this allows for environment variables
#click.option('--config', default='~/config.yml', type=click.Path()) # this allows us to change config path
#click.pass_context
def foop(ctx, config):
if os.path.exists(config):
with open(config, 'r') as f:
config = load(f.read(), Loader=Loader)
ctx.default_map = config
#foop.command()
#click.option('--port', default=8000)
def runserver(port):
click.echo(f"Serving on http://127.0.0.1:{port}/")
if __name__ == '__main__':
foop()
Assuming our config file (~/config.yml) looks like:
runserver:
port: 5000
and we have a second config file (at ~/config2.yml) that looks like:
runserver:
port: 9000
Then if we call it from bash:
$ foop runserver
# ==> Serving on http://127.0.0.1:5000/
$ FOOP_RUNSERVER_PORT=23 foop runserver
# ==> Serving on http://127.0.0.1:23/
$ FOOP_RUNSERVER_PORT=23 foop runserver --port 34
# ==> Serving on http://127.0.0.1:34/
$ foop --config ~/config2.yml runserver
# ==> Serving on http://127.0.0.1:9000/
Single Commands
If you don't want to use command groups and want to have configs for a single command:
import os
import click
from yaml import load
try:
from yaml import CLoader as Loader
except ImportError:
from yaml import Loader
def set_default(ctx, param, value):
if os.path.exists(value):
with open(value, 'r') as f:
config = load(f.read(), Loader=Loader)
ctx.default_map = config
return value
#click.command(context_settings={'auto_envvar_prefix': 'FOOP'})
#click.option('--config', default='config.yml', type=click.Path(),
callback=set_default, is_eager=True, expose_value=False)
#click.option('--port')
def foop(port):
click.echo(f"Serving on http://127.0.0.1:{port}/")
will give similar behavior.
Python has ConfigParser library for parsing mysql configuration files.
https://docs.python.org/2/library/configparser.html
But i need to parse configuration files of PostgreSQL as well. But i couldn't find any parser for them.
In PostgreSQL there is no section or heading, so it can't be done by configparser.
Example:
# The default values of these variables are driven from the -D command-line
# option or PGDATA environment variable, represented here as ConfigDir.
data_directory = '/data/postgresql' # use data in another directory
# (change requires restart)
hba_file = '/etc/postgresql/9.1/main/pg_hba.conf' # host-based authentication file
# (change requires restart)
ident_file = '/etc/postgresql/9.1/main/pg_ident.conf' # ident configuration file
# (change requires restart)
# If external_pid_file is not explicitly set, no extra PID file is written.
external_pid_file = '/var/run/postgresql/9.1-main.pid' # write an extra PID file
# (change requires restart)
I was using the below code for mysql and it's working fine but not for postgresql.conf file.
import ConfigParser
class MysqlParserModule:
def __init__(self):
print 'Mysql Parser Module'
def parse_option(self, conf_file, section, option):
try:
config = ConfigParser.ConfigParser()
config.read(conf_file)
result = config.get(section, option)
print "'%s' value is '%s' under '%s' section" % (option, result, section)
except Exception as ex:
print 'in parse_option exception'
print ex
def test(self):
test = MysqlParserModule()
conf_file = '/root/Desktop/db_config/my_new.cnf'
section = 'client'
option = 'port'
test.parse_option(conf_file, section, option)
if __name__ == '__main__':
test = MysqlParserModule()
test.test()
My python script needs to read files from a directory passed on the command line. I have defined a readable_dir type as below to be used with argparse for validating that the directory passed on the command line is existent and readable.
Additionally, a default value (/tmp/non_existent_dir in the example below) has also been specified for the directory argument.
The problem here is that argparse invokes readable_dir() on the default value even in a situation where a directory argument is explicitly passed in on the command line. This causes the script to crap out as the default path /tmp/non_existent_dir does not exist in a context where a directory is explicitly passed in on the command line.
I could get around this by not specifying a default value and making this argument mandatory, or by deferring the validation until later in the script but is a more elegant solution that anyone is aware of?
#!/usr/bin/python
import argparse
import os
def readable_dir(prospective_dir):
if not os.path.isdir(prospective_dir):
raise Exception("readable_dir:{0} is not a valid path".format(prospective_dir))
if os.access(prospective_dir, os.R_OK):
return prospective_dir
else:
raise Exception("readable_dir:{0} is not a readable dir".format(prospective_dir))
parser = argparse.ArgumentParser(description='test', fromfile_prefix_chars="#")
parser.add_argument('-l', '--launch_directory', type=readable_dir, default='/tmp/non_existent_dir')
args = parser.parse_args()
I submitted a patch for "path arguments" to the Python standard library mailing list a few months ago.
With this PathType class, you can simply specify the following argument type to match only an existing directory--anything else will give an error message:
type = PathType(exists=True, type='dir')
Here's the code, which could be easily modified to require specific file/directory permissions as well:
from argparse import ArgumentTypeError as err
import os
class PathType(object):
def __init__(self, exists=True, type='file', dash_ok=True):
'''exists:
True: a path that does exist
False: a path that does not exist, in a valid parent directory
None: don't care
type: file, dir, symlink, None, or a function returning True for valid paths
None: don't care
dash_ok: whether to allow "-" as stdin/stdout'''
assert exists in (True, False, None)
assert type in ('file','dir','symlink',None) or hasattr(type,'__call__')
self._exists = exists
self._type = type
self._dash_ok = dash_ok
def __call__(self, string):
if string=='-':
# the special argument "-" means sys.std{in,out}
if self._type == 'dir':
raise err('standard input/output (-) not allowed as directory path')
elif self._type == 'symlink':
raise err('standard input/output (-) not allowed as symlink path')
elif not self._dash_ok:
raise err('standard input/output (-) not allowed')
else:
e = os.path.exists(string)
if self._exists==True:
if not e:
raise err("path does not exist: '%s'" % string)
if self._type is None:
pass
elif self._type=='file':
if not os.path.isfile(string):
raise err("path is not a file: '%s'" % string)
elif self._type=='symlink':
if not os.path.symlink(string):
raise err("path is not a symlink: '%s'" % string)
elif self._type=='dir':
if not os.path.isdir(string):
raise err("path is not a directory: '%s'" % string)
elif not self._type(string):
raise err("path not valid: '%s'" % string)
else:
if self._exists==False and e:
raise err("path exists: '%s'" % string)
p = os.path.dirname(os.path.normpath(string)) or '.'
if not os.path.isdir(p):
raise err("parent path is not a directory: '%s'" % p)
elif not os.path.exists(p):
raise err("parent directory does not exist: '%s'" % p)
return string
You can create a custom action instead of a type:
import argparse
import os
import tempfile
import shutil
import atexit
class readable_dir(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
prospective_dir=values
if not os.path.isdir(prospective_dir):
raise argparse.ArgumentTypeError("readable_dir:{0} is not a valid path".format(prospective_dir))
if os.access(prospective_dir, os.R_OK):
setattr(namespace,self.dest,prospective_dir)
else:
raise argparse.ArgumentTypeError("readable_dir:{0} is not a readable dir".format(prospective_dir))
ldir = tempfile.mkdtemp()
atexit.register(lambda dir=ldir: shutil.rmtree(ldir))
parser = argparse.ArgumentParser(description='test', fromfile_prefix_chars="#")
parser.add_argument('-l', '--launch_directory', action=readable_dir, default=ldir)
args = parser.parse_args()
print (args)
But this seems a little fishy to me -- if no directory is given, it passes a non-readable directory which seems to defeat the purpose of checking if the directory is accessible in the first place.
Note that as pointed out in the comments, it might be nicer to raise argparse.ArgumentError(self, ...) rather than argparse.ArgumentTypeError.
EDIT
As far as I'm aware, there is no way to validate the default argument. I suppose the argparse developers just assumed that if you're providing a default, then it should be valid. The quickest and easiest thing to do here is to simply validate the arguments immediately after you parse them. It looks like, you're just trying to get a temporary directory to do some work. If that's the case, you can use the tempfile module to get a new directory to work in. I updated my answer above to reflect this. I create a temporary directory, use that as the default argument (tempfile already guarantees the directory it creates will be writeable) and then I register it to be deleted when your program exits.
If your script can't work without a valid launch_directory then it should be made a mandatory argument:
parser.add_argument('launch_directory', type=readable_dir)
btw, you should use argparse.ArgumentTypeError instead of Exception in readable_dir().