From the code that runs the tests using nose, how do I retrieve a list of config files that have been passed on the command line (without parsing the args myself since nose should expose these values somewhere) as in,
nosetests -c default.ini -c staging.ini
which would then result in,
[default.ini, staging.ini]
I can't seem to find these values on the nose.config object.
Seems like your problem is that you're naming your configuration files differently than what the default nose configuration files should be named.
From nose.config
config_files = [
# Linux users will prefer this
"~/.noserc",
# Windows users will prefer this
"~/nose.cfg"
]
def user_config_files():
"""Return path to any existing user config files
"""
return filter(os.path.exists,
map(os.path.expanduser, config_files))
def all_config_files():
"""Return path to any existing user config files, plus any setup.cfg
in the current working directory.
"""
user = user_config_files()
if os.path.exists('setup.cfg'):
return user + ['setup.cfg']
return user
The short of this is, that nose is looking for default configuration files named ~/.noserc or ~/nose.cfg. If they're not named like this nose will not pick them up and you will have to manually specify the name of the configuration files, like you are doing on the command line
Now say for instance that you have some object config which is an instance of nose.config.Config then the best way to get your config file names would be to say
>>> from nose.config import Config
>>> c = Config()
>>> c.configure(argv=["nosetests", "-c", "foo.txt"])
>>> c.options.files
['foo.txt']
Related
When packaging a Hydra application and including a script as an entry point, one can call the script from the command line anywhere. When called the script will utilize the configuration that was included in the package. Is there a way to modify the config search path such that a different configuration can be passed along to the script?
For example:
app.py
import hydra
#hydra.main(config_path="conf", config_name="config")
def my_func(cfg):
print(cfg)
config.yaml
key: value
another_key: second_value
Creating and installing this package with entry_points={"console_scripts": ["hydra_app_test = hydra_app_test.app:my_func"]}.
$ hydra_app_test
{'key': 'value', 'another_key': 'second_value'}
I would like to be able to define a local config and pass it to hydra_test_app. Is something like this possible?
some/config/path/config.yaml
key: not_value
mon: key
$ hydra_app_test --config-override=some/config/path
{'key': 'not_value', 'mon': 'key'}
I have tried using the --config-dir and --config-path overrides, but without any luck.
$ hydra_app_test --config-dir=some/config/path
{'key': 'value', 'another_key': 'second_value'}
$ hydra_app_test --config-path=some/config/path
Primary config module 'hydra_app_test.some.config.path' not found.
Check that it's correct and contains an __init__.py file
Interestingly enough this pattern works if you do not use the installed app, but run app.py as a script (with the requisite if __name__ == "__main__" logic)
python path/to/app.py --config-path=some/config/path
{'key': 'not_value', 'mon': 'key'}
Perhaps I am just missing something here, but it would seem you should be able to replicate the same behavior for both installed package scripts and python scripts.
Use --config-dir|-cp to add a directory to the config path for a specific run.
You can also use the hydra.searchpath config variable which can be overridden from the command line or from the primary config.
There is a page dedicated to the config search path here.
I have verified that this solutions works. I found this response in this feature request thread on the Hydra Github
Thread:
https://github.com/facebookresearch/hydra/issues/874
Solution:
https://github.com/facebookresearch/hydra/issues/874#issue-678621609
Basically any area of the config that you want to be overridable is set using a default config, this then allows the --config-dir flag to work in conjunction with a specified config file. The only caveat to this answer is that the new configs must be in a directory structure like the default configs.
I have trace32 installed at C drive and have mentioned that directory in my code. Suppose if some other user run this code in their system, the code does not work because the user has installed application in different location. How can I make this directory generic and dynamic and make it work for all users?
You have multiple possibilities. Bevor explaining them some generic tips:
Make the TRACE32 system path configurable, not a path inside the installation. In your case this would be r"C:\T32". This path is called t32sys or T32SYS.
Make sure you use os.path.join to concatenate your strings, so it works on the users operating system: os.path.join(r"C:\T32", "bin/windows64")
Command line arguments using argparse. This is the simplest solution which requires the user to start the Python script like this: python script.py --t32sys="C:\t32".
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--t32sys", help="TRACE32 system directory.")
args = parser.parse_args()
t32sys = args["t32sys"]
Instead of command line parameters you could also use a configuration file. For this you can use the built-in configparser module. This has the advantage that the user doesn't need to specify the directory as a command line argument, but the disadvantage that the user needs to be aware of the configuration file.
Configuration file (example.ini):
[DEFAULT]
t32sys = C:\T32
import configparser
parser = configparser.ConfigParser()
parser.read("example.ini")
args = parser["DEFAULT"]
t32sys = args["t32sys"]
Environment variables using os.environ. T32SYS is a environment variable often used for this, but it's not ensured to be set. So you have to tell users that they have to set the variable before using your tool. This approach has the advantage to work in the background, but also in my opinion a little bit obfuscating. I'd only use this in combination with argparse or configparse to override.
import os
t32sys = os.environ.get('T32SYS')
You can of course combine multiple ways with fallbacks / overrides.
I wonder, is it possible to pass behave arguments, eg. "-D environment". by default they are taken from the behave file.
Maybe there is some way to keep each configuration in a different file? Maybe many behave files? or Such as: behave "path to file with arguments"?
At this point i figured out that I could put bash scripts containing various configuration " #!/bin/bash behave ..."
I asking because i want easily manage my configurations when I run "behave -..." without editing many arguments
I think you could take advantage of using custom json configuration files described in behave Advanced Cases section: https://behave.readthedocs.io/en/latest/new_and_noteworthy_v1.2.5.html#advanced-cases
from documentation:
# -- FILE: features/environment.py
import json
import os.path
def before_all(context):
"""Load and update userdata from JSON configuration file."""
userdata = context.config.userdata
configfile = userdata.get("configfile", "userconfig.json")
if os.path.exists(configfile):
assert configfile.endswith(".json")
more_userdata = json.load(open(configfile))
context.config.update_userdata(more_userdata)
# -- NOTE: Reapplies userdata_defines from command-line, too.
So, if you like to use custom config by specifying it at the command line, you could run it as for example:
behave -D conffile=myconfig.json
Then I would parametrize this line to something like:
myconfigfile = context.config.userdata["conffile"]
configfile = userdata.get("configfile", myconfigfile)
I have two folders named Student and Faculty in the directory /home/ubuntu/Desktop/Pythontraining. I need to save totally 10 files inside Student folder and 3 files inside the Faculty folder.I need to do the same in another system where the Student and Faculty folders are present in different directory(say: /home/documents/college).How to store the file into the respective folders on two different machines without hardcoding the path?
For this kind of problem, you have multiple solutions :
Create environment variable on each machine, and inside the script do stuff like this :
import os
student_path = os.environ['STUDENT_PATH']
faculty_path = os.environ['FACULTY_PATH']
print(student_path, faculty_path)
Personal opinion : I don't like to configure my scripts using environment variables as the ones you chose may be used by another software + it is always messy to debug
Use arguments
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-s", "--student")
parser.add_argument("-f", "--faculty")
args = parser.parse_args()
student_path = args.student
faculty_path = args.faculty
print(student_path, faculty_path)
And then call your script like this and adapt this line depending on the machine
python <yourscript> -s <student_path> -f <faculty_path>
Personal opinion : I use arguments when I want to control a small amount of parameters on my scripts to change its behavior (verbose, nb of cpus, ...).
Create a config file and use configparser
config.ini file
[Paths]
student_path=<path_on_machine>
faculty_path=<path_on_machine>
Usage on script :
import configparser
config = configparser.ConfigParser()
config.read('config.ini')
student_path = config.get('Paths', 'student_path')
faculty_path = config.get('Paths', 'faculty_path')
print(student_path, faculty_path)
And then deploy different config.ini files on each machine (tools like ansible can help you to automate this)
Personal opinion : I use config files when I need to adapt parameters when deploying on new machines. I don't like to use arguments for this purpose as I don't want to specify the same values every time I use my script (often these kind of parameters don't have good default values).
Create a module
You can also create a module to store these parameters instead of a config file.
my_config.py
student_path="<path_on_machine>"
faculty_path="<path_on_machine>"
And then import it
script.py
import my_config
print(my_config.student_path, my_config.faculty_path)
I don't have any personal opinion on config files vs config modules. Read this if you want some elements of comparison.
You can use walk library for finding destination folder path. It's working best if you have only one folder for each search name:
import os
start = "/home/"
for dirpath, dirnames, filenames in os.walk(start):
found = False
for dirname in dirnames:
if dirname == 'Student':
full_path = os.path.join(dirpath, dirname)
found = True
break
if found:
break
Output:
/home/.../Student
I installed Salt in a Python 3 virtual environment and created a Salt configuration that uses a non-root folder for everything (/home/user/saltenv). When using the salt-ssh command inside the venv, e.g. salt-ssh '*' test.ping, everything works as exptected. (Please note that the config dir is resolved via a Saltfile, so the -c option is omitted, but that should not matter.)
When calling the SSHClient directly via Python however, I get no results. I already figured out that the roster file is not read, obviously resulting in an empty target list. I am stuck somehow and the documentation is not that helpful.
Here is the code:
import salt.config
from salt.client.ssh.client import SSHClient
def main():
c_path = '/home/user/saltenv/etc/salt/master'
master_opts = salt.config.client_config(c_path)
c = SSHClient(c_path=c_path, mopts=master_opts)
res = c.cmd(tgt='*', fun='test.ping')
print(res)
if __name__ == '__main__':
main()
As it seems, the processing of some options differs between the CLI and the Client. salt-ssh does not use the SSHClient. Instead, the class salt.client.ssh.SSH is used directly.
While salt-ssh adds the config_dir from the Saltfile to the opts dictionary to resolve the master config file, the SSHClient reads the config file passed to the constructor directly and config_dir is not added to the options (resulting in the roster file not being found).
My solution is to include config_dir in the master config file as well. The code from the question will then be working unchanged.
Alternative 1: If you only have one Salt configuration, it is also possible to set the environment variable SALT_CONFIG_DIR.
Alternative 2: The mopts argument of SSHClient can be used to pass a custom configuration directory, but it requires more lines of code:
config = '/home/user/saltenv/etc/salt/master'
defaults = dict(salt.config.DEFAULT_MASTER_OPTS)
defaults['config_dir'] = os.path.dirname(config)
master_opts = salt.config.client_config(config, defaults=defaults)
c = SSHClient(mopts=master_opts)