We Keep Coding

Python project through cmd with different config files

i want to start my python project through a cmd window with different configurations, e.g the function that need different parameter:
download.rawData.download(starttime=starttime, endtime=endtime)
the starttime and endtime values are from a configfile: (config1.cfg)
[Parameter]
starttime=Monday
endtime=Tuesday
config2.cfg:
[Parameter]
starttime=tuesday
endtime=friday
What is the best way to start the project from the cmd like:
Python3 project.py --config1 //for time X
Python3 project.py --config2 //for time Y
...and so on, of course there are different start and endtimes declared in the config file
The goal is that the configurations for the start- and endtime are not hard-coded in the main project.
What I tried until now:
commandLineArgumentParser: ArgumentParser = argparse.ArgumentParser()
commandLineArgumentParser.add_argument("-config1", "--config1", help="Config file 1")
commandLineArgumentParser.add_argument("-config2", "--config2", help="Config file2")
commandLineArguments = commandLineArgumentParser.parse_args()
config1= commandLineArguments.odessa
starttime = config['Parameter']['starttime']
endtime = config['Parameter']['endtime']
But this didn´t work
Anyone has an idea?
Thanks a lot!

You wouldn't need to run the script multiple times for each parameter. Simply use the configparser module to parse your config file, which you specify on the command line (via the argparse module):
import argparse
import configparser
parser = argparse.ArgumentParser()
parser.add_argument('config', help="Config file to parse")
args = parser.parse_args()
config = configparser.ConfigParser()
config.read(args.config)
config.sections() # Returns ['Parameter']
start = config['Parameter']['starttime']
end = config['Parameter']['endtime']

Python Click - Supply arguments and options from a configuration file

Given the following program:
#!/usr/bin/env python
import click
#click.command()
#click.argument("arg")
#click.option("--opt")
#click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):
print("arg: {}".format(arg))
print("opt: {}".format(opt))
print("config_file: {}".format(config_file))
return
if __name__ == "__main__":
main()
I can run it with the arguments and options provided through command line.
$ ./click_test.py my_arg --config_file my_config_file
arg: my_arg
opt: None
config_file: my_config_file
How do I provide a configuration file (in ini? yaml? py? json?) to --config_file and accept the content as the value for the arguments and options?
For instance, I want my_config_file to contain
opt: my_opt
and have the output of the program show:
$ ./click_test.py my_arg --config_file my_config_file
arg: my_arg
opt: my_opt
config_file: my_config_file
I've found the callback function which looked to be useful but I couldn't find a way to modify the sibling arguments/options to the same function.

This can be done by over riding the click.Command.invoke() method like:
Custom Class:
def CommandWithConfigFile(config_file_param_name):
class CustomCommandClass(click.Command):
def invoke(self, ctx):
config_file = ctx.params[config_file_param_name]
if config_file is not None:
with open(config_file) as f:
config_data = yaml.safe_load(f)
for param, value in ctx.params.items():
if value is None and param in config_data:
ctx.params[param] = config_data[param]
return super(CustomCommandClass, self).invoke(ctx)
return CustomCommandClass
Using Custom Class:
Then to use the custom class, pass it as the cls argument to the command decorator like:
#click.command(cls=CommandWithConfigFile('config_file'))
#click.argument("arg")
#click.option("--opt")
#click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):
Test Code:
# !/usr/bin/env python
import click
import yaml
#click.command(cls=CommandWithConfigFile('config_file'))
#click.argument("arg")
#click.option("--opt")
#click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):
print("arg: {}".format(arg))
print("opt: {}".format(opt))
print("config_file: {}".format(config_file))
main('my_arg --config_file config_file'.split())
Test Results:
arg: my_arg
opt: my_opt
config_file: config_file

I realize that this is way old, but since Click 2.0, there's a more simple solution. The following is a slight modification of the example from the docs.
This example takes explicit --port args, it'll take an environment variable, or a config file (with that precedence).
Command Groups
Our code:
import os
import click
from yaml import load
try:
from yaml import CLoader as Loader
except ImportError:
from yaml import Loader
#click.group(context_settings={'auto_envvar_prefix': 'FOOP'}) # this allows for environment variables
#click.option('--config', default='~/config.yml', type=click.Path()) # this allows us to change config path
#click.pass_context
def foop(ctx, config):
if os.path.exists(config):
with open(config, 'r') as f:
config = load(f.read(), Loader=Loader)
ctx.default_map = config
#foop.command()
#click.option('--port', default=8000)
def runserver(port):
click.echo(f"Serving on http://127.0.0.1:{port}/")
if __name__ == '__main__':
foop()
Assuming our config file (~/config.yml) looks like:
runserver:
port: 5000
and we have a second config file (at ~/config2.yml) that looks like:
runserver:
port: 9000
Then if we call it from bash:
$ foop runserver
# ==> Serving on http://127.0.0.1:5000/
$ FOOP_RUNSERVER_PORT=23 foop runserver
# ==> Serving on http://127.0.0.1:23/
$ FOOP_RUNSERVER_PORT=23 foop runserver --port 34
# ==> Serving on http://127.0.0.1:34/
$ foop --config ~/config2.yml runserver
# ==> Serving on http://127.0.0.1:9000/
Single Commands
If you don't want to use command groups and want to have configs for a single command:
import os
import click
from yaml import load
try:
from yaml import CLoader as Loader
except ImportError:
from yaml import Loader
def set_default(ctx, param, value):
if os.path.exists(value):
with open(value, 'r') as f:
config = load(f.read(), Loader=Loader)
ctx.default_map = config
return value
#click.command(context_settings={'auto_envvar_prefix': 'FOOP'})
#click.option('--config', default='config.yml', type=click.Path(),
callback=set_default, is_eager=True, expose_value=False)
#click.option('--port')
def foop(port):
click.echo(f"Serving on http://127.0.0.1:{port}/")
will give similar behavior.

Python calling a file from another directory

I've been trying to find the answer to this but everything i look at is reading a file from another directory or running a file from another directory. I'm sure what i want to do isn't that hard but i am new at this and don't know how to describe what it's called.
I have a python script run.py that is in the /src directory. I run that script form the /src directory. run.py calls two files (configuration.py and server.py). These two files are in a folder called lib (src/lib).All folders have an empty __init__.py file.
When i take these files out of lib and put them just in src i can run the script when the script looks like it does below.
import os
import inspect
import sys
import configuration
import server
# Initialize Parameters
f_path = os.path.abspath(inspect.getfile(inspect.currentframe()))
absolute_path = os.path.dirname(f_path)
if __name__ == "__main__":
from optparse import OptionParser, OptionGroup
parser = OptionParser()
parser.usage = "usage: %prog [options] "
parser.description = "Primary application entry point."
parser.add_option("-v", "--verbose", dest="verbose", action="store_true",
default=False, help="Run verbose.")
group = OptionGroup(parser, "Node operations")
group.add_option("--do-queue-job", dest="do_queue_job", action="store_true",
help="Run the next job in the quasar-server queue.")
parser.add_option_group(group)
(options, args) = parser.parse_args()
# Clear argv to prevent issues with web.py automatically using arguments to
# bind ip addresses.
sys.argv = []
configuration = configuration.Configuration("/home/mscarpa/PhpstormProjects/quasar-node/quasar-node/quasar-node/src/config.yml")
if (options.do_queue_job):
# Get server instance
server_connection = server.QuasarConnection(configuration)
#return server_connection
# Get next job from server
next_job = server_connection.get_next_job()
#return next_job
The two parts of the code i know i have to change if i move the two files to /src/lib are the following:
configuration = configuration.Configuration("/home/mscarpa/PhpstormProjects/quasar-node/quasar-node/quasar-node/src/config.yml")
server_connection = server.QuasarConnection(configuration)
i am thinking that i would just have to put.lib before them like so, but every time i try it it says lib is not defined.
configuration = lib.configuration.Configuration("/home/mscarpa/PhpstormProjects/quasar-node/quasar-node/quasar-node/src/config.yml")
server_connection = lib.server.QuasarConnection(configuration)
This is probably a noob question, but does anyone know how to target these files if the are in the src/lib directory as opposed to just the src directory

You just need to change your import statement to reflect the module's new location:
import lib.configuration as configuration
import lib.server as server
And the rest of your script doesn't really need to change.

I got it. I think your answer may have worked in certain cases but i think my problem being new at this is figuring out what to search.
it was a sys.arg thing so i had to include that path to that lib folder before i imported the files.
sys.path.insert(0, '/home/mscarpa/PhpstormProjects/quasar-node/quasar-node/quasar-node/src/lib')
import configuration
import server

Python OS get current File name on imported module

I am trying to write a logging module that I can use over and over again. The logging is not the problem. I want to call my logging module from any script and not have to pass the params for the logging file.
If I have a test script called mytest.py how would I return this name in the import logging module. I have tried this in the logging script itself but it returns the name of that file and not the file I am trying to log.
my_test.py:
from my_logger import logging
info.logging("something here")
print("something here")
if __name__ == "__main__":
logging()
I would expect the log file to be called my_test.log, but currently it is being named logging.log
Here is the part from the logging script:
def logging(filename=False, level=DEFAULT_LOG_LEVEL):
if filename is False:
file_ext = os.path.basename(__file__) # Need this to be my_test.py
filename = ("C:/Users/Logs/{0}").format(file_ext)
"Start logging with given filename and level."
#print(filename)
logging.basicConfig(filename=filename, level=LEVELS[level])
else:
"Start logging with given filename and level."
logging.basicConfig(filename=filename, level=LEVELS[level])

Assuming your compiler is CPython (thanks to Matthew Trevor);
You can use inspect.getouterframes to get the caller's frame, plus the filename and line number etc.
import inspect
def logging(filename=False, level=DEFAULT_LOG_LEVEL):
if filename is False:
caller_file = inspect.getouterframes(inspect.currentframe())[1][3]
# prints /home/foo/project/my_test.py
...

The __file__ binding you use in your logging function will hold the value of the file it's scoped in. You need to pass in the calling module's __file__ to get the behaviour you want:
# my_test.py
...
if __name__ == "__main__":
logging(__file__)
I'm surprised that your log file was called logging.log and not my_logger.log, though, as that's the name of the file in which the logging function is defined.

Passing command line argument to another file imported in Python

I have a python file (html2text.py) which gives the desired result when i pass command line argument to it i.e., in the following way:
python html2text.py file.txt
where file.txt contains the source code of a web-site and the result is displayed on the console...
I want to use it in another file (let say a.py) and store the result (which was getting printed on the console) in a string.
For this I need to first import the file (html2text.py) in my file (a.py). Can anyone tell me how do I proceed further...?

Good way is to create some API in your html2text.py. For example:
# html2text.py
def parse(filename):
f = open(filename)
# do the stuff
return output_string
def main():
import sys
print parse(sys.argv[1])
if __name__ == '__main__':
main()
Then you will be able to use it in your a.py:
import html2text # main() will not run
import sys
output = html2text.parse(sys.argv[1])

I think the best way is the reorganize a little your html2text.py file. Append the line like this to your file:
def main():
message = sys.stdin.readlines()
a = your_def(message)
if __name__ == '__main__': main()
Now you're sure, that when invoking the file from command line, everything will go fine. Moreover, if you have everything kept in functions and classes, you can now in your a.py
import html2text
and work on it already in a.py.