Everyone at Class too big and hard to add new features is completely unphased by the question, which somehow connects command line options to methods, but I can find no documentation for this. It's not optparse, or argparse, or sys.argv - the question implies some kind of direct relationship between methods and command line options. What am I missing?
There isn't any set-in-stone link between them. The question you link to appears to be a program that can do one of several different things, with command-line arguments switching between them. These things happen to be implemented in the program using methods.
It is implied by the question that they have used something like argparse to write the glue between these; but the use of methods is just an implementation detail of the particular program.
I simply use the class like this, what seems not to be a very good idea, because it is very hard to maintain once u got many commands.
class myprogram(object):
def __init__(self)
self.prepare()
def prepare(self):
# some initializations
self.prepareCommands()
def prepareCommands(self):
self.initCommand("--updateDatabase", self.updateDatabase)
self.initCommand("--getImages", self.getImages)
# and so on
def initCommand(self, cmd, func):
options = sys.argv
for option in options:
if option.find(cmd)!=-1:
return func()
# my commands
def updateDatabase(self):
#...
def getImages(self):
#...
if __name__ == "__main__":
p = myprogram()
EDIT1:
Here a cleaner way I just implemented:
myprogram.py:
from config import * # has settings
from commands import *
from logsys import log
import filesys
class myprogram(object):
def __init__(self):
log(_class=self.__name__, _func='__init__', _level=0)
log(_class=self.__name__, _func='__init__', text="DEBUG LEVEL %s" % settings["debug"], _level=0)
self.settings = settings
self.cmds = commands
def prepare(self):
log(_class=self.__name__, _func='prepare', _level=1)
self.dirs = {}
for key in settings["dir"].keys():
self.dirs[key] = settings["dir"][key]
filesys.checkDir(self.dirs[key])
def initCommands(self):
log(_class=self.__name__, _func='initCommands', _level=1)
options = sys.argv
for option in options:
for cmd in self.cmds.keys():
if option.find(cmd) != -1:
return self.cmds[cmd]()
if __name__ == '__main__':
p = myprogram()
p.prepare()
p.initCommands()
commands.py:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
commands = {}
#csv
import csvsys
commands["--getCSV"] = csvsys.getCSV
#commands["--getCSVSplitted"] = csvsys.getCSVSplitted
# update & insert
import database
commands["--insertProductSpecification"] = database.insertProductSpecification
# download
import download
commands["--downloadProductSites"] = download.downloadProductSites
commands["--downloadImages"] = download.downloadImages
# parse
import parse
commands["--parseProductSites"] = parse.parseProductSites
EDIT2: I have now updated my question you linked to your question with a more complete example Class too big and hard to add new features
Related
I am having the following problem and I am fearful there isn't a straghtforward way to solve it so I am asking here. I am using Click to implement a CLI and I have created several grouped commands under the main command. This is the code:
#click.group()
def main():
pass
#main.command()
def getq():
'''Parameters: --questionnaire_id, --question_id, --session_id, --option_id'''
click.echo('Question Answers')
When I type the main command alone in my terminal it lists all the subcommands with the help text next to each one. However, the text is not displayed fully for the case of getq. Instead, it displays only "Parameters: --questionnaire_id, --question_id,... ."
Is there a way to display it all?
Thank You
The easiest way to do this is to use the command's short_help argument:
#click.group()
def main():
pass
#main.command(short_help='Parameters: --questionnaire_id, --question_id, --session_id, --option_id')
def getq():
click.echo('Question Answers')
If you insist to use the docstring for this and want to override the automatic shortening of it, then you could use a custom Group class overriding the format_commands method to directly use cmd.help instead of the get_short_help_str method:
import click
from gettext import gettext as _
class FullHelpGroup(click.Group):
def format_commands(self, ctx: click.Context, formatter: click.HelpFormatter) -> None:
"""Extra format methods for multi methods that adds all the commands
after the options.
"""
commands = []
for subcommand in self.list_commands(ctx):
cmd = self.get_command(ctx, subcommand)
# What is this, the tool lied about a command. Ignore it
if cmd is None:
continue
if cmd.hidden:
continue
commands.append((subcommand, cmd))
# allow for 3 times the default spacing
if len(commands):
limit = formatter.width - 6 - max(len(cmd[0]) for cmd in commands)
rows = []
for subcommand, cmd in commands:
help = cmd.help if cmd.help is not None else ""
rows.append((subcommand, help))
if rows:
with formatter.section(_("Commands")):
formatter.write_dl(rows)
#click.group(cls=FullHelpGroup)
def main():
pass
#main.command()
def getq():
'''Parameters: --questionnaire_id, --question_id, --session_id, --option_id'''
click.echo('Question Answers')
if __name__ == "__main__":
main()
You most probably want to override the max_content_width (at most 80 columns by default) also. You could do this by overriding the context settings:
import shutil
#click.group(cls=FullHelpGroup,
context_settings={'max_content_width': shutil.get_terminal_size().columns - 10})
def main():
pass
I'm asked to develop unit tests for a program which is such badly developed that the tests don't run... but the program does. Thus, I need to explain the reason why and I actually don't know!
Here is a piece of code that intends to represent the code I need to test:
from services import myModule1
from services.spec1 import importedFunc
from services.spec2 import getTool
from services.spec3 import getDict
class myClass(object):
def __init__(self, param1, param2):
self.param1 = param1
self.param2 = param2
self.param3 = 0
self.param4 = 0
def myMethod(self):
try:
myVar1 = globalDict['key1']
myVar2 = globalDict['key2']
newVar = importedFunc(par1=myVar1, par2=myVar2, par3=extVar3)
calcParam = myModule1.methodMod1(self.param1)
self.param3 = calcParam["keyParam3"]
self.param4 = newVar.meth1(self.param2)
globTools.send_message(self.param3, self.param4)
except:
globTools.error_message(self.param3, self.param4)
return
class myClass2(object):
def __init__(self, *myclass2_params):
# some piece of code to intialize dedicated attributes
self.add_objects()
def add_objects(self):
# Some piece of code
my_class = myClass(**necessary_params)
# Some piece of code
return
if __name__ == '__main__':
globTools = getTool("my_program")
globalDict = getDict(some_params)
# Some piece of code
my_class2 = myClass2(**any_params)
# Some piece of code
As you can see, the problem is that the class and its methods uses global variables, defined in the main scope. And it's just a quick summary because it's actually a bit more complicated, but I hope it's enough to give you an overview of the context and help me understand why the unit test fail.
I tried to mock the imported modules, but I did not manage to a successful result, so I first tried to make it simple and just initialize all parameters.
I went to this test file:
import unittest
from my_module import myClass
from services import myModule1
from services.spec1 import importedFunc
from services.spec2 import getTool
from services.spec3 import getDict
def test_myClass(unittest.TestCase):
def setUp(self):
globTools = getTool("my_program")
globalDict = getDict(some_params)
def test_myMethod(self):
test_class = myClass(*necessary_parameters)
test_res = test_class.myMethod()
self.assertIsNotNone(test_res)
if __name__ == '__main__':
unittest.main()
But the test fail, telling me 'globTools is not defined' when trying to instantiate myClass
I also tried to initialize variables directly in the test method, but the result is the same
And to be complete about the technical environment, I cannot run python programs directly and need to launch a docker environment via a Jenkins pipeline - I'm not very familiar with this but I imagine it should not have an impact on the result
I guess the problem comes from the variable's scopes, but I'm not able to explain it in this case: why the test fail where as the method itself works (yes, it actually works, or at least the program globally runs without)
It's not as bad as you think. Your setUp method just needs to define the appropriate top-level globals in your module, rather than local variables.
import unittest
import my_module
from my_module import myClass
from services import myModule1
from services.spec1 import importedFunc
from services.spec2 import getTool
from services.spec3 import getDict
class test_myClass(unittest.TestCase):
def setUp(self):
my_module.globTools = getTool("my_program")
my_module.globalDict = getDict(some_params)
def test_myMethod(self):
test_class = myClass(*necessary_parameters)
test_res = test_class.myMethod()
self.assertIsNotNone(test_res)
if __name__ == '__main__':
unittest.main()
Depending on how the code uses the two globals, setUpClass might be a better place to initialize them, but it's probably not worth worrying about. Once you have tests for the code, you are in a better position to remove the dependency on these globals from the code.
I want to build a well-modularized python project, where all alternative modules should be registed and acessed via a function named xxx_builder.
Taking data class as an example:
register.py:
def register(key, module, module_dict):
"""Register and maintain the data classes
"""
if key in module_dict:
logger.warning(
'Key {} is already pre-defined, overwritten.'.format(key))
module_dict[key] = module
data_dict = {}
def register_data(key, module):
register(key, module, data_dict)
data.py:
from register import register_data
import ABCDEF
class MyData:
"""An alternative data class
"""
pass
def call_my_data(data_type):
if data_type == 'mydata'
return MyData
register_data('mydata', call_my_data)
builder.py:
import register
def get_data(type):
"""Obtain the corresponding data class
"""
for func in register.data_dict.values():
data = func(type)
if data is not None:
return data
main.py:
from data import MyData
from builder import get_data
if __name__ == '__main__':
data_type = 'mydata'
data = get_data(type=data_type)
My problem
In main.py, to register MyData class into register.data_dict before calling the function get_data, I need to import data.py in advance to execute register_data('mydata', call_my_data).
It's okay when the project is small, and all the data-related classes are placed according to some rules (e.g. all data-related class should be placed under the directory data) so that I can import them in advance.
However, this registeration mechanism means that all data-related classes will be imported, and I need to install all packages even if I won't use it actually. For example, when the indicator data_type in main.py is not mydata I still need to install ABCDEF package for the class MyData.
So is there any good idea to avoid importing all the packages?
Python's packaging tools come with a solution for this: entry points. There's even a tutorial about how to use entry points for plugins (which seems like what you're doing) (in conjunction with this Setuptools tutorial).
IOW, something like this (nb. untested), if you have a plugin package that has defined
[options.entry_points]
myapp.data_class =
someplugindata = my_plugin.data_classes:SomePluginData
in setup.cfg (or pypackage.toml or setup.py, with their respective syntaxes), you could register all of these plugin classes (here shown with an example with a locally registered class too).
from importlib.metadata import entry_points
data_class_registry = {}
def register(key):
def decorator(func):
data_class_registry[key] = func
return func
return decorator
#register("mydata")
class MyData:
...
def register_from_entrypoints():
for entrypoint in entry_points(group="myapp.data_class"):
register(entrypoint.name)(entrypoint.load())
def get_constructor(type):
return data_class_registry[type]
def main():
register_from_entrypoints()
get_constructor("mydata")(...)
get_constructor("someplugindata")(...)
I have config data that should be loaded before another code (because another code use it).
So, for now I see only way to do this is to call the function at the top before rest imports:
from Init.Loaders.InitPreLoader import InitPreLoader
# this is my config loader
InitPreLoader.load()
from World.WorldManager import WorldManager
from Init.Loaders.InitLoader import InitLoader
from Init.Registry.InitRegistry import InitRegistry
from Utils.Debug import Logger
# ...
if __name__ == '__main__':
# ...
InitLoader.load()
Does it possible to do this in more elegant way and avoid to violate pep8 ?
P.S. If I need to share more code please let me know
UPD: All my classes declared in separate files
This is PreLoader:
from Typings.Abstract.AbstractLoader import AbstractLoader
from Init.Registry.InitRegistry import InitRegistry
from Config.Init.configs import main_config
class InitPreLoader(AbstractLoader):
#staticmethod
def load(**kwargs):
InitRegistry.main_config = main_config
This is Registry (where I store all my initialized data):
from Typings.Abstract.AbstractRegistry import AbstractRegistry
class InitRegistry(AbstractRegistry):
main_config = None
login_server = None
world_server = None
world_observer = None
identifier_region_map = None
region_octree_map = None
Parent of all classes (except AbstractRegistry) is AbstractBase class (it contains mixin):
from abc import ABC
from Config.Mixins.ConfigurableMixin import ConfigurableMixin
class AbstractBase(ConfigurableMixin, ABC):
pass
This mixin works with main_config from InitRegistry.
Also, after PreLoader's load was called, I load rest data with my InitLoader.load() (see first code snapshot):
from Typings.Abstract.AbstractLoader import AbstractLoader
from Init.Registry.InitRegistry import InitRegistry
from Server.Init.servers import login_server, world_server
from World.Observer.Init.observers import world_observer
from World.Region.Init.regions import identifier_region_map, region_octree_map
class InitLoader(AbstractLoader):
#staticmethod
def load(**kwargs):
InitRegistry.login_server = login_server
InitRegistry.world_server = world_server
InitRegistry.world_observer = world_observer
InitRegistry.identifier_region_map = identifier_region_map
InitRegistry.region_octree_map = region_octree_map
Well, for now I have found solution: I moved from Init.Loaders.InitPreLoader import InitPreLoader to separate file and called InitPreLoader.load() there. But I not like this solution, because my PyCharm IDE highlights it as unused import:
import Init.Init.preloader
from World.WorldManager import WorldManager
# ...
Maybe it is possible to improve this solution ? Or maybe another (more elegant) solition exists ?
I'm writing unit tests to validate my project functionalities. I need to replace some of the functions with mock function and I thought to use the Python mock library. The implementation I used doesn't seem to work properly though and I don't understand where I'm doing wrong. Here a simplified scenario:
root/connector.py
from ftp_utils.py import *
def main():
config = yaml.safe_load("vendor_sftp.yaml")
downloaded_files = []
downloaded_files = get_files(config)
for f in downloaded_files:
#do something
root/utils/ftp_utils.py
import os
import sys
import pysftp
def get_files(config):
sftp = pysftp.Connection(config['host'], username=config['username'])
sftp.chdir(config['remote_dir'])
down_files = sftp.listdir()
if down_files is not None:
for f in down_files:
sftp.get(f, os.path.join(config['local_dir'], f), preserve_mtime=True)
return down_files
root/tests/connector_tester.py
import unittest
import mock
import ftp_utils
import connector
def get_mock_files():
return ['digital_spend.csv', 'tv_spend.csv']
class ConnectorTester(unittest.TestCase)
#mock.patch('ftp_utils.get_files', side_effect=get_mock_files)
def test_main_process(self, get_mock_files_function):
# I want to use a mock version of the get_files function
connector.main()
When I debug my test I expect that the get_files function called inside the main of connector.py is the get_mock_files(), but instead is the ftp_utils.get_files(). What am I doing wrong here? What should I change in my code to properly call the get_mock_file() mock?
Thanks,
Alessio
I think there are several problems with your scenario:
connector.py cannot import from ftp_utils.py that way
nor can connector_tester.py
as a habit, it is better to have your testing files under the form test_xxx.py
to use unittest with patching, see this example
In general, try to provide working minimal examples so that it is easier for everyone to run your code.
I modified rather heavily your example to make it work, but basically, the problem is that you patch 'ftp_utils.get_files' while it is not the reference that is actually called inside connector.main() but probably rather 'connector.get_files'.
Here is the modified example's directory:
test_connector.py
ftp_utils.py
connector.py
test_connector.py:
import unittest
import sys
import mock
import connector
def get_mock_files(*args, **kwargs):
return ['digital_spend.csv', 'tv_spend.csv']
class ConnectorTester(unittest.TestCase):
def setUp(self):
self.patcher = mock.patch('connector.get_files', side_effect=get_mock_files)
self.patcher.start()
def test_main_process(self):
# I want to use a mock version of the get_files function
connector.main()
suite = unittest.TestLoader().loadTestsFromTestCase(ConnectorTester)
if __name__ == "__main__":
unittest.main()
NB: what is called when running connector.main() is 'connector.get_files'
connector.py:
from ftp_utils import *
def main():
config = None
downloaded_files = []
downloaded_files = get_files(config)
for f in downloaded_files:
print(f)
connector/ftp_utils.py unchanged.