We Keep Coding

Is it bad practice to modify attributes of one module from another module?

I want to define a bunch of config variables that can be imported in all the modules in my project. The values of those variables will be constant during runtime but are not known before runtime; they depend on the input. Usually I'd define a dict in my top module which would be passed to all functions and classes from other modules; however, I was thinking it may be cleaner to simply create a blank config.py module which would be dynamically filled with config variables by the top module:
# top.py
import config
config.x = x
# config.py
x = None
# other.py
import config
print(config.x)
I like this approach because I don't have to save the parameters as attributes of classes in my other modules; which makes sense to me because parameters do not describe classes themselves.
This works but is it considered bad practice?

The question as such may be disputed. But I would generally say yes, it's "bad practice" because scope and impact of change is really getting blurred. Note the use case you're describing really is not about sharing configuration, but about different parts of the program functions, objects, modules exchanging data and as such it's a bit of a variation on (meta)global variable).
Reading common configuration values could be fine, but changing them along the way... you may lose track of what happened where and also in which order as modules get imported / values get modified. For instance assume the config.py and two modules m1.py:
import config
print(config.x)
config.x=1
and m2.py:
import config
print(config.x)
config.x=2
and a main.py that just does:
import m1
import m2
import config
print(config.x)
or:
import m2
import m1
import config
print(config.x)
The state in which you find config in each module and really any other (incl. main.py here) depends on order in which imports have occurred and who assigned what value when. Even for a program entirely under your control, this may get confusing (and source of mistakes) rather quickly.
For runtime data and passing information between objects and modules (and your example is really that and not configuration that is predefined and shared between modules) I would suggest you look into describing the information perhaps in a custom state (config) object and pass it around through appropriate interface. But really just a function / method argument may be all that is needed. The exact form depends on what exactly you're trying to achieve and what your overall design is.
In your example, other.py behaves differently when called or imported before top.py which may still seem obvious and manageable in a minimal example, but really is not a very sound design. Anyone reading the code (incl. future you) should be able to follow its logic and this IMO breaks its flow.
The most trivial (and procedural) example of what for what you've described and now I hopefully have a better grasp of would be other.py recreating your current behavior:
def do_stuff(value):
print(value) # We did something useful here
if __name__ == "__main__":
do_stuff(None) # Could also use config with defaults
And your top.py presumably being the entry point and orchestrating importing and execution doing:
import other
x = get_the_value()
other.do_stuff(x)
You can of course introduce an interface to configure do_stuff perhaps a dict or a custom class even with default implementation in config.py:
class Params:
def __init__(self, x=None):
self.x = x
and your other.py:
def do_stuff(params=config.Params()):
print(params.x) # We did something useful here
And on your top.py you can use:
params = config.Params(get_the_value())
other.do_stuff(params)
But you could also have any use case specific source of value(s):
class TopParams:
def __init__(self, url):
self.x = get_value_from_url(url)
params = TopParams("https://example.com/value-source")
other.do_stuff(params)
x could even be a property which you retrieve every time you access it... or lazily when needed and then cached... Again, it really then is a matter of what you need to do.

"Is it bad practice to modify attributes of one module from another module?"
that it is considered as bad practice - violation of the law of demeter, which means in fact "talk to friends, not to strangers".
Objects should expose behaviour and functions, but should HIDE the data.
DataStructures should EXPOSE data, but should not have any methods (which are exposed). The law of demeter does not apply to such DataStructures. OOP Purists might cover such DataStructures with setters and getters, but it really adds no value in Python.
there is a lot of literature about that like : https://en.wikipedia.org/wiki/Law_of_Demeter
and of course, a must to read: "Clean Code", by Robert C. Martin (Uncle Bob), check it out on Youtube also.
For procedural programming it is perfectly normal to keep data in a DataStructure which does not have any (exposed) methods.
The procedures in the program work with that data. Consider to use the module attrs, see : https://www.attrs.org/en/stable/ for easy creation of such classes.
my prefered method for keeping config is (here without using attrs):
# conf_xy.py
"""
config is code - so why use damned parsers, textfiles, xml, yaml, toml and all that
if You just can use testable code as config that can deliver the correct types, etc.
as well as hinting in Your favorite IDE ?
Here, for demonstration without using attrs package - usually I use attrs (read the docs)
"""
class ConfXY(object):
def __init__(self) -> None:
self.x: int = 1
self.z: float = get_z_from_input()
...
conf_xy=ConfXY()
# other.py
from conf_xy import conf_xy
...
y = conf_xy.x * 2
...

Is it possible to add attributes to built in python objects dynamically in Python?

I need to add an attribute (holding a tuple or object) to python objects dynamically. This works for Python classes written by me, but not for built in classes.
Consider the following program:
import numpy as np
class My_Class():
pass
my_obj = My_Class()
my_obj2 = My_Class()
my_obj.__my_hidden_field = (1,1)
my_obj2.__my_hidden_field = (2,1)
print(my_obj.__my_hidden_field, my_obj2.__my_hidden_field)
This correctly prints (1, 1) (2, 1). However the following program doesnt work.
X = np.random.random(size=(2,3))
X.__my_hidden_field = (3,1)
setattr(X, '__my_hidden_field', (3,1))
Both of the above line throws the following error # AttributeError: 'numpy.ndarray' object has no attribute '__my_hidden_field'
Now, the reason found from these questions (i.e., Attribute assignment to built-in object, Can't set attributes of object class, python: dynamically adding attributes to a built-in class) is Python does not allow dynamically adding attributes to built_in objects.
Excerpt from the answer: https://stackoverflow.com/a/22103924/8413477
This is prohibited intentionally to prevent accidental fatal changes to built-in types (fatal to parts of the code that you never though of). Also, it is done to prevent the changes to affect different interpreters residing in the address space, since built-in types (unlike user-defined classes) are shared between all such interpreters.
However, all the answers are quite old, and I am badly in need of doing this for my research project.
There is a module that allows to add methods to built in Class though:
https://pypi.org/project/forbiddenfruit/
However,it doesnt allow adding objects/attributes to each object.
Any help ?

You probably want weakref.WeakKeyDictionary. From the doc,
This can be used to associate additional data with an object owned by other parts of an application without adding attributes to those objects.
Like an attribute, and unlike a plain dict, this allows the objects to get garbage collected when there are no other references to it.
You'd look up the field with
my_hidden_field[X]
instead of
X._my_hidden_field
Two caveats: First, since a weak key may be deleted at any time without warning, you shouldn't iterate over a WeakKeyDictionary. Looking up an object you have a reference to is fine though. And second, you can't make a weakref to an object type written in C that doesn't have a slot for it (true for many builtins), or a type written in Python that doesn't allow a __weakref__ attribute (usually due to __slots__).
If this is a problem, you can just use a normal dict for those types, but you'll have to clean it up yourself.

Quick answer
Is it possible to add attributes to built in python objects dynamically in Python?
No, the reasons your read about in the links you posted, are the same now days. But I came out with a recipe I think might be the starting point of your tracer.
Instrumenting using subclassing combined with AST
After reading a lot about this, I came out with a recipe that might not be the complete solution, but it sure looks like you can start from here.
The good thing about this recipe is that it doesn't use third-party libraries, all is achieved with the standard (Python 3.5, 3.6, 3.7) libraries.
The target code.
This recipe will make code like this be instrumented (simple instrumentation is performed here, this is just a poof of concept) and executed.
# target/target.py
d = {1: 2}
d.update({3: 4})
print(d) # Should print "{1: 2, 3: 4}"
print(d.hidden_field) # Should print "(0, 0)"
Subclassing
Fist we have to add the hidden_field to anything we want to (this recipe have been tested only with dictionaries).
The following code receives a value, finds out its type/class and subclass it in order to add the mentioned hidden_field.
def instrument_node(value):
VarType = type(value)
class AnalyserHelper(VarType):
def __init__(self, *args, **kwargs):
self.hidden_field = (0, 0)
super(AnalyserHelper, self).__init__(*args, **kwargs)
return AnalyserHelper(value)
with that in place you are able to:
d = {1: 2}
d = instrument_node(d)
d.update({3: 4})
print(d) # Do print "{1: 2, 3: 4}"
print(d.hidden_field) # Do print "(0, 0)"
At this point, we know already a way to "add instrumentation to a built-in dictionary" but there is no transparency here.
Modify the AST.
The next step is to "hide" the instrument_node call and we will do that using the ast Python module.
The following is an AST node transformer that will take any dictionary it finds and wrap it in an instrument_node call:
class AnalyserNodeTransformer(ast.NodeTransformer):
"""Wraps all dicts in a call to instrument_node()"""
def visit_Dict(self, node):
return ast.Call(func=ast.Name(id='instrument_node', ctx=ast.Load()),
args=[node], keywords=[])
return node
Putting all together.
With thats tools you can the write a script that:
Read the target code.
Parse the program.
Apply AST changes.
Compile it.
And execute it.
import ast
import os
from ast_transformer import AnalyserNodeTransformer
# instrument_node need to be in the namespace here.
from ast_transformer import instrument_node
if __name__ == "__main__":
target_path = os.path.join(os.path.dirname(__file__), 'target/target.py')
with open(target_path, 'r') as program:
# Read and parse the target script.
tree = ast.parse(program.read())
# Make transformations.
tree = AnalyserNodeTransformer().visit(tree)
# Fix locations.
ast.fix_missing_locations(tree)
# Compile and execute.
compiled = compile(tree, filename='target.py', mode='exec')
exec(compiled)
This will take our target code, and wraps every dictionary with an instrument_node() and execute the result of such change.
The output of running this against our target code,
# target/target.py
d = {1: 2}
d.update({3: 4})
print(d) # Will print "{1: 2, 3: 4}"
print(d.hidden_field) # Will print "(0, 0)"
is:
>>> {1: 2, 3: 4}
>>> (0, 0)
Working example
You can clone a working example here.

Yes, it is possible, it is one of the coolest things of python, in Python, all the classes are created by the typeclass
You can read in detail here, but what you need to do is this
In [58]: My_Class = type("My_Class", (My_Class,), {"__my_hidden_field__": X})
In [59]: My_Class.__my_hidden_field__
Out[59]:
array([[0.73998002, 0.68213825, 0.41621582],
[0.05936479, 0.14348496, 0.61119082]])
*Edited because inheritance was missing, you need to pass the original class as a second argument (in tuple) so that it updates, otherwise it simply re-writes the class)

Most Pythonic way to provide function metadata at compile time?

I am building a very basic platform in the form of a Python 2.7 module. This module has a read-eval-print loop where entered user commands are mapped to function calls. Since I am trying to make it easy to build plugin modules for my platform, the function calls will be from my Main module to an arbitrary plugin module. I'd like a plugin builder to be able to specify the command that he wants to trigger his function, so I've been looking for a Pythonic way to remotely enter a mapping in the command->function dict in the Main module from the plugin module.
I've looked at several things:
Method name parsing: the Main module would import the plugin module
and scan it for method names that match a certain format. For
example, it might add the download_file_command(file) method to its
dict as "download file" -> download_file_command. However, getting a
concise, easy-to-type command name (say, "dl") requires that the
function's name also be short, which isn't good for code
readability. It also requires the plugin developer to conform to a
precise naming format.
Cross-module decorators: decorators would let
the plugin developer name his function whatever he wants and simply
add something like #Main.register("dl"), but they would necessarily
require that I both modify another module's namespace and keep
global state in the Main module. I understand this is very bad.
Same-module decorators: using the same logic as above, I could add a
decorator that adds the function's name to some command name->function mapping local to the
plugin module and retrieve the mapping to the Main module with an
API call. This requires that certain methods always be present or
inherited though, and - if my understanding of decorators is correct - the function will only register itself the first time it is run and will unnecessarily re-register itself every subsequent time
thereafter.
Thus, what I really need is a Pythonic way to annotate a function with the command name that should trigger it, and that way can't be the function's name. I need to be able to extract the command name->function mapping when I import the module, and any less work on the plugin developer's side is a big plus.
Thanks for the help, and my apologies if there are any flaws in my Python understanding; I'm relatively new to the language.

Building or Standing on the first part of #ericstalbot's answer, you might find it convenient to use a decorator like the following.
################################################################################
import functools
def register(command_name):
def wrapped(fn):
#functools.wraps(fn)
def wrapped_f(*args, **kwargs):
return fn(*args, **kwargs)
wrapped_f.__doc__ += "(command=%s)" % command_name
wrapped_f.command_name = command_name
return wrapped_f
return wrapped
################################################################################
#register('cp')
def copy_all_the_files(*args, **kwargs):
"""Copy many files."""
print "copy_all_the_files:", args, kwargs
################################################################################
print "Command Name: ", copy_all_the_files.command_name
print "Docstring : ", copy_all_the_files.__doc__
copy_all_the_files("a", "b", keep=True)
Output when run:
Command Name: cp
Docstring : Copy many files.(command=cp)
copy_all_the_files: ('a', 'b') {'keep': True}

User-defined functions can have arbitrary attributes. So you could specify that plug-in functions have an attribute with a certain name. For example:
def a():
return 1
a.command_name = 'get_one'
Then, in your module you could build a mapping like this:
import inspect #from standard library
import plugin
mapping = {}
for v in plugin.__dict__.itervalues():
if inspect.isfunction(v) and v.hasattr('command_name'):
mapping[v.command_name] = v
To read about arbitrary attributes for user-defined functions see the docs

There are two parts in a plugin system:
Discover plugins
Trigger some code execution in a plugin
The proposed solutions in your question address only the second part.
There many ways to implement both depending on your requirements e.g., to enable plugins, they could be specified in a configuration file for your application:
plugins = some_package.plugin_for_your_app
another_plugin_module
# ...
To implement loading of the plugin modules:
plugins = [importlib.import_module(name) for name in config.get("plugins")]
To get a dictionary: command name -> function:
commands = {name: func
for plugin in plugins
for name, func in plugin.get_commands().items()}
Plugin author can use any method to implement get_commands() e.g., using prefixes or decorators — your main application shouldn't care as long as get_commands() returns the command dictionary for each plugin.
For example, some_plugin.py (full source):
def f(a, b):
return a + b
def get_commands():
return {"add": f, "multiply": lambda x,y: x*y}
It defines two commands add, multiply.

How to mock pysvn

I am working on a Python module that suppose to checkout some code from SVN and build it. After much refactoring of some legacy code, I got a fairly decent coverage of the code, however, I have a gaping hole in the code that uses pysvn.
Admittedly the concept of Mock object is new to me, but after reading some of the documentation of MiniMock and pymox (both are available in my environment), I came to the conclusion that I will need to capture some pysvn output and have it returned in my test code.
But here I find myself (pardon the pun) in a pickle. The objects returned from the pysvn.Client() commands do not behave nicely when I try to pickle them, or even to compare them.
Any suggestion of how to serialize or otherwise mock pysvn or some other non-pythonic behaving objects?
Naturally, I am willing to accept that I am approaching this problem from the wrong direction, or that I am simply an idiot. In that case any advice will be helpful.
Additional information 0:
Some pysvn object can be reduced to a dict by accessing their data property, and can be reproduced by passing this dict into the appropriate __init__()
For example:
>>> svn=pysvn.Client()
>>> svn.list('http://svn/svn/')[0][0]
<PysvnList u'http://svn/svn'>
>>> d=svn.list('http://svn/svn/')[0][0].data
>>> pysvn.PysvnList(d)
<PysvnList u'http://svn/svn'>
However inside this object there might be some unpicklable objects:
>>> cPickle.dumps(d)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
cPickle.UnpickleableError: Cannot pickle <type 'node_kind'> objects
Additional Information 1:
As for #H. Dunlop request, here is a (simplified) snippet of my code,
It allow to get a list out of SVN, and let the user choose an item from that list:
class Menu(object):
"""a well covered class"""
# ...
class VersionControl(object):
"""A poorly covered class"""
def __init__(self):
self.svn = pysvn.Client()
# ...
def list(self, url):
"""svn ls $url"""
return [os.path.basename(x['path']) for (x,_) in self.svn.list(url)[1:]]
def choose(self, choice, url):
"""Displays a menu from svn list, and get's the users choice form it.
Returns the svn item (path).
"""
menu = Menu(prompt="Please choose %s from list:\n" % choice,
items=self.list(url),
muliple_choice=False)
menu.present()
return menu.chosen()

In this answer I used minimock, I'm not actually that familiar with it and would suggest using http://www.voidspace.org.uk/python/mock/ instead. This code would end up a bit cleaner . But you specified minimock or pymox so here goes:
from minimock import TraceTracker, Mock, mock
import unittest
import pysvn
from code_under_test import VersionControl
class TestVersionControl(unittest.TestCase):
def test_init(self):
mock_svn = Mock(name='svn_client')
mock('pysvn.Client', returns=mock_svn)
vc = VersionControl()
self.assertEqual(vc.svn, mock_svn)
def test_list_calls_svn_list_and_returns_urls(self):
tracker = TraceTracker()
test_url = 'a test_url'
mock_data = [
({'path': 'first result excluded'}, None),
({'path': 'url2'}, None),
({'path': 'url3', 'info': 'not in result'}, None),
({'path': 'url4'}, None),
]
vc = VersionControl()
mock('vc.svn.list', returns=mock_data, tracker=tracker)
response = vc.list(test_url)
self.assertEqual(['url2', 'url3', 'url4'], response)
self.assertTrue("Called vc.svn.list('a test_url')" in tracker.dump())
if __name__ == '__main__':
unittest.main()
If you wanted to test more of the underlying dictionary returned by pysvn then you can just modify the list of tuples with dictionaries inside of it that you make it return. You could even write a little bit of code that just dumped out the dictionaries from the pysvn objects .

Have you considered the use of: pickle instead cPicles?
"cPickle module the callables Pickler() and Unpickler() are functions, not classes. This means that you cannot use them to derive custom pickling and unpickling subclasses."

How to initialize a class with data from a python file

I'd like to init a class from data stored in a simple python file specified while calling the script. The config file named myconfig.py is :
str='home'
val=2
flt=7.0
I'd like to call it during class initilization like so. One of the objectives is to define variable types as well in the file. I know of the configparser, but this method less verbose if it can be made to work.
class ClassInit(object):
def __init__(self, configFile):
fp, path, des = imp.find_module('',configFile)
imp.load_module(configFile, fp, path, des)
self.__dict__ = configFile.__dict__
fp.close()
def printVal(self):
print '%s %0.2f'%(self.str, self.val)
if __name__ == '__main__':
srcDir = 'src/'
config = osp.join(srcDir, argv[0]) # config for current run
ci = ClassInit(config)
ci.printVal()
Is anything like this possible?

Well, there are several ways to do this. The easiest way would be to use eval() or exec to evaluate this code within the class scope. But that's also the most dangerous way, especially if these files can be created by someone other than you. In that case, the creator can write malicious code that can pretty much do anything. You can override the __builtins__ key of the globals dictionary, but I'm not sure if this makes eval/exec entirely safe. For example:
class ClassInit(object):
def __init__(self, configFile):
f = open(configFile)
config = f.read()
f.close()
config_dic = { '__builtins__': None}
exec 'a = 4' in config_dic
for key, value in config_dic.iteritems():
if key != '__builtins__':
setattr(self, key, value)
This method kills the unsafe 'builtins' object, but it's still not quite safe. For instance, the file may be able to define a function which would override one of your class's functions with malicious code. So I really don't recommend it, unless you absolutely control thos .py files.
A safer but more complex way would be to create a custom interpreter that interprets this file but doesn't allow running any custom code.
You can read the following thread, to see some suggestions for parsing libraries or other safer alternatives to eval():
Python: make eval safe
Besides, if all you ever need your config.py file for is to initialize some variables in a nice way, and you don't need to be able to call fancy python functions from inside it, you should consider using JSON instead. Python 2.6 and up includes simplejson, which you can use to initialize an object from file. The syntax is Javascript and not Python, but for initializing variables there's little difference there.

Can you try self.__dict__.update(configFile.__dict__)? I don't see why that wouldn't work.