We Keep Coding

Is it possible to add attributes to built in python objects dynamically in Python?

I need to add an attribute (holding a tuple or object) to python objects dynamically. This works for Python classes written by me, but not for built in classes.
Consider the following program:
import numpy as np
class My_Class():
pass
my_obj = My_Class()
my_obj2 = My_Class()
my_obj.__my_hidden_field = (1,1)
my_obj2.__my_hidden_field = (2,1)
print(my_obj.__my_hidden_field, my_obj2.__my_hidden_field)
This correctly prints (1, 1) (2, 1). However the following program doesnt work.
X = np.random.random(size=(2,3))
X.__my_hidden_field = (3,1)
setattr(X, '__my_hidden_field', (3,1))
Both of the above line throws the following error # AttributeError: 'numpy.ndarray' object has no attribute '__my_hidden_field'
Now, the reason found from these questions (i.e., Attribute assignment to built-in object, Can't set attributes of object class, python: dynamically adding attributes to a built-in class) is Python does not allow dynamically adding attributes to built_in objects.
Excerpt from the answer: https://stackoverflow.com/a/22103924/8413477
This is prohibited intentionally to prevent accidental fatal changes to built-in types (fatal to parts of the code that you never though of). Also, it is done to prevent the changes to affect different interpreters residing in the address space, since built-in types (unlike user-defined classes) are shared between all such interpreters.
However, all the answers are quite old, and I am badly in need of doing this for my research project.
There is a module that allows to add methods to built in Class though:
https://pypi.org/project/forbiddenfruit/
However,it doesnt allow adding objects/attributes to each object.
Any help ?

You probably want weakref.WeakKeyDictionary. From the doc,
This can be used to associate additional data with an object owned by other parts of an application without adding attributes to those objects.
Like an attribute, and unlike a plain dict, this allows the objects to get garbage collected when there are no other references to it.
You'd look up the field with
my_hidden_field[X]
instead of
X._my_hidden_field
Two caveats: First, since a weak key may be deleted at any time without warning, you shouldn't iterate over a WeakKeyDictionary. Looking up an object you have a reference to is fine though. And second, you can't make a weakref to an object type written in C that doesn't have a slot for it (true for many builtins), or a type written in Python that doesn't allow a __weakref__ attribute (usually due to __slots__).
If this is a problem, you can just use a normal dict for those types, but you'll have to clean it up yourself.

Quick answer
Is it possible to add attributes to built in python objects dynamically in Python?
No, the reasons your read about in the links you posted, are the same now days. But I came out with a recipe I think might be the starting point of your tracer.
Instrumenting using subclassing combined with AST
After reading a lot about this, I came out with a recipe that might not be the complete solution, but it sure looks like you can start from here.
The good thing about this recipe is that it doesn't use third-party libraries, all is achieved with the standard (Python 3.5, 3.6, 3.7) libraries.
The target code.
This recipe will make code like this be instrumented (simple instrumentation is performed here, this is just a poof of concept) and executed.
# target/target.py
d = {1: 2}
d.update({3: 4})
print(d) # Should print "{1: 2, 3: 4}"
print(d.hidden_field) # Should print "(0, 0)"
Subclassing
Fist we have to add the hidden_field to anything we want to (this recipe have been tested only with dictionaries).
The following code receives a value, finds out its type/class and subclass it in order to add the mentioned hidden_field.
def instrument_node(value):
VarType = type(value)
class AnalyserHelper(VarType):
def __init__(self, *args, **kwargs):
self.hidden_field = (0, 0)
super(AnalyserHelper, self).__init__(*args, **kwargs)
return AnalyserHelper(value)
with that in place you are able to:
d = {1: 2}
d = instrument_node(d)
d.update({3: 4})
print(d) # Do print "{1: 2, 3: 4}"
print(d.hidden_field) # Do print "(0, 0)"
At this point, we know already a way to "add instrumentation to a built-in dictionary" but there is no transparency here.
Modify the AST.
The next step is to "hide" the instrument_node call and we will do that using the ast Python module.
The following is an AST node transformer that will take any dictionary it finds and wrap it in an instrument_node call:
class AnalyserNodeTransformer(ast.NodeTransformer):
"""Wraps all dicts in a call to instrument_node()"""
def visit_Dict(self, node):
return ast.Call(func=ast.Name(id='instrument_node', ctx=ast.Load()),
args=[node], keywords=[])
return node
Putting all together.
With thats tools you can the write a script that:
Read the target code.
Parse the program.
Apply AST changes.
Compile it.
And execute it.
import ast
import os
from ast_transformer import AnalyserNodeTransformer
# instrument_node need to be in the namespace here.
from ast_transformer import instrument_node
if __name__ == "__main__":
target_path = os.path.join(os.path.dirname(__file__), 'target/target.py')
with open(target_path, 'r') as program:
# Read and parse the target script.
tree = ast.parse(program.read())
# Make transformations.
tree = AnalyserNodeTransformer().visit(tree)
# Fix locations.
ast.fix_missing_locations(tree)
# Compile and execute.
compiled = compile(tree, filename='target.py', mode='exec')
exec(compiled)
This will take our target code, and wraps every dictionary with an instrument_node() and execute the result of such change.
The output of running this against our target code,
# target/target.py
d = {1: 2}
d.update({3: 4})
print(d) # Will print "{1: 2, 3: 4}"
print(d.hidden_field) # Will print "(0, 0)"
is:
>>> {1: 2, 3: 4}
>>> (0, 0)
Working example
You can clone a working example here.

Yes, it is possible, it is one of the coolest things of python, in Python, all the classes are created by the typeclass
You can read in detail here, but what you need to do is this
In [58]: My_Class = type("My_Class", (My_Class,), {"__my_hidden_field__": X})
In [59]: My_Class.__my_hidden_field__
Out[59]:
array([[0.73998002, 0.68213825, 0.41621582],
[0.05936479, 0.14348496, 0.61119082]])
*Edited because inheritance was missing, you need to pass the original class as a second argument (in tuple) so that it updates, otherwise it simply re-writes the class)

How to import method-wrapper type?

I have a library module-wrapper that recursively wraps objects. I want to determine if an object has a function-like type. I can check almost all function-like object using:
inspect.isbuiltin(object=obj) or
inspect.isfunction(object=obj) or
inspect.ismethod(object=obj) or
inspect.ismethoddescriptor(object=obj)
The problem is that some bound methods are not detected with this code, for example:
s = "Hello, world!"
type(s.__add__)
# method-wrapper
I guess I cannot check objects for being method-wrapper using inspect module. But how do I import this type? I didn't find it.
Now I have an ugly hack in my code:
MethodWrapper = type(''.__add__)
isinstance(obj, MethodWrapper)
UPD0:
I don't want to use callable because it detects classes and objects, that implement __call__, but I want those classes and objects to be handled separately.

The types module provides names for many “implementation” types, including several for functions implemented in C. The specific example of type("".__str__), MethodWrapperType, was just added in CPython 3.7.
Because this area is subtle and the number of types is large, callable may really be the best choice. You can easily check for type objects first to “exclude” classes, and various heuristics (e.g., trying to call vars on an object or examining type(x).__module__) can be used to recognize typical “callable objects” (not that those are fundamentally different from the built-in function types).

Python __doc__ documentation on instances

I'd like to provide documentation (within my program) on certain dynamically created objects, but still fall back to using their class documentation. Setting __doc__ seems a suitable way to do so. However, I can't find many details in the Python help in this regard, are there any technical problems with providing documentation on an instance? For example:
class MyClass:
"""
A description of the class goes here.
"""
a = MyClass()
a.__doc__ = "A description of the object"
print( MyClass.__doc__ )
print( a.__doc__ )

__doc__ is documented as a writable attribute for functions, but not for instances of user defined classes. pydoc.help(a), for example, will only consider the __doc__ defined on the type in Python versions < 3.9.
Other protocols (including future use-cases) may reasonably bypass the special attributes defined in the instance dict, too. See Special method lookup section of the datamodel documentation, specifically:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.
So, depending on the consumer of the attribute, what you intend to do may not be reliable. Avoid.
A safe and simple alternative is just to use a different attribute name of your own choosing for your own use-case, preferably not using the __dunder__ syntax convention which usually indicates a special name reserved for some specific use by the implementation and/or the stdlib.

There are some pretty obvious technical problems; the question is whether or not they matter for your use case.
Here are some major uses for docstrings that your idiom will not help with:
help(a): Type help(a) in an interactive terminal, and you get the docstring for MyClass, not the docstring for a.
Auto-generated documentation: Unless you write your own documentation generator, it's not going to understand that you've done anything special with your a value. Many doc generators do have some way to specify help for module and class constants, but I'm not aware of any that will recognize your idiom.
IDE help: Many IDEs will not only auto-complete an expression, but show the relevant docstring in a tooltip. They all do this statically, and without some special-case code designed around your idiom (which they're unlikely to have, given that it's an unusual idiom), they're almost certain to fetch the docstring for the class, not the object.
Here are some where it might help:
Source readability: As a human reading your source, I can tell the intent from the a.__doc__ = … right near the construction of a. Then again, I could tell the same intent just as easily from a Sphinx comment on the constant.
Debugging: pdb doesn't really do much with docstrings, but some GUI debuggers wrapped around it do, and most of them are probably going to show a.__doc__.
Custom dynamic use of docstrings: Obviously any code that you write that does something with a.__doc__ is going to get the instance docstring if you want it to, and therefore can do whatever it wants with it. However, keep in mind that if you want to define your own "protocol", you should use your own name, not one reserved for the implementation.
Notice that most of the same is true for using a descriptor for the docstring:
>>> class C:
... #property
... def __doc__(self):
... return('C doc')
>>> c = C()
If you type c.__doc__, you'll get 'C doc', but help(c) will treat it as an object with no docstring.
It's worth noting that making help work is one of the reasons some dynamic proxy libraries generate new classes on the fly—that is, a proxy to underlying type Spam has some new type like _SpamProxy, instead of the same GenericProxy type used for proxies to Hams and Eggseses. The former allows help(myspam) to show dynamically-generated information about Spam. But I don't know how important a reason it is; often you already need dynamic classes to, e.g., make special method lookup work, at which point adding dynamic docstrings comes for free.

I think it's preferred to keep it under the class via your doc string as it will also aid any developer that works on the code. However if you are doing something dynamic that requires this setup then I don't see any reason why not. Just understand that it adds a level of indirection that makes things less clear to others.
Remember to K.I.S.S. where applicable :)

I just stumbled over this and noticed that at least with python 3.9.5 the behavior seems to have changed.
E.g. using the above example, when I call:
help(a)
I get:
Help on MyClass in module __main__:
<__main__.MyClass object>
A description of the object
Also for reference, have a look at the pydoc implementation which shows:
def _getowndoc(obj):
"""Get the documentation string for an object if it is not
inherited from its class."""
try:
doc = object.__getattribute__(obj, '__doc__')
if doc is None:
return None
if obj is not type:
typedoc = type(obj).__doc__
if isinstance(typedoc, str) and typedoc == doc:
return None
return doc
except AttributeError:
return None

Is there a way to get ad-hoc polymorphism in Python?

Many languages support ad-hoc polymorphism (a.k.a. function overloading) out of the box. However, it seems that Python opted out of it. Still, I can imagine there might be a trick or a library that is able to pull it off in Python. Does anyone know of such a tool?
For example, in Haskell one might use this to generate test data for different types:
-- In some testing library:
class Randomizable a where
genRandom :: a
-- Overload for different types
instance Randomizable String where genRandom = ...
instance Randomizable Int where genRandom = ...
instance Randomizable Bool where genRandom = ...
-- In some client project, we might have a custom type:
instance Randomizable VeryCustomType where genRandom = ...
The beauty of this is that I can extend genRandom for my own custom types without touching the testing library.
How would you achieve something like this in Python?

Python is not a strongly typed language, so it really doesn't matter if yo have an instance of Randomizable or an instance of some other class which has the same methods.
One way to get the appearance of what you want could be this:
types_ = {}
def registerType ( dtype , cls ) :
types_[dtype] = cls
def RandomizableT ( dtype ) :
return types_[dtype]
Firstly, yes, I did define a function with a capital letter, but it's meant to act more like a class. For example:
registerType ( int , TheLibrary.Randomizable )
registerType ( str , MyLibrary.MyStringRandomizable )
Then, later:
type = ... # get whatever type you want to randomize
randomizer = RandomizableT(type) ()
print randomizer.getRandom()

A Python function cannot be automatically specialised based on static compile-time typing. Therefore its result can only depend on its arguments received at run-time and on the global (or local) environment, unless the function itself is modifiable in-place and can carry some state.
Your generic function genRandom takes no arguments besides the typing information. Thus in Python it should at least receive the type as an argument. Since built-in classes cannot be modified, the generic function (instance) implementation for such classes should be somehow supplied through the global environment or included into the function itself.
I've found out that since Python 3.4, there is #functools.singledispatch decorator. However, it works only for functions which receive a type instance (object) as the first argument, so it is not clear how it could be applied in your example. I am also a bit confused by its rationale:
In addition, it is currently a common anti-pattern for Python code to inspect the types of received arguments, in order to decide what to do with the objects.
I understand that anti-pattern is a jargon term for a pattern which is considered undesirable (and does not at all mean the absence of a pattern). The rationale thus claims that inspecting types of arguments is undesirable, and this claim is used to justify introducing a tool that will simplify ... dispatching on the type of an argument. (Incidentally, note that according to PEP 20, "Explicit is better than implicit.")
The "Alternative approaches" section of PEP 443 "Single-dispatch generic functions" however seems worth reading. There are several references to possible solutions, including one to "Five-minute Multimethods in Python" article by Guido van Rossum from 2005.

Does this count for ad hock polymorphism?
class A:
def __init__(self):
pass
def aFunc(self):
print "In A"
class B:
def __init__(self):
pass
def aFunc(self):
print "In B"
f = A()
f.aFunc()
f = B()
f.aFunc()
output
In A
In B

Another version of polymorphism
from module import aName
If two modules use the same interface, you could import either one and use it in your code.
One example of this is from xml.etree.ElementTree import XMLParser

How to find out what methods, properties, etc a python module possesses

Lets say I import a module. In order for me to make the best use of it, I would like to know what properties, methods, etc. that I can use. Is there a way to find that out?
As an example: Determining running programs in Python
In this line:
os.system('WMIC /OUTPUT:C:\ProcessList.txt PROCESS get Caption,Commandline,Processid')
Let's say I wanted to also print out the memory consumed by the processes. How do I find out if that's possible? And what would be the correct 'label' for it? (just as the author uses 'Commandline', 'ProcessId')
Similarly, in this:
import win32com.client
def find_process(name):
objWMIService = win32com.client.Dispatch("WbemScripting.SWbemLocator")
objSWbemServices = objWMIService.ConnectServer(".", "root\cimv2")
colItems = objSWbemServices.ExecQuery(
"Select * from Win32_Process where Caption = '{0}'".format(name))
return len(colItems)
print find_process("SciTE.exe")
How would I make the function also print out the memory consumed, the executable path, etc.?

As for Python modules, you can do
>>> import module
>>> help(module)
and you'll get a list of supported methods (more exactly, you get the docstring, which might not contain every single method). If you want that, you can use
>>> dir(module)
although now you'd just get a long list of all properties, methods, classes etc. in that module.
In your first example, you're calling an external program, though. Of course Python has no idea which features wmic.exe has. How should it?

dir(module) returns the names of the attributes of the module
module.__dict__ is the mapping between the keys and the attributes objects themselves
module.__dict__.keys() and dir(module) are lists having the same elements, though they are not equals because the elements aren't in same order in them
it seems that help(module) iswhat you really need

Python has a build in function called dir(). I'm not sure if this is what you are referring to, but fire up a interactive python console and type:
import datetime
dir(datetime)
This should give you a list of methods, properties and submodules

#ldmvcd
Ok, excuse me, I think you are a beginner and you don't see to what fundamental notions I am refering.
Objects are Python’s abstraction for
data. All data in a Python program is
represented by objects or by relations
between objects.
http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy
I don't understand why it is called "abstraction": for me an object is something real in the machine, a series of bits organized according certain rules to represent conceptual data or functionning.
Names refer to objects. Names are
introduced by name binding operations.
Each occurrence of a name in the
program text refers to the binding of
that name established in the innermost
function block containing the use.
http://docs.python.org/reference/executionmodel.html#naming-and-binding
.
A namespace is a mapping from names to
objects. Most namespaces are currently
implemented as Python dictionaries,
but that’s normally not noticeable in
any way (except for performance), and
it may change in the future. Examples
of namespaces are: the set of built-in
names (containing functions such as
abs(), and built-in exception names);
the global names in a module; and the
local names in a function invocation.
In a sense the set of attributes of an
object also form a namespace.
http://docs.python.org/tutorial/classes.html#a-word-about-names-and-objects
.
By the way, I use the word attribute
for any name following a dot — for
example, in the expression z.real,
real is an attribute of the object z.
Strictly speaking, references to names
in modules are attribute references:
in the expression modname.funcname,
modname is a module object and
funcname is an attribute of it. In
this case there happens to be a
straightforward mapping between the
module’s attributes and the global
names defined in the module: they
share the same namespace!
http://docs.python.org/tutorial/classes.html#a-word-about-names-and-objects
.
Namespaces are created at different
moments and have different lifetimes.
http://docs.python.org/tutorial/classes.html#a-word-about-names-and-objects
.
The namespace for a module is
automatically created the first time a
module is imported. The main module
for a script is always called
main. http://docs.python.org/reference/executionmodel.html#naming-and-binding
.
Well, a Python programm is a big machine that plays with objects, references to these objects , names of these objects, and namespaces in which are binded the names and the objects , namespaces being implemented as dictionaries.
So, you're right: when I refer to keys , I refer to names being the keys in the diverse namespaces. Names are arbitrary or not , according if the objects they have been created to name are user's objects or built-in objects.
I give advise you to read thoroughly the parts
3.1. Objects , values and types
http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy
and
4.1. Naming and binding
http://docs.python.org/reference/executionmodel.html#naming-and-binding