I was trying IPython with a module I created and it does not show the actual representation of class objects. Instead it shows something like
TheClass.__module__ + '.' + TheClass.__name__
I heavily use metaclasses in this module and I have really meaningful class representations that should be shown to the user.
Is there an IPython specific method I can change to make the right representation available instead of this namespace thingy that is quite useless in this application?
Or, if that's not possible, how can I customize my version of IPython to show the information I want?
EDIT
As complementary information, if I get a class and change the __module__ attribute to e.g. None, it blows with this traceback when trying to show the representation:
Traceback (most recent call last):
... [Huge traceback] ...
File "C:\Python32\lib\site-packages\IPython\lib\pretty.py", line 599, in _type_pprint
name = obj.__module__ + '.' + obj.__name__
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
So my expectations were right and this function is used to show class objects:
def _type_pprint(obj, p, cycle):
I tried customizing it in my class but I don't think I'm doing it right. This module IPython.lib.pretty does have a big dictionary linking type (the parent of metaclasses) with this function.
EDIT 2
Things I tried:
Adding the _repr_pretty_ function to metaclass. It do work with instances but not with classes...
Using this function IPython.lib.pretty.for_type(typ, func). It only changes the big dictionary a wrote above but not the copy of it made by the RepresentationPrinter instance... So this function has no use at all?!
Calling the magic function %pprint. It disables (or enables) this pretty print feature, using the default Python __repr__ for all the objects. That's bad because the pretty printing of lists, dict and many others are quite nice.
The first approach is more of what I want because it does not affect the environment and is specific for this class.
This is just an issue with IPython 0.12 and older versions. Now is possible to do:
class A(type):
def _repr_pretty_(cls, p, cycle):
p.text(repr(self))
def __repr__(cls):
return 'This Information'
class B: #or for Py3K: class B(metaclass=A):
__metaclass__ = A
and it'll show the desired representation for B.
Related
Given the following display function,
def display(some_object):
print(some_object.value)
is there a way to programmatically determine that the attributes of some_object must include value?
Modern IDEs (like PyCharm) yield a syntax error if I try to pass an int to the display function, so they are obviously doing this kind of analysis behind the scenes. I am aware how to get the function signature, this question is only about how to get the (duck) type information, i.e. which attributes are expected for each function argument.
EDIT: In my specific use case, I have access to the source code (non obfuscated), but I am not in control of adding the type hints as the functions are user defined.
Toy example
For the simple display function, the following inspection code would do,
class DuckTypeInspector:
def __init__(self):
self.attrs = []
def __getattr__(self, attr):
return self.attrs.append(attr)
dti = DuckTypeInspector()
display(dti)
print(dti.attrs)
which outputs
None # from the print in display
['value'] # from the last print statement, this is what i am after
However, as the DuckTypeInspector always returns None, this approach won't work in general. A simple add function for example,
def add(a, b):
return a + b
dti1 = DuckTypeInspector()
dti2 = DuckTypeInspector()
add(dti1, dti2)
would yield the following error,
TypeError: unsupported operand type(s) for +: 'DuckTypeInspector' and 'DuckTypeInspector'
The way to do this with static analysis is to declare the parameters as adhering to a protocol and then use mypy to validate that the actual parameters implement that protocol:
from typing import Protocol
class ValueProtocol(Protocol):
value: str
class ValueThing:
def __init__(self):
self.value = "foo"
def display(some_object: ValueProtocol):
print(some_object.value)
display(ValueThing()) # no errors, because ValueThing implements ValueProtocol
display("foo") # mypy error: Argument 1 to "display" has incompatible type "str"; expected "ValueProtocol"
Doing this at runtime with mock objects is impossible to do in a generic way, because you can't be certain that the function will go through every possible code path; you would need to write a unit test with carefully constructed mock objects for each function and make sure that you maintain 100% code coverage.
Using type annotations and static analysis is much easier, because mypy (or similar tools) can check each branch of the function to make sure that the code is compatible with the declared type of the parameter, without having to generate fake values and actually execute the function against them.
If you want to programmatically inspect the annotations from someone else's module, you can use the magic __annotations__ attribute:
>>> display.__annotations__
{'some_object': <class '__main__.ValueProtocol'>, 'return': None}
import util
class C():
save = util.save
setattr(C, 'load', util.load)
C.save is visible to the linter - but C.load isn't. There's thus some difference between assigning class methods from within the class itself, and from outside. Same deal for documentation builders; e.g. Sphinx won't acknowledge :meth:C.load - instead, need to do :func:util.load, which is misleading if load is meant to be C's method. An IDE (Spyder) also fails to "go to" method via self.load code.
The end-goal is to make linter (+docs & IDE) recognize load as C's method just like C.save is, but class method assignment needs to be dynamic (context). Can this be accomplished?
Note: the purpose of dynamic assignment is to automatically pull methods from modules (e.g. util) instead of having to manually update C upon method addition / removal.
Disclaimer: This solution does not work in all use cases, see comments. I leave it here, since it might still be useful under some circumstances.
I don't know about the linter and Sphinx, but an IDE like PyCharm will recognize the method if you declare it upfront using type hinting. In the code below, without the line load: Callable[[], None], I get the warning 'Unresolved attribute reference', but with the line there are no warnings in the file. Check the docs for more information about type hinting.
Notes:
Even with the more general load: Callable, the type checker is satisfied.
If you don't always declare a callable, a valid declaration is load: Optional[Callable] = None. This means that the default value is None. If you then call it without setting it, you will get an error, but you got that already anyway, that's unrelated to this typing.
p.s. I don't have your utils, so I defined some functions in the file itself.
from typing import Callable
def save():
pass
def load():
pass
class C:
load: Callable[[], None]
save = save
setattr(C, 'load', load)
C.load()
The attr's package somehow ruins pytorch's parameter() method for a module. I am wondering if anyone has any work-arounds or solutions, so that the two packages can seamlessly integrate?
If not, any advice on which github to post the issue to? My instinct would be to post this onto attr's github, but the stack trace is almost entirely relevant to pytorch's codebase.
Python 3.7.3
attrs== 19.1.0
torch==1.1.0.post2
torchvision==0.3.0
import attr
import torch
class RegularModule(torch.nn.Module):
pass
#attr.s
class AttrsModule(torch.nn.Module):
pass
module = RegularModule()
print(list(module.parameters()))
module = AttrsModule()
print(list(module.parameters()))
The actual output is:
$python attrs_pytorch.py
[]
Traceback (most recent call last):
File "attrs_pytorch.py", line 18, in <module>
print(list(module.parameters()))
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 814, in parameters
for name, param in self.named_parameters(recurse=recurse):
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 840, in named_parameters
for elem in gen:
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 784, in _named_members
for module_prefix, module in modules:
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 975, in named_modules
if self not in memo:
TypeError: unhashable type: 'AttrsModule'
The expected output is:
$python attrs_pytorch.py
[]
[]
You may get it to work with one workaround and using dataclasses (which you should, as it's in standard Python library since 3.7 which you are apparently using). Though I think simple __init__ is more readable. One could do something similar using attrs library (disabling hashing), I just prefer the solution using standard libraries if possible.
The reason (if you manage to handle hashing related errors) is that you are calling torch.nn.Module.__init__() which generates _parameters attribute and other framework-specific data.
First solving hashing with dataclasses:
#dataclasses.dataclass(eq=False)
class AttrsModule(torch.nn.Module):
pass
This solves hashing issues as, as stated by the documentation, section about hash and eq:
By default, dataclass() will not implicitly add a hash() method
unless it is safe to do so.
which is needed by PyTorch so the model can be used in C++ backed (correct me if I'm wrong), furthermore:
If eq is false, hash() will be left untouched meaning the
hash() method of the superclass will be used (if the superclass is object, this means it will fall back to id-based hashing).
So you are fine using torch.nn.Module __hash__ function (refer to documentation of dataclasses if any further errors arise).
This leaves you with the error:
AttributeError: 'AttrsModule' object has no attribute '_parameters'
Because torch.nn.Module constructor is not called. Quick and dirty fix:
#dataclasses.dataclass(eq=False)
class AttrsModule(torch.nn.Module):
def __post_init__(self):
super().__init__()
__post_init__ is a function called after __init__ (who would of guessed), where you can initialize torch-specific parameters.
Still, I would advise against using those two modules together. For example, you are destroying PyTorch's __repr__ using your code, so repr=False should be passed to the dataclasses.dataclass constructor, which gives this final code (obvious collisions between libraries eliminated I hope):
import dataclasses
import torch
class RegularModule(torch.nn.Module):
pass
#dataclasses.dataclass(eq=False, repr=False)
class AttrsModule(torch.nn.Module):
def __post_init__(self):
super().__init__()
module = RegularModule()
print(list(module.parameters()))
module = AttrsModule()
print(list(module.parameters()))
For more on attrs please see hynek answer and his blog post.
attrs has a chapter on hashability that also explains the pitfalls of hashing in Python: https://www.attrs.org/en/stable/hashing.html
You’ll have to decide what behavior is adequate for your concrete problem. For more general information check out https://hynek.me/articles/hashes-and-equality/ — turns out hashing is surprisingly tricky in Python.
I'm working on a web-server type of application and as part of multi-language communication I need to serialize objects in a JSON file. The issue is that I'd like to create a function which can take any custom object and save it at run time rather than limiting the function to what type of objects it can store based on structure.
Apologies if this question is a duplicate, however from what I have searched the other questions and answers do not seem to tackle the dynamic structure aspect of the problem, thus leading me to open this question.
The function is going to be used to communicate between PHP server code and Python scripts, hence the need for such a solution
I have attempted to use the json.dump(data,outfile) function, however the issue is that I need to convert such objects to a legal data structure first
JSON is a rigidly structured format, and Python's json module, by design, won't try to coerce types it doesn't understand.
Check out this SO answer. While __dict__ might work in some cases, it's often not exactly what you want. One option is to write one or more classes that inherit JSONEncoder and provides a method that turns your type or types into basic types that json.dump can understand.
Another option would be to write a parent class, e.g. JSONSerializable and have these data types inherit it the way you'd use an interface in some other languages. Making it an abstract base class would make sense, but I doubt that's important to your situation. Define a method on your base class, e.g. def dictify(self), and either implement it if it makes sense to have a default behavior or just have it it raise NotImplementedError.
Note that I'm not calling the method serialize, because actual serialization will be handled by json.dump.
class JSONSerializable(ABC):
def dictify(self):
raise NotImplementedError("Missing serialization implementation!")
class YourDataType(JSONSerializable):
def __init__(self):
self.something = None
# etc etc
def dictify(self):
return {"something": self.something}
class YourIncompleteDataType(JSONSerializable):
# No dictify(self) implementation
pass
Example usage:
>>> valid = YourDataType()
>>> valid.something = "really something"
>>> valid.dictify()
{'something': 'really something'}
>>>
>>> invalid = YourIncompleteDataType()
>>> invalid.dictify()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in dictify
NotImplementedError: Missing dictify implementation!
Basically, though: You do need to handle this yourself, possibly on a per-type basis, depending on how different your types are. It's just a matter of what method of formatting your types for serialization is the best for your use case.
I'm trying to mimic methods.grep from Ruby which simply returns a list of available methods for any object (class or instance) called upon, filtered by regexp pattern passed to grep.
Very handy for investigating objects in an interactive prompt.
def methods_grep(self, pattern):
""" returns list of object's method by a regexp pattern """
from re import search
return [meth_name for meth_name in dir(self) \
if search(pattern, meth_name)]
Because of Python's limitation not quite clear to me it unfortunately can't be simply inserted in the object class ancestor:
object.mgrep = classmethod(methods_grep)
# TypeError: can't set attributes of built-in/extension type 'object'
Is there some workaround how to inject all classes or do I have to stick with a global function like dir ?
There is a module called forbiddenfruit that enables you to patch built-in objects. It also allows you to reverse the changes. You can find it here https://pypi.python.org/pypi/forbiddenfruit/0.1.1
from forbiddenfruit import curse
curse(object, "methods_grep", classmethod(methods_grep))
Of course, using this in production code is likely a bad idea.
There is no workaround AFAIK. I find it quite annoying that you can't alter built-in classes. Personal opinion though.
One way would be to create a base object and force all your objects to inherit from it.
But I don't see the problem to be honest. You can simply use methods_grep(object, pattern), right? You don't have to insert it anywhere.