Here's a simple class created declaratively:
class Person:
def say_hello(self):
print("hello")
And here's a similar class, but it was defined by invoking the metaclass manually:
def say_hello(self):
print("sayolala")
say_hello.__qualname__ = 'Person.say_hello'
TalentedPerson = type('Person', (), {'say_hello': say_hello})
I'm interested to know whether they are indistinguishable. Is it possible to detect such a difference from the class object itself?
>>> def was_defined_declaratively(cls):
... # dragons
...
>>> was_defined_declaratively(Person)
True
>>> was_defined_declaratively(TalentedPerson)
False
This should not matter, at all. Even if we dig for more attributes that differ, it should be possible to inject these attributes into the dynamically created class.
Now, even without the source file around (from which, things like inspect.getsource can make their way, but see below), class body statements should have a corresponding "code" object that is run at some point. The dynamically created class won't have a code body (but if instead of calling type(...) you call types.new_class you can have a custom code object for the dynamic class as well - so, as for my first statement: it should be possible to render both classes indistinguishable.
As for locating the code object without relying on the source file (which, other than by inspect.getsource can be reached through a method's .__code__ attibute which anotates co_filename and co_fistlineno (I suppose one would have to parse the file and locate the class statement above the co_firstlineno then)
And yes, there it is:
given a module, you can use module.__loader__.get_code('full.path.tomodule') - this will return a code_object. This object has a co_consts attribute which is a sequence with all constants compiled in that module - among those are the code objects for the class bodies themselves. And these, have the line number, and code objects for the nested declared methods as well.
So, a naive implementation could be:
import sys, types
def was_defined_declarative(cls):
module_name = cls.__module__
module = sys.modules[module_name]
module_code = module.__loader__.get_code(module_name)
return any(
code_obj.co_name == cls.__name__
for code_obj in module_code.co_consts
if isinstance(code_obj, types.CodeType)
)
For simple cases. If you have to check if the class body is inside another function, or nested inside another class body, you have to do a recursive search in all code objects .co_consts attribute in the file> Samething if you find if safer to check for any attributes beyond the cls.__name__ to assert you got the right class.
And again, while this will work for "well behaved" classes, it is possible to dynamically create all these attributes if needed - but that would ultimately require one to replace the code object for a module in sys.__modules__ - it starts to get a little more cumbersome than simply providing a __qualname__ to the methods.
update
This version compares all strings defined inside all methods on the candidate class. This will work with the given example classess - more accuracy can be achieved by comparing other class members such as class attributes, and other method attributes such as variable names, and possibly even bytecode. (For some reason, the code object for methods in the module's code object and in the class body are different instances,though code_objects should be imutable) .
I will leave the implementation above, which only compares the class names, as it should be better for understanding what is going on.
def was_defined_declarative(cls):
module_name = cls.__module__
module = sys.modules[module_name]
module_code = module.__loader__.get_code(module_name)
cls_methods = set(obj for obj in cls.__dict__.values() if isinstance(obj, types.FunctionType))
cls_meth_strings = [string for method in cls_methods for string in method.__code__.co_consts if isinstance(string, str)]
for candidate_code_obj in module_code.co_consts:
if not isinstance(candidate_code_obj, types.CodeType):
continue
if candidate_code_obj.co_name != cls.__name__:
continue
candidate_meth_strings = [string for method_code in candidate_code_obj.co_consts if isinstance(method_code, types.CodeType) for string in method_code.co_consts if isinstance(string, str)]
if candidate_meth_strings == cls_meth_strings:
return True
return False
It is not possible to detect such difference at runtime with python.
You can check the files with a third-party app but not in the language since no matter how you define your classes they should be reduced to the objects which the interpreter knows how to manage.
Everything other is syntax sugar and its death with at the preprocessing step of the operations on the text.
The whole metaprogramming is a technique that lets you close to the compiler/interpreter work.
Revealing some of the type traits and giving you the freedom to work on the type with code.
It is possible — somewhat.
inspect.getsource(TalentedPerson) will fail with an OSError, whereas it will succeed with Person. This only works though if you don't have a class of that name in the file where it was defined:
If your file consists of both of these definitions, and TalentedPerson also believes it is Person, then inspect.getsource will simply find Person's definition.
Obviously this relies on the source code still being around and findable by inspect — this won't work with compiled code, e.g. in the REPL, can be tricked, and is sort of cheating. The actual code objects don't differ AFAIK.
Related
I'm working on a some classes, and for the testing process it would be very useful to be able to run the class methods in a for loop. I'm adding methods and changing their names, and I want this to automatically change in the file where I run the class for testing.
I use the function below to get a list of the methods I need to run automatically (there are some other conditional statements I deleted for the example to make sure that I only run certain methods that require testing and which only have self as an argument)
def get_class_methods(class_to_get_methods_from):
import inspect
methods = []
for name, type in (inspect.getmembers(class_to_get_methods_from)):
if 'method' in str(type) and str(name).startswith('_') == False:
methods.append(name)
return methods
Is it possible to use the returned list 'methods' to run the class methods in a for loop?
Or is there any other way to make sure i can run my class methods in my testingrunning file without having to alter or add things i changed in the class?
Thanks!
Looks like you want getattr(object, name[, default]):
class Foo(object):
def bar(self):
print("bar({})".format(self))
f = Foo()
method = getattr(f, "bar")
method()
As a side note : I'm not sure that dynamically generating lists of methods to test is such a good idea (looks rather like an antipattern to me) - now it's hard to tell without the whole project's context so take this remarks with the required grain of salt ;)
How can i used the rt function, as i understand leading & trailing underscores __and__() is available for native python objects or you wan't to customize behavior in specific situations. how can the user take advantages of it . For ex: in the below code can i use this function at all,
class A(object):
def __rt__(self,r):
return "Yes special functions"
a=A()
print dir(a)
print a.rt('1') # AttributeError: 'A' object has no attribute 'rt'
But
class Room(object):
def __init__(self):
self.people = []
def add(self, person):
self.people.append(person)
def __len__(self):
return len(self.people)
room = Room()
room.add("Igor")
print len(room) #prints 1
Python doesn't translate one name into another. Specific operations will under the covers call a __special_method__ if it has been defined. For example, the __and__ method is called by Python to hook into the & operator, because the Python interpreter explicitly looks for that method and documented how it should be used.
In other words, calling object.rt() is not translated to object.__rt__() anywhere, not automatically.
Note that Python reserves such names; future versions of Python may use that name for a specific purpose and then your existing code using a __special_method__ name for your own purposes would break.
From the Reserved classes of identifiers section:
__*__
System-defined names. These names are defined by the interpreter and its implementation (including the standard library). Current system names are discussed in the Special method names section and elsewhere. More will likely be defined in future versions of Python. Any use of __*__ names, in any context, that does not follow explicitly documented use, is subject to breakage without warning.
You can ignore that advice of course. In that case, you'll have to write code that actually calls your method:
class SomeBaseClass:
def rt(self):
"""Call the __rt__ special method"""
try:
return self.__rt__()
except AttributeError:
raise TypeError("The object doesn't support this operation")
and subclass from SomeBaseClass.
Again, Python won't automatically call your new methods. You still need to actually write such code.
Because there are builtin methods that you can overriden and then you can use them, ex __len__ -> len(), __str__ -> str() and etc.
Here is the list of these functions
The following methods can be defined to customize the meaning of attribute access (use of, assignment to, or deletion of x.name) for class instances.
I'm new in programming so please don't kill me for asking stupid questions.
I've been trying to understand all that class business in Python and I got to the point where could not find answer for my question just by google it.
In my program I need to call a class from within other class based on string returned by function. I found two solutions: one by using getattr() and second one by using globals() / locals().
Decided to go for second solution and got it working but I'm really don't understand how it's working.
So there is the code example:
class Test(object):
def __init__(self):
print "WORKS!"
room = globals()['Test']
room()
type(room()) gives:
<class '__main__.Test'>
type(room) gives:
<type 'type'> # What????
It looks like room() is a class object, but shouldn't that be room instead of room()?
Please help me because it is a little bit silly if I write a code which I don't understand myself.
What happens here is the following:
class Test(object):
def __init__(self):
print "WORKS!"
room = globals()['Test']
Here you got Test as room the way you wanted. Verify this:
room is Test
should give True.
type(room()) gives:
<class '__main__.Test'>
You do one step an go it backwards: room() returns the same as Test() would - an instance of that class. type() "undoes" this step resp. gets the type of the object - this is, of course, Test.
type(room) gives:
<type 'type'> # What????
Of course - it is the type of a (new style) class. The same as type(Test).
Be aware, however, that for
In my program I need to call a class from within other class based on string returned by function. I found two solutions: one by using getattr() and second one by using globals() / locals().
it could be better to create an explicitly separate dict. Here you have full control over which objects/classes/... are allowed in that context and which are not.
First of all, I'd go with getattr instead.
In your example, room equals Test and is a class. Its type is type.
When you call room(), you instantiate Test, so room() evaluates to an instance of Test, whose type is Test.
Classes are objects too, in Python. All this does:
class Test(object):
def __init__(self):
print "WORKS!"
is create a class object and bind it to the name Test. Much as this:
x = []
creates a list object and binds it to the name x.
Test() isn't magic syntax for creating an instance. The Test is perfectly ordinary variable lookup, and the () is perfectly ordinary "call with empty arguments". It just so happens that calling a class will create an instance of that class.
If follows then that your problem of instantiating a class chosen based on having the name of the class as a string boils down to the much simpler problem of finding an object stored in a variable. It's exactly the same problem as getting that list bound to the name x, given the string "x". Once you've got a reference to the class in any old variable, you can simply call it to create your instance.
globals() returns a dictionary mapping the names of globals to their values. So globals()['Test'] will get you the class Test just as easily as globals()['x'] will get you the list. However it's usually not considered great style to use globals() like this; your module probably contains a large number of callables (including a bunch imported from other modules) that you don't want to be accidentally invoked if the function can be made to return their name. Given that classes are just ordinary objects, you can put them in a dictionary of your own making:
classes = {
'Test': Test,
'SomethingElse': Something,
...
}
This involves a bit more typing, but it's also easier to see what the intended usage is, and it gives you a bit more flexibility, since you can also easily pass this dictionary to other modules and have the instantiation take place elsewhere (you could do that with globals(), but then you're getting very weird).
Now, for the type(room) being type. Again, this is just a simple consequence of the fact that classes themselves are also objects. If a class is an object, then it should also be an instance of some class. What class is that? type, the "type of types". Much as any class defines the common behaviour of all its instances, the class type defines the common behaviour of all classes.
And just to make your brain hurt, type is an instance of itself (since type is also a class, and type is the class of classes). And it's a subclass of object (since all type instances are object instances, but not all object instances are type instances), and also an instance of object (since object is the root class of which everything is an instance).
You can generally ignore type as an advanced topic, however. :)
tl;dr: How come property decorators work with class-level function definitions, but not with module-level definitions?
I was applying property decorators to some module-level functions, thinking they would allow me to invoke the methods by mere attribute lookup.
This was particularly tempting because I was defining a set of configuration functions, like get_port, get_hostname, etc., all of which could have been replaced with their simpler, more terse property counterparts: port, hostname, etc.
Thus, config.get_port() would just be the much nicer config.port
I was surprised when I found the following traceback, proving that this was not a viable option:
TypeError: int() argument must be a string or a number, not 'property'
I knew I had seen some precedant for property-like functionality at module-level, as I had used it for scripting shell commands using the elegant but hacky pbs library.
The interesting hack below can be found in the pbs library source code. It enables the ability to do property-like attribute lookups at module-level, but it's horribly, horribly hackish.
# this is a thin wrapper around THIS module (we patch sys.modules[__name__]).
# this is in the case that the user does a "from pbs import whatever"
# in other words, they only want to import certain programs, not the whole
# system PATH worth of commands. in this case, we just proxy the
# import lookup to our Environment class
class SelfWrapper(ModuleType):
def __init__(self, self_module):
# this is super ugly to have to copy attributes like this,
# but it seems to be the only way to make reload() behave
# nicely. if i make these attributes dynamic lookups in
# __getattr__, reload sometimes chokes in weird ways...
for attr in ["__builtins__", "__doc__", "__name__", "__package__"]:
setattr(self, attr, getattr(self_module, attr))
self.self_module = self_module
self.env = Environment(globals())
def __getattr__(self, name):
return self.env[name]
Below is the code for inserting this class into the import namespace. It actually patches sys.modules directly!
# we're being run as a stand-alone script, fire up a REPL
if __name__ == "__main__":
globs = globals()
f_globals = {}
for k in ["__builtins__", "__doc__", "__name__", "__package__"]:
f_globals[k] = globs[k]
env = Environment(f_globals)
run_repl(env)
# we're being imported from somewhere
else:
self = sys.modules[__name__]
sys.modules[__name__] = SelfWrapper(self)
Now that I've seen what lengths pbs has to go through, I'm left wondering why this facility of Python isn't built into the language directly. The property decorator in particular seems like a natural place to add such functionality.
Is there any partiuclar reason or motivation for why this isn't built directly in?
This is related to a combination of two factors: first, that properties are implemented using the descriptor protocol, and second that modules are always instances of a particular class rather than being instantiable classes.
This part of the descriptor protocol is implemented in object.__getattribute__ (the relevant code is PyObject_GenericGetAttr starting at line 1319). The lookup rules go like this:
Search through the class mro for a type dictionary that has name
If the first matching item is a data descriptor, call its __get__ and return its result
If name is in the instance dictionary, return its associated value
If there was a matching item from the class dictionaries and it was a non-data descriptor, call its __get__ and return the result
If there was a matching item from the class dictionaries, return it
raise AttributeError
The key to this is at number 3 - if name is found in the instance dictionary (as it will be with modules), then its value will just be returned - it won't be tested for descriptorness, and its __get__ won't be called. This leads to this situation (using Python 3):
>>> class F:
... def __getattribute__(self, attr):
... print('hi')
... return object.__getattribute__(self, attr)
...
>>> f = F()
>>> f.blah = property(lambda: 5)
>>> f.blah
hi
<property object at 0xbfa1b0>
You can see that .__getattribute__ is being invoked, but isn't treating f.blah as a descriptor.
It is likely that the reason for the rules being structured this way is an explicit tradeoff between the usefulness of allowing descriptors on instances (and, therefore, in modules) and the extra code complexity that this would lead to.
Properties are a feature specific to classes (new-style classes specifically) so by extension the property decorator can only be applied to class methods.
A new-style class is one that derives from object, i.e. class Foo(object):
Further info: Can modules have properties the same way that objects can?
I'd like to serialize Python objects to and from the plist format (this can be done with plistlib). My idea was to write a class PlistObject which wraps other objects:
def __init__(self, anObject):
self.theObject = anObject
and provides a "write" method:
def write(self, pathOrFile):
plistlib.writeToPlist(self.theObject.__dict__, pathOrFile)
Now it would be nice if the PlistObject behaved just like wrapped object itself, meaning that all attributes and methods are somehow "forwarded" to the wrapped object. I realize that the methods __getattr__ and __setattr__ can be used for complex attribute operations:
def __getattr__(self, name):
return self.theObject.__getattr__(name)
But then of course I run into the problem that the constructor now produces an infinite recursion, since also self.theObject = anObject tries to access the wrapped object.
How can I avoid this? If the whole idea seems like a bad one, tell me too.
Unless I'm missing something, this will work just fine:
def __getattr__(self, name):
return getattr(self.theObject, name)
Edit: for those thinking that the lookup of self.theObject will result in an infinite recursive call to __getattr__, let me show you:
>>> class Test:
... a = "a"
... def __init__(self):
... self.b = "b"
... def __getattr__(self, name):
... return 'Custom: %s' % name
...
>>> Test.a
'a'
>>> Test().a
'a'
>>> Test().b
'b'
>>> Test().c
'Custom: c'
__getattr__ is only called as a last resort. Since theObject can be found in __dict__, no issues arise.
But then of course I run into the problem that the constructor now produces an infinite recursion, since also self.theObject = anObject tries to access the wrapped object.
That's why the manual suggests that you do this for all "real" attribute accesses.
theobj = object.__getattribute__(self, "theObject")
I'm glad to see others have been able to help you with the recursive call to __getattr__. Since you've asked for comments on the general approach of serializing to plist, I just wanted to chime in with a few thoughts.
Python's plist implementation handles basic types only, and provides no extension mechanism for you to instruct it on serializing/deserializing complex types. If you define a custom class, for example, writePlist won't be able to help, as you've discovered since you're passing the instance's __dict__ for serialization.
This has a couple implications:
You won't be able to use this to serialize any objects that contain other objects of non-basic type without converting them to a __dict__, and so-on recursively for the entire network graph.
If you roll your own network graph walker to serialize all non-basic objects that can be reached, you'll have to worry about circles in the graph where one object has another in a property, which in turn holds a reference back to the first, etc etc.
Given then, you may wish to look at pickle instead as it can handle all of these and more. If you need the plist format for other reasons, and you're sure you can stick to "simple" object dicts, then you may wish to just use a simple function... trying to have the PlistObject mock every possible function in the contained object is an onion with potentially many layers as you need to handle all the possibilities of the wrapped instance.
Something as simple as this may be more pythonic, and keep the usability of the wrapped object simpler by not wrapping it in the first place:
def to_plist(obj, f_handle):
writePlist(obj.__dict__, f_handle)
I know that doesn't seem very sexy, but it is a lot more maintainable in my opinion than a wrapper given the severe limits of the plist format, and certainly better than artificially forcing all objects in your application to inherit from a common base class when there's nothing in your business domain that actually indicates those disparate objects are related.