We Keep Coding

Where is the __bases__ attribute defined?

​‫I define a python class in python interpreter
class A:
pass
I get base class of A using A.__bases__, it shows
(object,)
but when I enter dir(A), the output don't contain __bases__ attribute, then I try dir(object), __bases__ is not found either, where does the __bases__ come from?

The __bases__ attribute in a class is implemented by a descriptor in the metaclass, type. You have to be a little careful though, since type, as one of the building blocks of the Python object model, is an instance of itself, and so type.__bases__ doesn't do what you would want for introspection.
Try this:
descriptor = type.__dict__['__bases__']
print(descriptor, type(descriptor))
You can reproduce the same kind of thing with your own descriptors:
class MyMeta(type):
#property # a property is a descriptor
def foo(cls):
return "foo"
class MyClass(metaclass=MyMeta):
pass
Now if you access MyClass.foo you'll get the string foo. But you won't see foo in the variables defined in MyClass (if you check with vars or dir). Nor can you access it through an instance of MyClass (my_obj = MyClass(); my_obj.foo raises an AttributeError).

it is a special attribute akin to __name__ or __dict__. While the result of function dir actually depends on the implementation of __dir__ function.
You might want to look it on the doc here https://docs.python.org/3/reference/datamodel.html

How do overridden method calls from base-class methods work?

According to the docs on inheritance:
Derived classes may override methods of their base classes. Because methods have no special privileges when calling other methods of the same object, a method of a base class that calls another method defined in the same base class may end up calling a method of a derived class that overrides it.
How does that happen? Can someone illustrate this concept with a simple example?

Here's the example you requested. This prints chocolate.
class Base:
def foo(self):
print("foo")
def bar(self):
self.foo()
class Derived(Base):
def foo(self):
print("chocolate")
d = Derived()
d.bar() # prints "chocolate"
The string chocolate is printed instead of foo because Derived overrides the foo() function. Even though bar() is defined in Base, it ends up calling the Derived implementation of foo() instead of the Base implementation.

How does it work?
When an attribute look-up is performed on an instance of the class, the class dictionary and the dictionaries of its base classes are searched in a certain order (see: Method Resolution Order) for the appropriate method. What is found first is going to get called.
Using the following Spam example:
class Spam:
def produce_spam(self):
print("spam")
def get_spam(self):
self.produce_spam()
class SuperSpam(Spam):
def produce_spam(self):
print("super spam")
Spam defines the functions produce_spam and get_spam. These live in its Spam.__dict__ (class namespace). The sub-class SuperSpam, by means of inheritance, has access to both these methods. SuperSpam.produce_spam doesn't replace Spam.produce_spam, it is simply found first when the look-up for the name 'produce_spam' is made on one of its instances.
Essentially, the result of inheritance is that the dictionaries of any base classes are also going to get searched if, after an attribute look-up on the sub-class is made, the attribute isn't found in the sub-class's dictionary.
When the function get_spam is first invoked with:
s = SuperSpam()
s.get_spam()
the sequence of events roughly goes like this:
Look into SuperSpams __dict__ for get_spam.
Since it isn't found in SuperSpams __dict__ look into the dictionaries of it's base classes (mro chain).
Spam is next in the mro chain, so get_spam is found in Spam's dictionary.
Now, when produce_spam is looked up in the body of get_spam with self.produce_spam, the sequence is much shorter:
Look into SuperSpam's (self) __dict__ for produce_spam.
Find it, get it and call it.
produce_spam is found in the __dict__ first so that gets fetched.

class Base():
def m1(self):
return self.m2()
def m2(self):
return 'base'
class Sub(Base):
def m2(self):
return 'sub'
b = Base()
s = Sub()
print(b.m1(), s.m1())
prints "base sub"

To illustrate how it works consider these two classes:
class Parent(object):
def eat(self):
print("I don't want to eat that {}.".format(self.takefrompocket()))
def takefrompocket(self):
return 'apple'
def __getattribute__(self, name):
print('Looking for:', name)
method_to_use = object.__getattribute__(self, name)
print('Found method:', method_to_use)
return method_to_use
class Child(Parent):
def takefrompocket(self):
return 'salad'
The __getattribute__ method is responsible in new-style-classes (like all classes in python3) for the attribute lookup. It is just implemented to print what each lookup does - normally you don't want to and shouldn't implement it yourself. The lookup follows pythons method resolution order (MRO) just if you are really interested.
>>> some_kid = Child()
>>> some_kid.eat()
Looking for: eat
Found method: <bound method Parent.eat of <__main__.Child object at 0x0000027BCA4EEA58>>
Looking for: takefrompocket
Found method: <bound method Child.takefrompocket of <__main__.Child object at 0x0000027BCA4EEA58>>
I don't want to eat that salad.
So when you want to use eat then it uses Parent.eat in this example. But self.takefrompocket is used from Child.
>>> some_parent = Parent()
>>> some_parent.eat()
Looking for: eat
Found method: <bound method Parent.eat of <__main__.Parent object at 0x0000027BCA4EE358>>
Looking for: takefrompocket
Found method: <bound method Parent.takefrompocket of <__main__.Parent object at 0x0000027BCA4EE358>>
I don't want to eat that apple.
Here both methods are taken from Parent. Inherited classes don't (generally) interfere with their ancestors!

If your child class doesn't implement the method, raise an exception!
class Base(object):
def something (self):
raise ('Not implemented')

How to Inherit multiple classes in python dynamically [duplicate]

This article has a snippet showing usage of __bases__ to dynamically change the inheritance hierarchy of some Python code, by adding a class to an existing classes collection of classes from which it inherits. Ok, that's hard to read, code is probably clearer:
class Friendly:
def hello(self):
print 'Hello'
class Person: pass
p = Person()
Person.__bases__ = (Friendly,)
p.hello() # prints "Hello"
That is, Person doesn't inherit from Friendly at the source level, but rather this inheritance relation is added dynamically at runtime by modification of the __bases__attribute of the Person class. However, if you change Friendly and Person to be new style classes (by inheriting from object), you get the following error:
TypeError: __bases__ assignment: 'Friendly' deallocator differs from 'object'
A bit of Googling on this seems to indicate some incompatibilities between new-style and old style classes in regards to changing the inheritance hierarchy at runtime. Specifically: "New-style class objects don't support assignment to their bases attribute".
My question, is it possible to make the above Friendly/Person example work using new-style classes in Python 2.7+, possibly by use of the __mro__ attribute?
Disclaimer: I fully realise that this is obscure code. I fully realize that in real production code tricks like this tend to border on unreadable, this is purely a thought experiment, and for funzies to learn something about how Python deals with issues related to multiple inheritance.

Ok, again, this is not something you should normally do, this is for informational purposes only.
Where Python looks for a method on an instance object is determined by the __mro__ attribute of the class which defines that object (the M ethod R esolution O rder attribute). Thus, if we could modify the __mro__ of Person, we'd get the desired behaviour. Something like:
setattr(Person, '__mro__', (Person, Friendly, object))
The problem is that __mro__ is a readonly attribute, and thus setattr won't work. Maybe if you're a Python guru there's a way around that, but clearly I fall short of guru status as I cannot think of one.
A possible workaround is to simply redefine the class:
def modify_Person_to_be_friendly():
# so that we're modifying the global identifier 'Person'
global Person
# now just redefine the class using type(), specifying that the new
# class should inherit from Friendly and have all attributes from
# our old Person class
Person = type('Person', (Friendly,), dict(Person.__dict__))
def main():
modify_Person_to_be_friendly()
p = Person()
p.hello() # works!
What this doesn't do is modify any previously created Person instances to have the hello() method. For example (just modifying main()):
def main():
oldperson = Person()
ModifyPersonToBeFriendly()
p = Person()
p.hello()
# works! But:
oldperson.hello()
# does not
If the details of the type call aren't clear, then read e-satis' excellent answer on 'What is a metaclass in Python?'.

I've been struggling with this too, and was intrigued by your solution, but Python 3 takes it away from us:
AttributeError: attribute '__dict__' of 'type' objects is not writable
I actually have a legitimate need for a decorator that replaces the (single) superclass of the decorated class. It would require too lengthy a description to include here (I tried, but couldn't get it to a reasonably length and limited complexity -- it came up in the context of the use by many Python applications of an Python-based enterprise server where different applications needed slightly different variations of some of the code.)
The discussion on this page and others like it provided hints that the problem of assigning to __bases__ only occurs for classes with no superclass defined (i.e., whose only superclass is object). I was able to solve this problem (for both Python 2.7 and 3.2) by defining the classes whose superclass I needed to replace as being subclasses of a trivial class:
## T is used so that the other classes are not direct subclasses of object,
## since classes whose base is object don't allow assignment to their __bases__ attribute.
class T: pass
class A(T):
def __init__(self):
print('Creating instance of {}'.format(self.__class__.__name__))
## ordinary inheritance
class B(A): pass
## dynamically specified inheritance
class C(T): pass
A() # -> Creating instance of A
B() # -> Creating instance of B
C.__bases__ = (A,)
C() # -> Creating instance of C
## attempt at dynamically specified inheritance starting with a direct subclass
## of object doesn't work
class D: pass
D.__bases__ = (A,)
D()
## Result is:
## TypeError: __bases__ assignment: 'A' deallocator differs from 'object'

I can not vouch for the consequences, but that this code does what you want at py2.7.2.
class Friendly(object):
def hello(self):
print 'Hello'
class Person(object): pass
# we can't change the original classes, so we replace them
class newFriendly: pass
newFriendly.__dict__ = dict(Friendly.__dict__)
Friendly = newFriendly
class newPerson: pass
newPerson.__dict__ = dict(Person.__dict__)
Person = newPerson
p = Person()
Person.__bases__ = (Friendly,)
p.hello() # prints "Hello"
We know that this is possible. Cool. But we'll never use it!

Right of the bat, all the caveats of messing with class hierarchy dynamically are in effect.
But if it has to be done then, apparently, there is a hack that get's around the "deallocator differs from 'object" issue when modifying the __bases__ attribute for the new style classes.
You can define a class object
class Object(object): pass
Which derives a class from the built-in metaclass type.
That's it, now your new style classes can modify the __bases__ without any problem.
In my tests this actually worked very well as all existing (before changing the inheritance) instances of it and its derived classes felt the effect of the change including their mro getting updated.

I needed a solution for this which:
Works with both Python 2 (>= 2.7) and Python 3 (>= 3.2).
Lets the class bases be changed after dynamically importing a dependency.
Lets the class bases be changed from unit test code.
Works with types that have a custom metaclass.
Still allows unittest.mock.patch to function as expected.
Here's what I came up with:
def ensure_class_bases_begin_with(namespace, class_name, base_class):
""" Ensure the named class's bases start with the base class.
:param namespace: The namespace containing the class name.
:param class_name: The name of the class to alter.
:param base_class: The type to be the first base class for the
newly created type.
:return: ``None``.
Call this function after ensuring `base_class` is
available, before using the class named by `class_name`.
"""
existing_class = namespace[class_name]
assert isinstance(existing_class, type)
bases = list(existing_class.__bases__)
if base_class is bases[0]:
# Already bound to a type with the right bases.
return
bases.insert(0, base_class)
new_class_namespace = existing_class.__dict__.copy()
# Type creation will assign the correct ‘__dict__’ attribute.
del new_class_namespace['__dict__']
metaclass = existing_class.__metaclass__
new_class = metaclass(class_name, tuple(bases), new_class_namespace)
namespace[class_name] = new_class
Used like this within the application:
# foo.py
# Type `Bar` is not available at first, so can't inherit from it yet.
class Foo(object):
__metaclass__ = type
def __init__(self):
self.frob = "spam"
def __unicode__(self): return "Foo"
# … later …
import bar
ensure_class_bases_begin_with(
namespace=globals(),
class_name=str('Foo'), # `str` type differs on Python 2 vs. 3.
base_class=bar.Bar)
Use like this from within unit test code:
# test_foo.py
""" Unit test for `foo` module. """
import unittest
import mock
import foo
import bar
ensure_class_bases_begin_with(
namespace=foo.__dict__,
class_name=str('Foo'), # `str` type differs on Python 2 vs. 3.
base_class=bar.Bar)
class Foo_TestCase(unittest.TestCase):
""" Test cases for `Foo` class. """
def setUp(self):
patcher_unicode = mock.patch.object(
foo.Foo, '__unicode__')
patcher_unicode.start()
self.addCleanup(patcher_unicode.stop)
self.test_instance = foo.Foo()
patcher_frob = mock.patch.object(
self.test_instance, 'frob')
patcher_frob.start()
self.addCleanup(patcher_frob.stop)
def test_instantiate(self):
""" Should create an instance of `Foo`. """
instance = foo.Foo()

The above answers are good if you need to change an existing class at runtime. However, if you are just looking to create a new class that inherits by some other class, there is a much cleaner solution. I got this idea from https://stackoverflow.com/a/21060094/3533440, but I think the example below better illustrates a legitimate use case.
def make_default(Map, default_default=None):
"""Returns a class which behaves identically to the given
Map class, except it gives a default value for unknown keys."""
class DefaultMap(Map):
def __init__(self, default=default_default, **kwargs):
self._default = default
super().__init__(**kwargs)
def __missing__(self, key):
return self._default
return DefaultMap
DefaultDict = make_default(dict, default_default='wug')
d = DefaultDict(a=1, b=2)
assert d['a'] is 1
assert d['b'] is 2
assert d['c'] is 'wug'
Correct me if I'm wrong, but this strategy seems very readable to me, and I would use it in production code. This is very similar to functors in OCaml.

This method isn't technically inheriting during runtime, since __mro__ can't be changed. But what I'm doing here is using __getattr__ to be able to access any attributes or methods from a certain class. (Read comments in order of numbers placed before the comments, it makes more sense)
class Sub:
def __init__(self, f, cls):
self.f = f
self.cls = cls
# 6) this method will pass the self parameter
# (which is the original class object we passed)
# and then it will fill in the rest of the arguments
# using *args and **kwargs
def __call__(self, *args, **kwargs):
# 7) the multiple try / except statements
# are for making sure if an attribute was
# accessed instead of a function, the __call__
# method will just return the attribute
try:
return self.f(self.cls, *args, **kwargs)
except TypeError:
try:
return self.f(*args, **kwargs)
except TypeError:
return self.f
# 1) our base class
class S:
def __init__(self, func):
self.cls = func
def __getattr__(self, item):
# 5) we are wrapping the attribute we get in the Sub class
# so we can implement the __call__ method there
# to be able to pass the parameters in the correct order
return Sub(getattr(self.cls, item), self.cls)
# 2) class we want to inherit from
class L:
def run(self, s):
print("run" + s)
# 3) we create an instance of our base class
# and then pass an instance (or just the class object)
# as a parameter to this instance
s = S(L) # 4) in this case, I'm using the class object
s.run("1")
So this sort of substitution and redirection will simulate the inheritance of the class we wanted to inherit from. And it even works with attributes or methods that don't take any parameters.

python dynamic class names

Trying to instantiate a class based on a string value and... failing. The parser object below is a dict, in the example let's say we have one called foo and here parser['name'] is 'foo':
obj = parser['name']()
Fails, yielding TypeError: 'str' object is not callable. But, since I have:
class foo:
def __init__(self():
print 'Hello'
And if I do obj = foo() it works fine and creates the correct object. Also, calling obj = type(parser['name'])() doesn't work.
How to resolve this? Update: I don't really want to use a mapping system: the names of these classes are defined INI files, and parsed that way, so they will be strings..

classmap = {
'foo': foo
}
obj = classmap[parser['name']]()

As answered in:
Python dynamic class names
There is an easier way to do this if you know which module the classes are defined in, for example:
getattr(my_module, my_class_name)()

Don't use strings:
parser = {}
class foo:
pass
parser['foo'] = foo
obj = parser['foo']()

You can use a metaclass that stores a dict of known classes:
# a metaclass
class Registry(type):
# store all the types we know
registered = {}
def __new__(cls, name, bases, attrs):
# create the new type
newtype = super(Registry, cls).__new__(cls, name, bases, attrs)
# store it
cls.registered[name] = newtype
return newtype
#classmethod
def class_by_name(cls, name):
# get a class from the registerd classes
return cls.registered[name]
# arbitrary base class for every class that should be in the Register
class Registered(object):
__metaclass__ = Registry
# some classes
class Foo(Registered):
pass
class Bar(Foo):
pass
# get the class object:
print Registry.class_by_name('Foo') # <class '__main__.Foo'>
# it can be instanciated too ofc:
print Registry.class_by_name('Bar')() # <__main__.Bar object at 0x01F0F9F0>
But not everyone understands metaclasses, so you might want to avoid them to prevent any confusion. They can be useful for stuff like this, but as you can see from the other answers, there are plenty other ways to do it.

The type(name, bases, dict) built-in function is the correct way to dynamically construct classes--especially when given strings for class names. See the documentation here: http://docs.python.org/library/functions.html#type
In you particular example, it might look like this:
>>> def init(self):
... print 'Hello'
...
>>> Foo = type('Foo', (object,), {'__init__': init})
>>> foo = Foo()
Hello

You could use inspect to create the class map:
def get_classes():
classes = {}
for name, obj in inspect.getmembers(sys.modules[__name__]):
if inspect.isclass(obj):
classes[name] = obj
return classes
Then instantiate a class
>>> classes = get_classes()
>>> c = classes['ClassName']
>>> c
<class ClassName...>
Mostly nicked from How can I get a list of all classes within current module in Python?

In response to your update: The only way to do this is with a mapping system. If you don't want to use an explicit mapping system, then you can use one of Python's built-in mapping systems, though it's much less nice and explicit in my opinion.
obj = globals()[parser['name']]()
will access the global object of with name parser['name'] == 'foo'. If this happens to be a class (or a class that you actually want instantiated based on user input), then you should be good to go. Otherwise, you will have to build logic around it to whitelist the classes that you actually want.
If the classes are coming from a module, then you can use that module's __dict__ attribute to the same effect.
obj = somemodule.__dict__[parser['name']]()
The same caveats apply to this situation as the previous one. It's really better to just use an explicit mapping

I would use a map of class name to class object like everyone else is saying. You can initialize it using statements like parser[foo.__name__] = foo. If you do not want to use a mapping object, then you will have to fall back to using the eval function like the following:
>>> class foo:
... pass
...
>>> klass_name = 'foo'
>>> klass_inst = eval(klass_name)
>>> klass_inst
<class __main__.foo at 0x1004b2b90>
>>> inst = klass_inst()
>>> inst
<__main__.foo instance at 0x1004d2680>
>>>
Of course if you want to use classes that are embedded within a package, then the package will have to be imported before you do the eval. You really should build a mapping object so that you can limit the classes that can be accessed using this code.

How can I pickle a dynamically created nested class in python?

I have a nested class:
class WidgetType(object):
class FloatType(object):
pass
class TextType(object):
pass
.. and an object that refers the nested class type (not an instance of it) like this
class ObjectToPickle(object):
def __init__(self):
self.type = WidgetType.TextType
Trying to serialize an instance of the ObjectToPickle class results in:
PicklingError: Can't pickle <class
'setmanager.app.site.widget_data_types.TextType'>
Is there a way to pickle nested classes in python?

I know this is a very old question, but I have never explicitly seen a satisfactory solution to this question other than the obvious, and most likely correct, answer to re-structure your code.
Unfortunately, it is not always practical to do such a thing, in which case as a very last resort, it is possible to pickle instances of classes which are defined inside another class.
The python documentation for the __reduce__ function states that you can return
A callable object that will be called to create the initial version of the object. The next element of the tuple will provide arguments for this callable.
Therefore, all you need is an object which can return an instance of the appropriate class. This class must itself be picklable (hence, must live on the __main__ level), and could be as simple as:
class _NestedClassGetter(object):
"""
When called with the containing class as the first argument,
and the name of the nested class as the second argument,
returns an instance of the nested class.
"""
def __call__(self, containing_class, class_name):
nested_class = getattr(containing_class, class_name)
# return an instance of a nested_class. Some more intelligence could be
# applied for class construction if necessary.
return nested_class()
All that is left therefore, is to return the appropriate arguments in a __reduce__ method on FloatType:
class WidgetType(object):
class FloatType(object):
def __reduce__(self):
# return a class which can return this class when called with the
# appropriate tuple of arguments
return (_NestedClassGetter(), (WidgetType, self.__class__.__name__, ))
The result is a class which is nested but instances can be pickled (further work is needed to dump/load the __state__ information, but this is relatively straightforward as per the __reduce__ documentation).
This same technique (with slight code modifications) can be applied for deeply nested classes.
A fully worked example:
import pickle
class ParentClass(object):
class NestedClass(object):
def __init__(self, var1):
self.var1 = var1
def __reduce__(self):
state = self.__dict__.copy()
return (_NestedClassGetter(),
(ParentClass, self.__class__.__name__, ),
state,
)
class _NestedClassGetter(object):
"""
When called with the containing class as the first argument,
and the name of the nested class as the second argument,
returns an instance of the nested class.
"""
def __call__(self, containing_class, class_name):
nested_class = getattr(containing_class, class_name)
# make an instance of a simple object (this one will do), for which we can change the
# __class__ later on.
nested_instance = _NestedClassGetter()
# set the class of the instance, the __init__ will never be called on the class
# but the original state will be set later on by pickle.
nested_instance.__class__ = nested_class
return nested_instance
if __name__ == '__main__':
orig = ParentClass.NestedClass(var1=['hello', 'world'])
pickle.dump(orig, open('simple.pickle', 'w'))
pickled = pickle.load(open('simple.pickle', 'r'))
print type(pickled)
print pickled.var1
My final note on this is to remember what the other answers have said:
If you are in a position to do so, consider re-factoring your code to
avoid the nested classes in the first place.

The pickle module is trying to get the TextType class from the module. But since the class is nested it doesn't work. jasonjs's suggestion will work.
Here are the lines in pickle.py responsible for the error message:
try:
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
except (ImportError, KeyError, AttributeError):
raise PicklingError(
"Can't pickle %r: it's not found as %s.%s" %
(obj, module, name))
klass = getattr(mod, name) will not work in the nested class case of course. To demonstrate what is going on try to add these lines before pickling the instance:
import sys
setattr(sys.modules[__name__], 'TextType', WidgetType.TextType)
This code adds TextType as an attribute to the module. The pickling should work just fine. I don't advice you to use this hack though.

If you use dill instead of pickle, it works.
>>> import dill
>>>
>>> class WidgetType(object):
... class FloatType(object):
... pass
... class TextType(object):
... pass
...
>>> class ObjectToPickle(object):
... def __init__(self):
... self.type = WidgetType.TextType
...
>>> x = ObjectToPickle()
>>>
>>> _x = dill.dumps(x)
>>> x_ = dill.loads(_x)
>>> x_
<__main__.ObjectToPickle object at 0x10b20a250>
>>> x_.type
<class '__main__.TextType'>
Get dill here: https://github.com/uqfoundation/dill

In Sage (www.sagemath.org), we have many instances of this pickling issue. The way we decided to systematically solve it is to put the outer class inside a specific metaclass whose goal is to implement and hide the hack. Note that this automatically propagate through nested classes if there are several level of nesting.

Pickle only works with classes defined in module scope (top level). In this case, it looks like you could define the nested classes in module scope and then set them as properties on WidgetType, assuming there's a reason not to just reference TextType and FloatType in your code. Or, import the module they're in and use widget_type.TextType and widget_type.FloatType.

Nadia's answer is pretty complete - it is practically not something you want to be doing; are you sure you can't use inheritance in WidgetTypes instead of nested classes?
The only reason to use nested classes is to encapsulate classes working together closely, your specific example looks like an immediate inheritance candidate to me - there is no benefit in nesting WidgetType classes together; put them in a module and inherit from the base WidgetType instead.

This seems to work fine in newer versions of Python. I tried it in v3.8 and it was able to pickle and unpickle the nested class.