What is the purpose of __slots__ in Python — especially with respect to when I would want to use it, and when not?
In Python, what is the purpose of __slots__ and what are the cases one should avoid this?
TLDR:
The special attribute __slots__ allows you to explicitly state which instance attributes you expect your object instances to have, with the expected results:
faster attribute access.
space savings in memory.
The space savings is from
Storing value references in slots instead of __dict__.
Denying __dict__ and __weakref__ creation if parent classes deny them and you declare __slots__.
Quick Caveats
Small caveat, you should only declare a particular slot one time in an inheritance tree. For example:
class Base:
__slots__ = 'foo', 'bar'
class Right(Base):
__slots__ = 'baz',
class Wrong(Base):
__slots__ = 'foo', 'bar', 'baz' # redundant foo and bar
Python doesn't object when you get this wrong (it probably should), problems might not otherwise manifest, but your objects will take up more space than they otherwise should. Python 3.8:
>>> from sys import getsizeof
>>> getsizeof(Right()), getsizeof(Wrong())
(56, 72)
This is because the Base's slot descriptor has a slot separate from the Wrong's. This shouldn't usually come up, but it could:
>>> w = Wrong()
>>> w.foo = 'foo'
>>> Base.foo.__get__(w)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: foo
>>> Wrong.foo.__get__(w)
'foo'
The biggest caveat is for multiple inheritance - multiple "parent classes with nonempty slots" cannot be combined.
To accommodate this restriction, follow best practices: Factor out all but one or all parents' abstraction which their concrete class respectively and your new concrete class collectively will inherit from - giving the abstraction(s) empty slots (just like abstract base classes in the standard library).
See section on multiple inheritance below for an example.
Requirements:
To have attributes named in __slots__ to actually be stored in slots instead of a __dict__, a class must inherit from object (automatic in Python 3, but must be explicit in Python 2).
To prevent the creation of a __dict__, you must inherit from object and all classes in the inheritance must declare __slots__ and none of them can have a '__dict__' entry.
There are a lot of details if you wish to keep reading.
Why use __slots__: Faster attribute access.
The creator of Python, Guido van Rossum, states that he actually created __slots__ for faster attribute access.
It is trivial to demonstrate measurably significant faster access:
import timeit
class Foo(object): __slots__ = 'foo',
class Bar(object): pass
slotted = Foo()
not_slotted = Bar()
def get_set_delete_fn(obj):
def get_set_delete():
obj.foo = 'foo'
obj.foo
del obj.foo
return get_set_delete
and
>>> min(timeit.repeat(get_set_delete_fn(slotted)))
0.2846834529991611
>>> min(timeit.repeat(get_set_delete_fn(not_slotted)))
0.3664822799983085
The slotted access is almost 30% faster in Python 3.5 on Ubuntu.
>>> 0.3664822799983085 / 0.2846834529991611
1.2873325658284342
In Python 2 on Windows I have measured it about 15% faster.
Why use __slots__: Memory Savings
Another purpose of __slots__ is to reduce the space in memory that each object instance takes up.
My own contribution to the documentation clearly states the reasons behind this:
The space saved over using __dict__ can be significant.
SQLAlchemy attributes a lot of memory savings to __slots__.
To verify this, using the Anaconda distribution of Python 2.7 on Ubuntu Linux, with guppy.hpy (aka heapy) and sys.getsizeof, the size of a class instance without __slots__ declared, and nothing else, is 64 bytes. That does not include the __dict__. Thank you Python for lazy evaluation again, the __dict__ is apparently not called into existence until it is referenced, but classes without data are usually useless. When called into existence, the __dict__ attribute is a minimum of 280 bytes additionally.
In contrast, a class instance with __slots__ declared to be () (no data) is only 16 bytes, and 56 total bytes with one item in slots, 64 with two.
For 64 bit Python, I illustrate the memory consumption in bytes in Python 2.7 and 3.6, for __slots__ and __dict__ (no slots defined) for each point where the dict grows in 3.6 (except for 0, 1, and 2 attributes):
Python 2.7 Python 3.6
attrs __slots__ __dict__* __slots__ __dict__* | *(no slots defined)
none 16 56 + 272† 16 56 + 112† | †if __dict__ referenced
one 48 56 + 272 48 56 + 112
two 56 56 + 272 56 56 + 112
six 88 56 + 1040 88 56 + 152
11 128 56 + 1040 128 56 + 240
22 216 56 + 3344 216 56 + 408
43 384 56 + 3344 384 56 + 752
So, in spite of smaller dicts in Python 3, we see how nicely __slots__ scale for instances to save us memory, and that is a major reason you would want to use __slots__.
Just for completeness of my notes, note that there is a one-time cost per slot in the class's namespace of 64 bytes in Python 2, and 72 bytes in Python 3, because slots use data descriptors like properties, called "members".
>>> Foo.foo
<member 'foo' of 'Foo' objects>
>>> type(Foo.foo)
<class 'member_descriptor'>
>>> getsizeof(Foo.foo)
72
Demonstration of __slots__:
To deny the creation of a __dict__, you must subclass object. Everything subclasses object in Python 3, but in Python 2 you had to be explicit:
class Base(object):
__slots__ = ()
now:
>>> b = Base()
>>> b.a = 'a'
Traceback (most recent call last):
File "<pyshell#38>", line 1, in <module>
b.a = 'a'
AttributeError: 'Base' object has no attribute 'a'
Or subclass another class that defines __slots__
class Child(Base):
__slots__ = ('a',)
and now:
c = Child()
c.a = 'a'
but:
>>> c.b = 'b'
Traceback (most recent call last):
File "<pyshell#42>", line 1, in <module>
c.b = 'b'
AttributeError: 'Child' object has no attribute 'b'
To allow __dict__ creation while subclassing slotted objects, just add '__dict__' to the __slots__ (note that slots are ordered, and you shouldn't repeat slots that are already in parent classes):
class SlottedWithDict(Child):
__slots__ = ('__dict__', 'b')
swd = SlottedWithDict()
swd.a = 'a'
swd.b = 'b'
swd.c = 'c'
and
>>> swd.__dict__
{'c': 'c'}
Or you don't even need to declare __slots__ in your subclass, and you will still use slots from the parents, but not restrict the creation of a __dict__:
class NoSlots(Child): pass
ns = NoSlots()
ns.a = 'a'
ns.b = 'b'
And:
>>> ns.__dict__
{'b': 'b'}
However, __slots__ may cause problems for multiple inheritance:
class BaseA(object):
__slots__ = ('a',)
class BaseB(object):
__slots__ = ('b',)
Because creating a child class from parents with both non-empty slots fails:
>>> class Child(BaseA, BaseB): __slots__ = ()
Traceback (most recent call last):
File "<pyshell#68>", line 1, in <module>
class Child(BaseA, BaseB): __slots__ = ()
TypeError: Error when calling the metaclass bases
multiple bases have instance lay-out conflict
If you run into this problem, You could just remove __slots__ from the parents, or if you have control of the parents, give them empty slots, or refactor to abstractions:
from abc import ABC
class AbstractA(ABC):
__slots__ = ()
class BaseA(AbstractA):
__slots__ = ('a',)
class AbstractB(ABC):
__slots__ = ()
class BaseB(AbstractB):
__slots__ = ('b',)
class Child(AbstractA, AbstractB):
__slots__ = ('a', 'b')
c = Child() # no problem!
Add '__dict__' to __slots__ to get dynamic assignment:
class Foo(object):
__slots__ = 'bar', 'baz', '__dict__'
and now:
>>> foo = Foo()
>>> foo.boink = 'boink'
So with '__dict__' in slots we lose some of the size benefits with the upside of having dynamic assignment and still having slots for the names we do expect.
When you inherit from an object that isn't slotted, you get the same sort of semantics when you use __slots__ - names that are in __slots__ point to slotted values, while any other values are put in the instance's __dict__.
Avoiding __slots__ because you want to be able to add attributes on the fly is actually not a good reason - just add "__dict__" to your __slots__ if this is required.
You can similarly add __weakref__ to __slots__ explicitly if you need that feature.
Set to empty tuple when subclassing a namedtuple:
The namedtuple builtin make immutable instances that are very lightweight (essentially, the size of tuples) but to get the benefits, you need to do it yourself if you subclass them:
from collections import namedtuple
class MyNT(namedtuple('MyNT', 'bar baz')):
"""MyNT is an immutable and lightweight object"""
__slots__ = ()
usage:
>>> nt = MyNT('bar', 'baz')
>>> nt.bar
'bar'
>>> nt.baz
'baz'
And trying to assign an unexpected attribute raises an AttributeError because we have prevented the creation of __dict__:
>>> nt.quux = 'quux'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'MyNT' object has no attribute 'quux'
You can allow __dict__ creation by leaving off __slots__ = (), but you can't use non-empty __slots__ with subtypes of tuple.
Biggest Caveat: Multiple inheritance
Even when non-empty slots are the same for multiple parents, they cannot be used together:
class Foo(object):
__slots__ = 'foo', 'bar'
class Bar(object):
__slots__ = 'foo', 'bar' # alas, would work if empty, i.e. ()
>>> class Baz(Foo, Bar): pass
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
multiple bases have instance lay-out conflict
Using an empty __slots__ in the parent seems to provide the most flexibility, allowing the child to choose to prevent or allow (by adding '__dict__' to get dynamic assignment, see section above) the creation of a __dict__:
class Foo(object): __slots__ = ()
class Bar(object): __slots__ = ()
class Baz(Foo, Bar): __slots__ = ('foo', 'bar')
b = Baz()
b.foo, b.bar = 'foo', 'bar'
You don't have to have slots - so if you add them, and remove them later, it shouldn't cause any problems.
Going out on a limb here: If you're composing mixins or using abstract base classes, which aren't intended to be instantiated, an empty __slots__ in those parents seems to be the best way to go in terms of flexibility for subclassers.
To demonstrate, first, let's create a class with code we'd like to use under multiple inheritance
class AbstractBase:
__slots__ = ()
def __init__(self, a, b):
self.a = a
self.b = b
def __repr__(self):
return f'{type(self).__name__}({repr(self.a)}, {repr(self.b)})'
We could use the above directly by inheriting and declaring the expected slots:
class Foo(AbstractBase):
__slots__ = 'a', 'b'
But we don't care about that, that's trivial single inheritance, we need another class we might also inherit from, maybe with a noisy attribute:
class AbstractBaseC:
__slots__ = ()
#property
def c(self):
print('getting c!')
return self._c
#c.setter
def c(self, arg):
print('setting c!')
self._c = arg
Now if both bases had nonempty slots, we couldn't do the below. (In fact, if we wanted, we could have given AbstractBase nonempty slots a and b, and left them out of the below declaration - leaving them in would be wrong):
class Concretion(AbstractBase, AbstractBaseC):
__slots__ = 'a b _c'.split()
And now we have functionality from both via multiple inheritance, and can still deny __dict__ and __weakref__ instantiation:
>>> c = Concretion('a', 'b')
>>> c.c = c
setting c!
>>> c.c
getting c!
Concretion('a', 'b')
>>> c.d = 'd'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Concretion' object has no attribute 'd'
Other cases to avoid slots:
Avoid them when you want to perform __class__ assignment with another class that doesn't have them (and you can't add them) unless the slot layouts are identical. (I am very interested in learning who is doing this and why.)
Avoid them if you want to subclass variable length builtins like long, tuple, or str, and you want to add attributes to them.
Avoid them if you insist on providing default values via class attributes for instance variables.
You may be able to tease out further caveats from the rest of the __slots__ documentation (the 3.7 dev docs are the most current), which I have made significant recent contributions to.
Critiques of other answers
The current top answers cite outdated information and are quite hand-wavy and miss the mark in some important ways.
Do not "only use __slots__ when instantiating lots of objects"
I quote:
"You would want to use __slots__ if you are going to instantiate a lot (hundreds, thousands) of objects of the same class."
Abstract Base Classes, for example, from the collections module, are not instantiated, yet __slots__ are declared for them.
Why?
If a user wishes to deny __dict__ or __weakref__ creation, those things must not be available in the parent classes.
__slots__ contributes to reusability when creating interfaces or mixins.
It is true that many Python users aren't writing for reusability, but when you are, having the option to deny unnecessary space usage is valuable.
__slots__ doesn't break pickling
When pickling a slotted object, you may find it complains with a misleading TypeError:
>>> pickle.loads(pickle.dumps(f))
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled
This is actually incorrect. This message comes from the oldest protocol, which is the default. You can select the latest protocol with the -1 argument. In Python 2.7 this would be 2 (which was introduced in 2.3), and in 3.6 it is 4.
>>> pickle.loads(pickle.dumps(f, -1))
<__main__.Foo object at 0x1129C770>
in Python 2.7:
>>> pickle.loads(pickle.dumps(f, 2))
<__main__.Foo object at 0x1129C770>
in Python 3.6
>>> pickle.loads(pickle.dumps(f, 4))
<__main__.Foo object at 0x1129C770>
So I would keep this in mind, as it is a solved problem.
Critique of the (until Oct 2, 2016) accepted answer
The first paragraph is half short explanation, half predictive. Here's the only part that actually answers the question
The proper use of __slots__ is to save space in objects. Instead of having a dynamic dict that allows adding attributes to objects at anytime, there is a static structure which does not allow additions after creation. This saves the overhead of one dict for every object that uses slots
The second half is wishful thinking, and off the mark:
While this is sometimes a useful optimization, it would be completely unnecessary if the Python interpreter was dynamic enough so that it would only require the dict when there actually were additions to the object.
Python actually does something similar to this, only creating the __dict__ when it is accessed, but creating lots of objects with no data is fairly ridiculous.
The second paragraph oversimplifies and misses actual reasons to avoid __slots__. The below is not a real reason to avoid slots (for actual reasons, see the rest of my answer above.):
They change the behavior of the objects that have slots in a way that can be abused by control freaks and static typing weenies.
It then goes on to discuss other ways of accomplishing that perverse goal with Python, not discussing anything to do with __slots__.
The third paragraph is more wishful thinking. Together it is mostly off-the-mark content that the answerer didn't even author and contributes to ammunition for critics of the site.
Memory usage evidence
Create some normal objects and slotted objects:
>>> class Foo(object): pass
>>> class Bar(object): __slots__ = ()
Instantiate a million of them:
>>> foos = [Foo() for f in xrange(1000000)]
>>> bars = [Bar() for b in xrange(1000000)]
Inspect with guppy.hpy().heap():
>>> guppy.hpy().heap()
Partition of a set of 2028259 objects. Total size = 99763360 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1000000 49 64000000 64 64000000 64 __main__.Foo
1 169 0 16281480 16 80281480 80 list
2 1000000 49 16000000 16 96281480 97 __main__.Bar
3 12284 1 987472 1 97268952 97 str
...
Access the regular objects and their __dict__ and inspect again:
>>> for f in foos:
... f.__dict__
>>> guppy.hpy().heap()
Partition of a set of 3028258 objects. Total size = 379763480 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1000000 33 280000000 74 280000000 74 dict of __main__.Foo
1 1000000 33 64000000 17 344000000 91 __main__.Foo
2 169 0 16281480 4 360281480 95 list
3 1000000 33 16000000 4 376281480 99 __main__.Bar
4 12284 0 987472 0 377268952 99 str
...
This is consistent with the history of Python, from Unifying types and classes in Python 2.2
If you subclass a built-in type, extra space is automatically added to the instances to accomodate __dict__ and __weakrefs__. (The __dict__ is not initialized until you use it though, so you shouldn't worry about the space occupied by an empty dictionary for each instance you create.) If you don't need this extra space, you can add the phrase "__slots__ = []" to your class.
Quoting Jacob Hallen:
The proper use of __slots__ is to save space in objects. Instead of having
a dynamic dict that allows adding attributes to objects at anytime,
there is a static structure which does not allow additions after creation.
[This use of __slots__ eliminates the overhead of one dict for every object.] While this is sometimes a useful optimization, it would be completely
unnecessary if the Python interpreter was dynamic enough so that it would
only require the dict when there actually were additions to the object.
Unfortunately there is a side effect to slots. They change the behavior of
the objects that have slots in a way that can be abused by control freaks
and static typing weenies. This is bad, because the control freaks should
be abusing the metaclasses and the static typing weenies should be abusing
decorators, since in Python, there should be only one obvious way of doing something.
Making CPython smart enough to handle saving space without __slots__ is a major
undertaking, which is probably why it is not on the list of changes for P3k (yet).
You would want to use __slots__ if you are going to instantiate a lot (hundreds, thousands) of objects of the same class. __slots__ only exists as a memory optimization tool.
It's highly discouraged to use __slots__ for constraining attribute creation.
Pickling objects with __slots__ won't work with the default (oldest) pickle protocol; it's necessary to specify a later version.
Some other introspection features of python may also be adversely affected.
Each python object has a __dict__ atttribute which is a dictionary containing all other attributes. e.g. when you type self.attr python is actually doing self.__dict__['attr']. As you can imagine using a dictionary to store attribute takes some extra space & time for accessing it.
However, when you use __slots__, any object created for that class won't have a __dict__ attribute. Instead, all attribute access is done directly via pointers.
So if want a C style structure rather than a full fledged class you can use __slots__ for compacting size of the objects & reducing attribute access time. A good example is a Point class containing attributes x & y. If you are going to have a lot of points, you can try using __slots__ in order to conserve some memory.
In addition to the other answers, here is an example of using __slots__:
>>> class Test(object): #Must be new-style class!
... __slots__ = ['x', 'y']
...
>>> pt = Test()
>>> dir(pt)
['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__',
'__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__slots__', '__str__', 'x', 'y']
>>> pt.x
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: x
>>> pt.x = 1
>>> pt.x
1
>>> pt.z = 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Test' object has no attribute 'z'
>>> pt.__dict__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Test' object has no attribute '__dict__'
>>> pt.__slots__
['x', 'y']
So, to implement __slots__, it only takes an extra line (and making your class a new-style class if it isn't already). This way you can reduce the memory footprint of those classes 5-fold, at the expense of having to write custom pickle code, if and when that becomes necessary.
Slots are very useful for library calls to eliminate the "named method dispatch" when making function calls. This is mentioned in the SWIG documentation. For high performance libraries that want to reduce function overhead for commonly called functions using slots is much faster.
Now this may not be directly related to the OPs question. It is related more to building extensions than it does to using the slots syntax on an object. But it does help complete the picture for the usage of slots and some of the reasoning behind them.
A very simple example of __slot__ attribute.
Problem: Without __slots__
If I don't have __slot__ attribute in my class, I can add new attributes to my objects.
class Test:
pass
obj1=Test()
obj2=Test()
print(obj1.__dict__) #--> {}
obj1.x=12
print(obj1.__dict__) # --> {'x': 12}
obj1.y=20
print(obj1.__dict__) # --> {'x': 12, 'y': 20}
obj2.x=99
print(obj2.__dict__) # --> {'x': 99}
If you look at example above, you can see that obj1 and obj2 have their own x and y attributes and python has also created a dict attribute for each object (obj1 and obj2).
Suppose if my class Test has thousands of such objects? Creating an additional attribute dict for each object will cause lot of overhead (memory, computing power etc.) in my code.
Solution: With __slots__
Now in the following example my class Test contains __slots__ attribute. Now I can't add new attributes to my objects (except attribute x) and python doesn't create a dict attribute anymore. This eliminates overhead for each object, which can become significant if you have many objects.
class Test:
__slots__=("x")
obj1=Test()
obj2=Test()
obj1.x=12
print(obj1.x) # --> 12
obj2.x=99
print(obj2.x) # --> 99
obj1.y=28
print(obj1.y) # --> AttributeError: 'Test' object has no attribute 'y'
An attribute of a class instance has 3 properties: the instance, the name of the attribute, and the value of the attribute.
In regular attribute access, the instance acts as a dictionary and the name of the attribute acts as the key in that dictionary looking up value.
instance(attribute) --> value
In __slots__ access, the name of the attribute acts as the dictionary and the instance acts as the key in the dictionary looking up value.
attribute(instance) --> value
In flyweight pattern, the name of the attribute acts as the dictionary and the value acts as the key in that dictionary looking up the instance.
attribute(value) --> instance
In addition to the other answers, __slots__ also adds a little typographical security by limiting attributes to a predefined list. This has long been a problem with JavaScript which also allows you to add new attributes to an existing object, whether you meant to or not.
Here is a normal unslotted object which does nothing, but allows you to add attributes:
class Unslotted:
pass
test = Unslotted()
test.name = 'Fred'
test.Name = 'Wilma'
Since Python is case sensitive, the two attributes, spelled the same but with different case, are different. If you suspect that one of those is a typing error, then bad luck.
Using slots, you can limit this:
class Slotted:
__slots__ = ('name')
pass
test = Slotted()
test.name = 'Fred' # OK
test.Name = 'Wilma' # Error
This time, the second attribute (Name) is disallowed because it’s not in the __slots__ collection.
I would suggest that it’s probably better to use __slots__ where possible to keep more control over the object.
Beginning in Python 3.9, a dict may be used to add descriptions to attributes via __slots__. None may be used for attributes without descriptions, and private variables will not appear even if a description is given.
class Person:
__slots__ = {
"birthday":
"A datetime.date object representing the person's birthday.",
"name":
"The first and last name.",
"public_variable":
None,
"_private_variable":
"Description",
}
help(Person)
"""
Help on class Person in module __main__:
class Person(builtins.object)
| Data descriptors defined here:
|
| birthday
| A datetime.date object representing the person's birthday.
|
| name
| The first and last name.
|
| public_variable
"""
Another somewhat obscure use of __slots__ is to add attributes to an object proxy from the ProxyTypes package, formerly part of the PEAK project. Its ObjectWrapper allows you to proxy another object, but intercept all interactions with the proxied object. It is not very commonly used (and no Python 3 support), but we have used it to implement a thread-safe blocking wrapper around an async implementation based on tornado that bounces all access to the proxied object through the ioloop, using thread-safe concurrent.Future objects to synchronise and return results.
By default any attribute access to the proxy object will give you the result from the proxied object. If you need to add an attribute on the proxy object, __slots__ can be used.
from peak.util.proxies import ObjectWrapper
class Original(object):
def __init__(self):
self.name = 'The Original'
class ProxyOriginal(ObjectWrapper):
__slots__ = ['proxy_name']
def __init__(self, subject, proxy_name):
# proxy_info attributed added directly to the
# Original instance, not the ProxyOriginal instance
self.proxy_info = 'You are proxied by {}'.format(proxy_name)
# proxy_name added to ProxyOriginal instance, since it is
# defined in __slots__
self.proxy_name = proxy_name
super(ProxyOriginal, self).__init__(subject)
if __name__ == "__main__":
original = Original()
proxy = ProxyOriginal(original, 'Proxy Overlord')
# Both statements print "The Original"
print "original.name: ", original.name
print "proxy.name: ", proxy.name
# Both statements below print
# "You are proxied by Proxy Overlord", since the ProxyOriginal
# __init__ sets it to the original object
print "original.proxy_info: ", original.proxy_info
print "proxy.proxy_info: ", proxy.proxy_info
# prints "Proxy Overlord"
print "proxy.proxy_name: ", proxy.proxy_name
# Raises AttributeError since proxy_name is only set on
# the proxy object
print "original.proxy_name: ", proxy.proxy_name
The original question was about general use cases not only about memory.
So it should be mentioned here that you also get better performance when instantiating large amounts of objects - interesting e.g. when parsing large documents into objects or from a database.
Here is a comparison of creating object trees with a million entries, using slots and without slots. As a reference also the performance when using plain dicts for the trees (Py2.7.10 on OSX):
********** RUN 1 **********
1.96036410332 <class 'css_tree_select.element.Element'>
3.02922606468 <class 'css_tree_select.element.ElementNoSlots'>
2.90828204155 dict
********** RUN 2 **********
1.77050495148 <class 'css_tree_select.element.Element'>
3.10655999184 <class 'css_tree_select.element.ElementNoSlots'>
2.84120798111 dict
********** RUN 3 **********
1.84069895744 <class 'css_tree_select.element.Element'>
3.21540498734 <class 'css_tree_select.element.ElementNoSlots'>
2.59615707397 dict
********** RUN 4 **********
1.75041103363 <class 'css_tree_select.element.Element'>
3.17366290092 <class 'css_tree_select.element.ElementNoSlots'>
2.70941114426 dict
Test classes (ident, appart from slots):
class Element(object):
__slots__ = ['_typ', 'id', 'parent', 'childs']
def __init__(self, typ, id, parent=None):
self._typ = typ
self.id = id
self.childs = []
if parent:
self.parent = parent
parent.childs.append(self)
class ElementNoSlots(object): (same, w/o slots)
testcode, verbose mode:
na, nb, nc = 100, 100, 100
for i in (1, 2, 3, 4):
print '*' * 10, 'RUN', i, '*' * 10
# tree with slot and no slot:
for cls in Element, ElementNoSlots:
t1 = time.time()
root = cls('root', 'root')
for i in xrange(na):
ela = cls(typ='a', id=i, parent=root)
for j in xrange(nb):
elb = cls(typ='b', id=(i, j), parent=ela)
for k in xrange(nc):
elc = cls(typ='c', id=(i, j, k), parent=elb)
to = time.time() - t1
print to, cls
del root
# ref: tree with dicts only:
t1 = time.time()
droot = {'childs': []}
for i in xrange(na):
ela = {'typ': 'a', id: i, 'childs': []}
droot['childs'].append(ela)
for j in xrange(nb):
elb = {'typ': 'b', id: (i, j), 'childs': []}
ela['childs'].append(elb)
for k in xrange(nc):
elc = {'typ': 'c', id: (i, j, k), 'childs': []}
elb['childs'].append(elc)
td = time.time() - t1
print td, 'dict'
del droot
You have — essentially — no use for __slots__.
For the time when you think you might need __slots__, you actually want to use Lightweight or Flyweight design patterns. These are cases when you no longer want to use purely Python objects. Instead, you want a Python object-like wrapper around an array, struct, or numpy array.
class Flyweight(object):
def get(self, theData, index):
return theData[index]
def set(self, theData, index, value):
theData[index]= value
The class-like wrapper has no attributes — it just provides methods that act on the underlying data. The methods can be reduced to class methods. Indeed, it could be reduced to just functions operating on the underlying array of data.
In addition to the myriad advantages described in other answers herein – compact instances for the memory-conscious, less error-prone than the more mutable __dict__-bearing instances, et cetera – I find that using __slots__ offers more legible class declarations, as the instance variables of the class are explicitly out in the open.
To contend with inheritance issues with __slots__ declarations I use this metaclass:
import abc
class Slotted(abc.ABCMeta):
""" A metaclass that ensures its classes, and all subclasses,
will be slotted types.
"""
def __new__(metacls, name, bases, attributes, **kwargs):
""" Override for `abc.ABCMeta.__new__(…)` setting up a
derived slotted class.
"""
if '__slots__' not in attributes:
attributes['__slots__'] = tuple()
return super(Slotted, metacls).__new__(metacls, name, # type: ignore
bases,
attributes,
**kwargs)
… which, if declared as the metaclass of the base class in an inheritance tower, ensures that everything that derives from that base class will properly inherit __slots__ attributes, even if an intermediate class fails to declare any. Like so:
# note no __slots__ declaration necessary with the metaclass:
class Base(metaclass=Slotted):
pass
# class is properly slotted, no __dict__:
class Derived(Base):
__slots__ = 'slot', 'another_slot'
# class is also properly slotted:
class FurtherDerived(Derived):
pass
Related
For some reason this fails in Python 3.8:
setattr(iter(()), '_hackkk', 'bad idea')
Error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-3-c046f8521130> in <module>
----> 1 setattr(iter(()), '_hackkk', 'bad idea')
AttributeError: 'tuple_iterator' object has no attribute '_hackkk'
How do I attach random data where I shouldn't, i.e., on an iterator or a generator?
You can attach data only to objects, which have __dict__-member. Not all objects have it - for example builtin classes like int, float, list and so on do not. This is an optimization, because otherwise instances of those classes would need too much memory - a dictionary has a quite large memory footprint.
Also for normal classes one could use __slots__, thus removing __dict__-member and prohibiting dynamic addition of attributes to an object of this class. E.g.
class A:
pass
setattr(A(),'b', 2)
works, but
class B:
__slots__ = 'b'
setattr(B(),'c', 2)
doesn't work, as class B has no slot with name c and no __dict__.
Thus, the answer to your question is: for some classes (as the tuple_iterator) you cannot.
If you really need to, you can wrap tuple_iterator in a class with __dict__ and append the new attribute to the wrapper-object:
class IterWrapper:
def __init__(self, it):
self.it=it
def __next__(self):
return next(self.it)
def __iter__(self): # for testing
return self
and now:
iw=IterWrapper(iter((1,2,3)))
setattr(iw, "a", 2)
print(iw.a) # prints 2
print(list(iw)) # prints [1,2,3]
has the desired behavior.
Is there any meaningful distinction between:
class A(object):
foo = 5 # some default value
vs.
class B(object):
def __init__(self, foo=5):
self.foo = foo
If you're creating a lot of instances, is there any difference in performance or space requirements for the two styles? When you read the code, do you consider the meaning of the two styles to be significantly different?
There is a significant semantic difference (beyond performance considerations):
when the attribute is defined on the instance (which is what we usually do), there can be multiple objects referred to. Each gets a totally separate version of that attribute.
when the attribute is defined on the class, there is only one underlying object referred to, so if operations on different instances of that class both attempt to set/(append/extend/insert/etc.) the attribute, then:
if the attribute is a builtin type (like int, float, boolean, string), operations on one object will overwrite (clobber) the value
if the attribute is a mutable type (like a list or a dict), we will get unwanted leakage.
For example:
>>> class A: foo = []
>>> a, b = A(), A()
>>> a.foo.append(5)
>>> b.foo
[5]
>>> class A:
... def __init__(self): self.foo = []
>>> a, b = A(), A()
>>> a.foo.append(5)
>>> b.foo
[]
The difference is that the attribute on the class is shared by all instances. The attribute on an instance is unique to that instance.
If coming from C++, attributes on the class are more like static member variables.
Here is a very good post, and summary it as below.
class Bar(object):
## No need for dot syntax
class_var = 1
def __init__(self, i_var):
self.i_var = i_var
## Need dot syntax as we've left scope of class namespace
Bar.class_var
## 1
foo = MyClass(2)
## Finds i_var in foo's instance namespace
foo.i_var
## 2
## Doesn't find class_var in instance namespace…
## So look's in class namespace (Bar.__dict__)
foo.class_var
## 1
And in visual form
Class attribute assignment
If a class attribute is set by accessing the class, it will override the value for all instances
foo = Bar(2)
foo.class_var
## 1
Bar.class_var = 2
foo.class_var
## 2
If a class variable is set by accessing an instance, it will override the value only for that instance. This essentially overrides the class variable and turns it into an instance variable available, intuitively, only for that instance.
foo = Bar(2)
foo.class_var
## 1
foo.class_var = 2
foo.class_var
## 2
Bar.class_var
## 1
When would you use class attribute?
Storing constants. As class attributes can be accessed as attributes of the class itself, it’s often nice to use them for storing Class-wide, Class-specific constants
class Circle(object):
pi = 3.14159
def __init__(self, radius):
self.radius = radius
def area(self):
return Circle.pi * self.radius * self.radius
Circle.pi
## 3.14159
c = Circle(10)
c.pi
## 3.14159
c.area()
## 314.159
Defining default values. As a trivial example, we might create a bounded list (i.e., a list that can only hold a certain number of elements or fewer) and choose to have a default cap of 10 items
class MyClass(object):
limit = 10
def __init__(self):
self.data = []
def item(self, i):
return self.data[i]
def add(self, e):
if len(self.data) >= self.limit:
raise Exception("Too many elements")
self.data.append(e)
MyClass.limit
## 10
Since people in the comments here and in two other questions marked as dups all appear to be confused about this in the same way, I think it's worth adding an additional answer on top of Alex Coventry's.
The fact that Alex is assigning a value of a mutable type, like a list, has nothing to do with whether things are shared or not. We can see this with the id function or the is operator:
>>> class A: foo = object()
>>> a, b = A(), A()
>>> a.foo is b.foo
True
>>> class A:
... def __init__(self): self.foo = object()
>>> a, b = A(), A()
>>> a.foo is b.foo
False
(If you're wondering why I used object() instead of, say, 5, that's to avoid running into two whole other issues which I don't want to get into here; for two different reasons, entirely separately-created 5s can end up being the same instance of the number 5. But entirely separately-created object()s cannot.)
So, why is it that a.foo.append(5) in Alex's example affects b.foo, but a.foo = 5 in my example doesn't? Well, try a.foo = 5 in Alex's example, and notice that it doesn't affect b.foo there either.
a.foo = 5 is just making a.foo into a name for 5. That doesn't affect b.foo, or any other name for the old value that a.foo used to refer to.* It's a little tricky that we're creating an instance attribute that hides a class attribute,** but once you get that, nothing complicated is happening here.
Hopefully it's now obvious why Alex used a list: the fact that you can mutate a list means it's easier to show that two variables name the same list, and also means it's more important in real-life code to know whether you have two lists or two names for the same list.
* The confusion for people coming from a language like C++ is that in Python, values aren't stored in variables. Values live off in value-land, on their own, variables are just names for values, and assignment just creates a new name for a value. If it helps, think of each Python variable as a shared_ptr<T> instead of a T.
** Some people take advantage of this by using a class attribute as a "default value" for an instance attribute that instances may or may not set. This can be useful in some cases, but it can also be confusing, so be careful with it.
There is one more situation.
Class and instance attributes is Descriptor.
# -*- encoding: utf-8 -*-
class RevealAccess(object):
def __init__(self, initval=None, name='var'):
self.val = initval
self.name = name
def __get__(self, obj, objtype):
return self.val
class Base(object):
attr_1 = RevealAccess(10, 'var "x"')
def __init__(self):
self.attr_2 = RevealAccess(10, 'var "x"')
def main():
b = Base()
print("Access to class attribute, return: ", Base.attr_1)
print("Access to instance attribute, return: ", b.attr_2)
if __name__ == '__main__':
main()
Above will output:
('Access to class attribute, return: ', 10)
('Access to instance attribute, return: ', <__main__.RevealAccess object at 0x10184eb50>)
The same type of instance access through class or instance return different result!
And i found in c.PyObject_GenericGetAttr definition，and a great post.
Explain
If the attribute is found in the dictionary of the classes which make up.
the objects MRO, then check to see if the attribute being looked up points to a Data Descriptor (which is nothing more that a class implementing both the __get__ and the __set__ methods).
If it does, resolve the attribute lookup by calling the __get__ method of the Data Descriptor (lines 28–33).
I would like to do the following:
class A(object): pass
a = A()
a.__int__ = lambda self: 3
i = int(a)
Unfortunately, this throws:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: int() argument must be a string or a number, not 'A'
This only seems to work if I assign the "special" method to the class A instead of an instance of it. Is there any recourse?
One way I thought of was:
def __int__(self):
# No infinite loop
if type(self).__int__.im_func != self.__int__.im_func:
return self.__int__()
raise NotImplementedError()
But that looks rather ugly.
Thanks.
Python always looks up special methods on the class, not the instance (except in the old, aka "legacy", kind of classes -- they're deprecated and have gone away in Python 3, because of the quirky semantics that mostly comes from looking up special methods on the instance, so you really don't want to use them, believe me!-).
To make a special class whose instances can have special methods independent from each other, you need to give each instance its own class -- then you can assign special methods on the instance's (individual) class without affecting other instances, and live happily ever after. If you want to make it look like you're assigning to an attribute the instance, while actually assigning to an attribute of the individualized per-instance class, you can get that with a special __setattr__ implementation, of course.
Here's the simple case, with explicit "assign to class" syntax:
>>> class Individualist(object):
... def __init__(self):
... self.__class__ = type('GottaBeMe', (self.__class__, object), {})
...
>>> a = Individualist()
>>> b = Individualist()
>>> a.__class__.__int__ = lambda self: 23
>>> b.__class__.__int__ = lambda self: 42
>>> int(a)
23
>>> int(b)
42
>>>
and here's the fancy version, where you "make it look like" you're assigning the special method as an instance attribute (while behind the scene it still goes to the class of course):
>>> class Sophisticated(Individualist):
... def __setattr__(self, n, v):
... if n[:2]=='__' and n[-2:]=='__' and n!='__class__':
... setattr(self.__class__, n, v)
... else:
... object.__setattr__(self, n, v)
...
>>> c = Sophisticated()
>>> d = Sophisticated()
>>> c.__int__ = lambda self: 54
>>> d.__int__ = lambda self: 88
>>> int(c)
54
>>> int(d)
88
The only recourse that works for new-style classes is to have a method on the class that calls the attribute on the instance (if it exists):
class A(object):
def __int__(self):
if '__int__' in self.__dict__:
return self.__int__()
raise ValueError
a = A()
a.__int__ = lambda: 3
int(a)
Note that a.__int__ will not be a method (only functions that are attributes of the class will become methods) so self is not passed implicitly.
I have nothing to add about the specifics of overriding __int__. But I noticed one thing about your sample that bears discussing.
When you manually assign new methods to an object, "self" is not automatically passed in. I've modified your sample code to make my point clearer:
class A(object): pass
a = A()
a.foo = lambda self: 3
a.foo()
If you run this code, it throws an exception because you passed in 0 arguments to "foo" and 1 is required. If you remove the "self" it works fine.
Python only automatically prepends "self" to the arguments if it had to look up the method in the class of the object and the function it found is a "normal" function. (Examples of "abnormal" functions: class methods, callable objects, bound method objects.) If you stick callables in to the object itself they won't automatically get "self".
If you want self there, use a closure.
In the following, setattr succeeds in the first invocation, but fails in the second, with:
AttributeError: 'method' object has no attribute 'i'
Why is this, and is there a way of setting an attribute on a method such that it will only exist on one instance, not for each instance of the class?
class c:
def m(self):
print(type(c.m))
setattr(c.m, 'i', 0)
print(type(self.m))
setattr(self.m, 'i', 0)
Python 3.2.2
The short answer: There is no way of adding custom attributes to bound methods.
The long answer follows.
In Python, there are function objects and method objects. When you define a class, the def statement creates a function object that lives within the class' namespace:
>>> class c:
... def m(self):
... pass
...
>>> c.m
<function m at 0x025FAE88>
Function objects have a special __dict__ attribute that can hold user-defined attributes:
>>> c.m.i = 0
>>> c.m.__dict__
{'i': 0}
Method objects are different beasts. They are tiny objects just holding a reference to the corresponding function object (__func__) and one to its host object (__self__):
>>> c().m
<bound method c.m of <__main__.c object at 0x025206D0>>
>>> c().m.__self__
<__main__.c object at 0x02625070>
>>> c().m.__func__
<function m at 0x025FAE88>
>>> c().m.__func__ is c.m
True
Method objects provide a special __getattr__ that forwards attribute access to the function object:
>>> c().m.i
0
This is also true for the __dict__ property:
>>> c().m.__dict__['a'] = 42
>>> c.m.a
42
>>> c().m.__dict__ is c.m.__dict__
True
Setting attributes follows the default rules, though, and since they don't have their own __dict__, there is no way to set arbitrary attributes.
This is similar to user-defined classes defining __slots__ and no __dict__ slot, when trying to set a non-existing slot raises an AttributeError (see the docs on __slots__ for more information):
>>> class c:
... __slots__ = ('a', 'b')
...
>>> x = c()
>>> x.a = 1
>>> x.b = 2
>>> x.c = 3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'c' object has no attribute 'c'
Q: "Is there a way of setting an attribute on a method such that it will only exist on one instance, not for each instance of the class?"
A: Yes:
class c:
def m(self):
print(type(c.m))
setattr(c.m, 'i', 0)
print(type(self))
setattr(self, 'i', 0)
The static variable on functions in the post you link to is not useful for methods. It sets an attribute on the function so that this attribute is available the next time the function is called, so you can make a counter or whatnot.
But methods have an object instance associated with them (self). Hence you have no need to set attributes on the method, as you simply can set it on the instance instead. That is in fact exactly what the instance is for.
The post you link to shows how to make a function with a static variable. I would say that in Python doing so would be misguided. Instead look at this answer: What is the Python equivalent of static variables inside a function?
That is the way to do it in Python in a way that is clear and easily understandable. You use a class and make it callable. Setting attributes on functions is possible and there are probably cases where it's a good idea, but in general it will just end up confusing people.
What is the difference between doing
class a:
def __init__(self):
self.val=1
to doing
class a:
val=1
def __init__(self):
pass
class a:
def __init__(self):
self.val=1
this creates a class (in Py2, a cruddy, legacy, old-style, don't do that! class; in Py3, the nasty old legacy classes have finally gone away so this would be a class of the one and only kind -- the **good* kind, which requires class a(object): in Py2) such that each instance starts out with its own reference to the integer object 1.
class a:
val=1
def __init__(self):
pass
this creates a class (of the same kind) which itself has a reference to the integer object 1 (its instances start out with no per-instance reference).
For immutables like int values, it's hard to see a practical difference. For example, in either case, if you later do self.val = 2 on one instance of a, this will make an instance reference (the existing answer is badly wrong in this respect).
The distinction is important for mutable objects, because they have mutator methods, so it's pretty crucial to know if a certain list is unique per-instance or shared among all instances. But for immutable objects, since you can never change the object itself but only assign (e.g. to self.val, which will always make a per-instance reference), it's pretty minor.
Just about the only relevant difference for immutables: if you later assign a.val = 3, in the first case this will affect what's seen as self.val by each instance (except for instances that had their own self.val assigned to, or equivalent actions); in the second case, it will not affect what's seen as self.val by any instance (except for instances for which you had performed del self.val or equivalent actions).
Others have explained the technical differences. I'll try to explain why you might want to use class variables.
If you're only instantiating the class once, then class variables effectively are instance variables. However, if you're making many copies, or want to share state among a few instances, then class variables are very handy. For example:
class Foo(object):
def __init__(self):
self.bar = expensivefunction()
myobjs = [Foo() for _ in range(1000000)]
will cause expensivefunction() to be called a million times. If it's going to return the same value each time, say fetching a configuration parameter from a database, then you should consider moving it into the class definition so that it's only called once and then shared across all instances.
I also use class variables a lot when memoizing results. Example:
class Foo(object):
bazcache = {}
#classmethod
def baz(cls, key):
try:
result = cls.bazcache[key]
except KeyError:
result = expensivefunction(key)
cls.bazcache[key] = result
return result
In this case, baz is a class method; its result doesn't depend on any instance variables. That means we can keep one copy of the results cache in the class variable, so that 1) you don't store the same results multiple times, and 2) each instance can benefit from results that were cached from other instances.
To illustrate, suppose that you have a million instances, each operating on the results of a Google search. You'd probably much prefer that all those objects share those results than to have each one execute the search and wait for the answer.
So I'd disagree with Lennart here. Class variables are very convenient in certain cases. When they're the right tool for the job, don't hesitate to use them.
As mentioned by others, in one case it's an attribute on the class on the other an attribute on the instance. Does this matter? Yes, in one case it does. As Alex said, if the value is mutable. The best explanation is code, so I'll add some code to show it (that's all this answer does, really):
First a class defining two instance attributes.
>>> class A(object):
... def __init__(self):
... self.number = 45
... self.letters = ['a', 'b', 'c']
...
And then a class defining two class attributes.
>>> class B(object):
... number = 45
... letters = ['a', 'b', 'c']
...
Now we use them:
>>> a1 = A()
>>> a2 = A()
>>> a2.number = 15
>>> a2.letters.append('z')
And all is well:
>>> a1.number
45
>>> a1.letters
['a', 'b', 'c']
Now use the class attribute variation:
>>> b1 = B()
>>> b2 = B()
>>> b2.number = 15
>>> b2.letters.append('z')
And all is...well...
>>> b1.number
45
>>> b1.letters
['a', 'b', 'c', 'z']
Yeah, notice that when you changed, the mutable class attribute it changed for all classes. That's usually not what you want.
If you are using the ZODB, you use a lot of class attributes because it's a handy way of upgrading existing objects with new attributes, or adding information on a class level that doesn't get persisted. Otherwise you can pretty much ignore them.