I want to subclass dict in python such that all the dictionaries of the sub-class are immutable.
I don't understand how does __hash__ affects the immutability, since in my understanding it just signifies the equality or non-equality of objects !
So, can __hash__ be used to implement immutability ? How ?
Update:
Objective is that common response from an API is available as a dict, which has to be shared as a global variable. So, that needs to be intact no matter what ?
I found a Official reference : suggestion contained in a rejected PEP.
class imdict(dict):
def __hash__(self):
return id(self)
def _immutable(self, *args, **kws):
raise TypeError('object is immutable')
__setitem__ = _immutable
__delitem__ = _immutable
clear = _immutable
update = _immutable
setdefault = _immutable
pop = _immutable
popitem = _immutable
Attribution : http://www.python.org/dev/peps/pep-0351/
It is possible to create immutable dict using just standard library.
from types import MappingProxyType
power_levels = MappingProxyType(
{
"Kevin": 9001,
"Benny": 8000,
}
)
See source of idea with more detailed explanation
In frozendict, hash is simply implemented following the rejected PEP 416 of Victor Stinner:
def __hash__(self):
try:
fs = frozenset(self.items())
except TypeError:
hash = -1
else:
hash = hash(fs)
if hash == -1:
raise TypeError("Not all values are hashable.")
return hash
PS: I'm the new maintainer of the package.
So, can __hash__ be used to implement immutability ?
No, it can't. The object can be made mutable (or not) irrespective of what its __hash__ method does.
The relationship between immutable objects and __hash__ is that, since an immutable object cannot be changed, the value returned by __hash__ remains constant post-construction. For mutable objects, this may or may not be the case (the recommended practice is that such objects simply fail to hash).
For further discussion, see Issue 13707: Clarify hash() constency period.
Regarding the relationship between hashability and mutability:
To be useful, a hash implementation needs to fulfil the following properties:
The hash value of two objects that compare equal using == must be equal.
The hash value may not change over time.
These two properties imply that hashable classes cannot take mutable properties into account when comparing instances, and by contraposition that classes which do take mutable properties into account when comparing instances are not hashable. Immutable classes can be made hashable without any implications for comparison.
All of the built-in mutable types are not hashable, and all of the immutable built-in types are hashable. This is mainly a consequence of the above observations.
User-defined classes by default define comparison based on object identity, and use the id() as hash. They are mutable, but the mutable data is not taken into account when comparing instances, so they can be made hashable.
Making a class hashable does not make it immutable in some magic way. On the contrary, to make a dictionary hashable in a reasonable way while keeping the original comparison operator, you will first need to make it immutable.
Edit: Regarding your update:
There are several ways to provide the equivalent of global immutable dictionary:
Use a collections.namedtuple() instance instead.
Use a user-defined class with read-only properties.
I'd usually go with something like this:
_my_global_dict = {"a": 42, "b": 7}
def request_value(key):
return _my_global_dict[key]
By the leading underscore, you make clear that _my_global_dict is an implementation detail not to be touched by application code. Note that this code would still allow to modify dictionary values if they happen to be mutable objects. You could solve this problem by returning copy.copy()s or copy.deepcopy()s of the values if necessary.
Since Python 3.3, it's possible to use MappingProxyType to create an immutable mapping:
>>> from types import MappingProxyType
>>> MappingProxyType({'a': 1})
mappingproxy({'a': 1})
>>> immutable_mapping = MappingProxyType({'a': 1})
>>> immutable_mapping['a']
1
>>> immutable_mapping['b'] = 2
Traceback (most recent call last):
(...)
TypeError: 'mappingproxy' object does not support item assignment
It's not hashable so you can't use it as a dictionary key (and it's "final", so you can't subclass it to override __hash__), but it's good enough if you want an immutable mapping to prevent accidental modification of a global value (like a class default attribute).
Careful not to add mutable values that could themselves be modified.
Related
I am not familiar whih python.I found class object and instance object can be dictionary's key recently.So i assume class object and instance is mutable object.As we all kown,dictionary's key must be immutable object and like tuple must contains immutable object.in other words,If use a tuple as a dictionary's key.It can't contains list object,ect. But class object can ,class object could modify it's attribute.it confused me for a long time.Please lighten me.And i am not understand class namespace concept , instance namespace and the relationship with each other. Could you explain this for me? Thanks in advance. The following is my testing
class Student(object):
name='tests'
pass
dic={Student:'test'} #not error
print(id(Student))
Student.name = 'modified'
print(id(Student))
You need to be careful here. You're mixing 2 separate (but closely related) concepts.
The first concept is immutability. Immutable objects cannot change once you've created them.
The second concept is hashability -- The ability to construct a consistent integer value from an object that does not change over it's lifetime and to define a consistent equality function1. Note, these constraints are very important (as we'll see).
The latter concept determines what can be used as a dictionary key (or set item). By default, class instances have a well defined equality function (two objects are equal iff they have the same id()). Class instances also have an integer value that does not change over their lifetime (their id()). Because the id() is also exposed as the hash() return value, instances of classes (and, classes themselves which are instances of type) are hashable by default.
class Foo(object):
def __init__(self, a):
self.a = a
f1 = Foo(1)
f2 = Foo(1)
d = {
f1: 1,
f2: 2,
}
Here we have 2 separate Foo instances in our dictionary. Even though they're the same, they aren't equal and they have different hash values.
f1 == f2 # False -- They do not have the same id()
hash(f1) == hash(f2) # False. By default, the hash() is the id()
Ok, but not all things are hashable -- e.g. list and set instances aren't hashable. At some point, reference equality isn't so useful anymore. e.g. I write:
d = {[1, 2, 3]: 6}
print(d[1, 2, 3])
and I get a KeyError. Why? Because my two lists aren't the same list -- They just happen to have the same values. In other words, they equal, but they don't have reference equality. Now that just starts to get really confusing. To avoid all that confusion, the python devs have just decided to not expose the list's id() to the list's hash(). Instead, they raise a TypeError with a (hopefully) more helpful error message.
hash([]) # TypeError: unhashable type: 'list'
Note that equality is overridden to do the natural thing rather than compare by id():
l1 = [1]
l2 = [1]
l1 == l2 # True. Nice.
Alright, so far we've basically said that to put something in a dictionary, we need to have well behaving __hash__ and __eq__ methods and that objects have those by default. Some objects choose to remove them to avoid confusing situations. Where does immutability come in to this?
So far, our world consists of being able to store things in a table and look them up solely by the object's id(). That's super useful sometimes, but it's still really restrictive. I wouldn't be able to use integers in a lookup table naturally if all I can rely on is their id() (what if I store it using a literal but then do a lookup using the result of a computation?). Fortunately, we live in a world that lets us get around that problem -- immutability aids in the construction of a hash() value that isn't tied to the object's id() and isn't in danger of changing during the object's lifetime. This can be super useful because now I can do:
d = {(1, 2, 3): 4}
d[(1, 2) + (3,)] # 4!
Now the two tuples that I used were not the same tuple (they didn't have the same id()), but they are equal and because they're immutable, we can construct a hash() function that uses the contents of the tuple rather than it's id(). This is super useful! Note that if the tuple was mutable and we tried to play this trick, we'd (potentially) violate the condition that hash() should not change over the lifetime of the object.
1Consistent here means that if two objects are equal, then they also must have the same hash. This is necessary for resolving hash collisions which I won't discuss here in detail...
Here below when I try to hash a list, it gives me an error but works with a tuple. Guess it has something to do with immutability. Can someone explain this in detail ?
List
x = [1,2,3]
y = {x: 9}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
Tuple
z = (5,6)
y = {z: 89}
print(y)
{(5, 6): 89}
Dicts and other objects use hashes to store and retrieve items really quickly. The mechanics of this all happens "under the covers" - you as the programmer don't need to do anything and Python handles it all internally. The basic idea is that when you create a dictionary with {key: value}, Python needs to be able to hash whatever you used for key so it can store and look up the value quickly.
Immutable objects, or objects that can't be altered, are hashable. They have a single unique value that never changes, so python can "hash" that value and use it to look up dictionary values efficiently. Objects that fall into this category include strings, tuples, integers and so on. You may think, "But I can change a string! I just go mystr = mystr + 'foo'," but in fact what this does is create a new string instance and assigns it to mystr. It doesn't modify the existing instance. Immutable objects never change, so you can always be sure that when you generate a hash for an immutable object, looking up the object by its hash will always return the same object you started with, and not a modified version.
You can try this for yourself: hash("mystring"), hash(('foo', 'bar')), hash(1)
Mutable objects, or objects that can be modified, aren't hashable. A list can be modified in-place: mylist.append('bar') or mylist.pop(0). You can't safely hash a mutable object because you can't guarantee that the object hasn't changed since you last saw it. You'll find that list, set, and other mutable types don't have a __hash__() method. Because of this, you can't use mutable objects as dictionary keys:
>>> hash([1,2,3])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
Eric Duminil's answer provides a great example of the unexpected behaviour that arises from using mutable objects as dictionary keys
Here are examples why it might not be a good idea to allow mutable types as keys. This behaviour might be useful in some cases (e.g. using the state of the object as a key rather than the object itself) but it also might lead to suprising results or bugs.
Python
It's possible to use a numeric list as a key by defining __hash__ on a subclass of list :
class MyList(list):
def __hash__(self):
return sum(self)
my_list = MyList([1, 2, 3])
my_dict = {my_list: 'a'}
print(my_dict.get(my_list))
# a
my_list[2] = 4 # __hash__() becomes 7
print(next(iter(my_dict)))
# [1, 2, 4]
print(my_dict.get(my_list))
# None
print(my_dict.get(MyList([1,2,3])))
# None
my_list[0] = 0 # __hash_() is 6 again, but for different elements
print(next(iter(my_dict)))
# [0, 2, 4]
print(my_dict.get(my_list))
# 'a'
Ruby
In Ruby, it's allowed to use a list as a key. A Ruby list is called an Array and a dict is a Hash, but the syntax is very similar to Python's :
my_list = [1]
my_hash = { my_list => 'a'}
puts my_hash[my_list]
#=> 'a'
But if this list is modified, the dict doesn't find the corresponding value any more, even if the key is still in the dict :
my_list << 2
puts my_list
#=> [1,2]
puts my_hash.keys.first
#=> [1,2]
puts my_hash[my_list]
#=> nil
It's possible to force the dict to calculate the key hashes again :
my_hash.rehash
puts my_hash[my_list]
#=> 'a'
A hashset calculates the hash of an object and based on that hash, stores the object in the structure for fast lookup. As a result, by contract once an object is added to the dictionary, the hash is not allowed to change. Most good hash functions will depend on the number of elements and the elements itself.
A tuple is immutable, so after construction, the values cannot change and therefore the hash cannot change either (or at least a good implementation should not let the hash change).
A list on the other hand is mutable: one can later add/remove/alter elements. As a result the hash can change violating the contract.
So all objects that cannot guarantee a hash function that remains stable after the object is added, violate the contract and thus are no good candidates. Because for a lookup, the dictionary will first calculate the hash of the key, and determine the correct bucket. If the key is meanwhile changed, this could result in false negatives: the object is in the dictionary, but it can no longer be retrieved because the hash is different so a different bucket will be searched than the one where the object was originally added to.
I would like to add the following aspect as it's not covered by other answers already.
There's nothing wrong about making mutable objects hashable, it's just not unambiguous and this is why it needs to be defined and implemented consistently by the programmer himself (not by the programming language).
Note that you can implement the __hash__ method for any custom class which allows its instances to be stored in contexts where hashable types are required (such as dict keys or sets).
Hash values are usually used to decide if two objects represent the same thing. So consider the following example. You have a list with two items: l = [1, 2]. Now you add an item to the list: l.append(3). And now you must answer the following question: Is it still the same thing? Both - yes and no - are valid answers. "Yes", it is still the same list and "no", it has not the same content anymore.
So the answer to this question depends on you as the programmer and so it is up to you to manually implement hash methods for your mutable types.
Based on Python Glossary
An object is hashable if it has a hash value which never changes during its lifetime (it needs a __hash__() method), and can be compared to other objects (it needs an __eq__() method). Hashable objects which compare equal must have the same hash value.
All of Python’s immutable built-in objects are hashable; mutable containers (such as lists or dictionaries) are not.
Because a list is mutable, while a tuple is not. When you store the hash of a value in, for example, a dict, if the object changes, the stored hash value won't find out, so it will remain the same. The next time you look up the object, the dictionary will try to look it up by the old hash value, which is not relevant anymore.
To prevent that, python does not allow you to has mutable items.
I have read that lists cannot be dictionary keys because mutable objects cannot be hashed.
However, custom objects appear to be mutable as well:
# custom object
class Vertex(object):
def __init__(self, key):
self.key = key
v = Vertex(1)
v.color = 'grey' # this line suggests the custom object is mutable
But, unlike lists, they can be used as dictionary keys; why is this? Couldn't we simply hash some sort of id (such as the address of the object in memory) in both cases?
as noted in Why Lists can't be Dictionary Keys:
Lists as Dictionary Keys
That said, the simple answer to why lists cannot be used as dictionary keys is that lists do not provide a valid hash method. Of course, the obvious question is, "Why not?"
Consider what kinds of hash functions could be provided for lists.
If lists hashed by id, this would certainly be valid given Python's definition of a hash function -- lists with different hash values would have different ids. But lists are containers, and most other operations on them deal with them as such. So hashing lists by their id instead would produce unexpected behavior such as:
Looking up different lists with the same contents would produce different results, even though comparing lists with the same contents would indicate them as equivalent.
Using a list literal in a dictionary lookup would be pointless -- it would always produce a KeyError.
User Defined Types as Dictionary Keys
What about instances of user defined types?
By default, all user defined types are usable as dictionary keys with hash(object) defaulting to id(object), and cmp(object1, object2) defaulting to cmp(id(object1), id(object2)). This same suggestion was discussed above for lists and found unsatisfactory. Why are user defined types different?
In the cases where an object must be placed in a mapping, object identity is often much more important than object contents.
In the cases where object content really is important, the default settings can be redefined by overridding __hash__ and __cmp__ or __eq__.
Note that it is often better practice, when an object is to be associated with a value, to simply assign that value as one of the object's attributes.
What exactly is meant by unhashable?
>>> a={1,2,3}
>>> b={4,5,6}
>>> set([a,b])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'set'
>>>
Can any one tell what the error is exactly? Also can i add set into another set in python?
Objects that doesn't have the __hash__() attribute called unhashable. Python documentation has described the reason very well:
If a class defines mutable objects and implements an __eq__() method, it should not implement __hash__(), since the implementation of hashable collections requires that a key’s hash value is immutable (if the object’s hash value changes, it will be in the wrong hash bucket).
As Kasramvd explained, objects in python that are mutable and implement the __eq__ function are unhashable.
Since sets, lists and dicts are mutable (i.e. they can be changed; for instance you can add, remove items to all of them) , they cannot be hashed.
Since a set of sets is not possible, perhaps a set of tuple might work, though you will need to do additional bookkeeping (e.g ensure unique values) in order to achieve exactly what you described.
a = (1,2,3)
b = (4,5,6)
c = set([a,b])
Or even better, a set of frozensets. Similar to sets, but immutable (you cannot add or remove elements from them).
a = frozenset(a)
b = frozenset(b)
c = set([a,b])
A hash function is any function that can be used to map data of
arbitrary size to data of fixed size. The values returned by a hash
function are called hash values, hash codes, hash sums, or simply
hashes.
The dictionary in python is just a hash map.
And sets could only contain strings or chars or numbers, but not dics or another sets.
You might wanna look at: https://docs.python.org/2/tutorial/datastructures.html#sets
I stumbled across a blog post detailing how to implement a powerset function in Python. So I went about trying my own way of doing it, and discovered that Python apparently cannot have a set of sets, since set is not hashable. This is irksome, since the definition of a powerset is that it is a set of sets, and I wanted to implement it using actual set operations.
>>> set([ set() ])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'set'
Is there a good reason Python sets are not hashable?
Generally, only immutable objects are hashable in Python. The immutable variant of set() -- frozenset() -- is hashable.
Because they're mutable.
If they were hashable, a hash could silently become "invalid", and that would pretty much make hashing pointless.
From the Python docs:
hashable
An object is hashable if it
has a hash value which never changes
during its lifetime (it needs a
hash() method), and can be compared to other objects (it needs an
eq() or cmp() method). Hashable objects which compare equal
must have the same hash value.
Hashability makes an object usable as
a dictionary key and a set member,
because these data structures use the
hash value internally.
All of Python’s immutable built-in
objects are hashable, while no mutable
containers (such as lists or
dictionaries) are. Objects which are
instances of user-defined classes are
hashable by default; they all compare
unequal, and their hash value is their
id().
In case this helps... if you really need to convert unhashable things into hashable equivalents for some reason you might do something like this:
from collections import Hashable, MutableSet, MutableSequence, MutableMapping
def make_hashdict(value):
"""
Inspired by https://stackoverflow.com/questions/1151658/python-hashable-dicts
- with the added bonus that it inherits from the dict type of value
so OrderedDict's maintain their order and other subclasses of dict() maintain their attributes
"""
map_type = type(value)
class HashableDict(map_type):
def __init__(self, *args, **kwargs):
super(HashableDict, self).__init__(*args, **kwargs)
def __hash__(self):
return hash(tuple(sorted(self.items())))
hashDict = HashableDict(value)
return hashDict
def make_hashable(value):
if not isinstance(value, Hashable):
if isinstance(value, MutableSet):
value = frozenset(value)
elif isinstance(value, MutableSequence):
value = tuple(value)
elif isinstance(value, MutableMapping):
value = make_hashdict(value)
return value
my_set = set()
my_set.add(make_hashable(['a', 'list']))
my_set.add(make_hashable({'a': 1, 'dict': 2}))
my_set.add(make_hashable({'a', 'new', 'set'}))
print my_set
My HashableDict implementation is the simplest and least rigorous example from here. If you need a more advanced HashableDict that supports pickling and other things, check the many other implementations. In my version above I wanted to preserve the original dict class, thus preserving the order of OrderedDicts. I also use AttrDict from here for attribute-like access.
My example above is not in any way authoritative, just my solution to a similar problem where I needed to store some things in a set and needed to "hashify" them first.