We Keep Coding

How do i expand my custom dict subclass into function arguments in python

so i'm subclassing python dict to only allow certain fields in order to use it as an options map. When i unpack it in my function call i want that the type of the custom dict inside the function is still my subclass.
e.g.
#!/usr/bin/env python3
class Options(dict):
_keys = ['a', 'b', 'c']
def __init__(self, **kwargs):
for key, val in kwargs.items():
if key not in self._keys:
raise KeyError
super().__init__(self, **kwargs)
def __setitem__(self, key, val):
if key not in self._keys:
raise KeyError
else:
super().__setitem__(self, key, val)
def foo(**kwargs):
print(type(kwargs))
if __name__ == '__main__':
options = Options(a=1, b=2, c=3)
print(type(options))
foo(**options)
output:
<class '__main__.Options'>
<class 'dict'>
What i want is that kwargs inside foo is of type Options.
Is this possible without messing with python internals?
I guess i could just do without the unpacking but i like to have consistent IO syntax.
Thanks
edit:
i worked around this by doing:
def foo(**kwargs):
kwargs = Options(**kwargs)
print(type(kwargs))

Please correct me if I am wrong, but I believe the optional function arguments *args and **kwargs are by definition of type tuple and dict respectively.
The reason for my assumption is that these optional arguments are passed by value, i.e. Python makes a local copy to be used in the function. See the example:
class MyDict(dict):
def __init__(self, **kwargs):
super().__init__(self, **kwargs)
def test(**kwargs) -> None:
kwargs['a'] = 'b'
print(kwargs)
d = MyDict(a='a')
print(d)
test(**d)
print(d)
Resulting in:
{'a': 'a'}
{'a': 'b'}
{'a': 'a'}
Consequently, in order to retain well-defined behaviour it makes perfect sense that you are not allowed to set the type of kwargs.

How to add behaviour to standard dictionary?

I want to make a simple wrapper to the standard python dictionary, or maybe the defaultdict class where there is a default value.
The change I want to make is very simple: I would like to store in the dictionary data-structures that are not hashable due to the possibility of mutation, but I have guarantee in my code that I won't ever mutate them anyways.
My approach is detect if the key to the dictionary is hashable, if so proceed as usual. However if the key is not hashable, turn it into a string first then proceed as usual.

You can inherit from dict and override __setitem__ method:
class CustomDict(dict):
def __setitem__(self, key, value):
try:
hash(key)
except TypeError:
key = str(key)
super(CustomDict, self).__setitem__(key, value)
def __getitem__(self, key):
try:
hash(key)
except TypeError:
key = str(key)
return super(CustomDict, self).__getitem__(key)
data = CustomDict()
data["x"] = True
data[dict(foo='bar')] = False
print(data)
>>> {'x': True, "{'foo': 'bar'}": False}
assert data[dict(foo='bar')] == False
Or you can create custom dict-like object as described here.

Python OrderedDict with lambda [duplicate]

I would like to combine OrderedDict() and defaultdict() from collections in one object, which shall be an ordered, default dict.
Is this possible?

The following (using a modified version of this recipe) works for me:
from collections import OrderedDict, Callable
class DefaultOrderedDict(OrderedDict):
# Source: http://stackoverflow.com/a/6190500/562769
def __init__(self, default_factory=None, *a, **kw):
if (default_factory is not None and
not isinstance(default_factory, Callable)):
raise TypeError('first argument must be callable')
OrderedDict.__init__(self, *a, **kw)
self.default_factory = default_factory
def __getitem__(self, key):
try:
return OrderedDict.__getitem__(self, key)
except KeyError:
return self.__missing__(key)
def __missing__(self, key):
if self.default_factory is None:
raise KeyError(key)
self[key] = value = self.default_factory()
return value
def __reduce__(self):
if self.default_factory is None:
args = tuple()
else:
args = self.default_factory,
return type(self), args, None, None, self.items()
def copy(self):
return self.__copy__()
def __copy__(self):
return type(self)(self.default_factory, self)
def __deepcopy__(self, memo):
import copy
return type(self)(self.default_factory,
copy.deepcopy(self.items()))
def __repr__(self):
return 'OrderedDefaultDict(%s, %s)' % (self.default_factory,
OrderedDict.__repr__(self))

Here is another possibility, inspired by Raymond Hettinger's super() Considered Super, tested on Python 2.7.X and 3.4.X:
from collections import OrderedDict, defaultdict
class OrderedDefaultDict(OrderedDict, defaultdict):
def __init__(self, default_factory=None, *args, **kwargs):
#in python3 you can omit the args to super
super(OrderedDefaultDict, self).__init__(*args, **kwargs)
self.default_factory = default_factory
If you check out the class's MRO (aka, help(OrderedDefaultDict)), you'll see this:
class OrderedDefaultDict(collections.OrderedDict, collections.defaultdict)
| Method resolution order:
| OrderedDefaultDict
| collections.OrderedDict
| collections.defaultdict
| __builtin__.dict
| __builtin__.object
meaning that when an instance of OrderedDefaultDict is initialized, it defers to the OrderedDict's init, but this one in turn will call the defaultdict's methods before calling __builtin__.dict, which is precisely what we want.

If you want a simple solution that doesn't require a class, you can just use OrderedDict.setdefault(key, default=None) or OrderedDict.get(key, default=None). If you only get / set from a few places, say in a loop, you can easily just setdefault.
totals = collections.OrderedDict()
for i, x in some_generator():
totals[i] = totals.get(i, 0) + x
It is even easier for lists with setdefault:
agglomerate = collections.OrderedDict()
for i, x in some_generator():
agglomerate.setdefault(i, []).append(x)
But if you use it more than a few times, it is probably better to set up a class, like in the other answers.

Here's another solution to think about if your use case is simple like mine and you don't necessarily want to add the complexity of a DefaultOrderedDict class implementation to your code.
from collections import OrderedDict
keys = ['a', 'b', 'c']
items = [(key, None) for key in keys]
od = OrderedDict(items)
(None is my desired default value.)
Note that this solution won't work if one of your requirements is to dynamically insert new keys with the default value. A tradeoff of simplicity.
Update 3/13/17 - I learned of a convenience function for this use case. Same as above but you can omit the line items = ... and just:
od = OrderedDict.fromkeys(keys)
Output:
OrderedDict([('a', None), ('b', None), ('c', None)])
And if your keys are single characters, you can just pass one string:
OrderedDict.fromkeys('abc')
This has the same output as the two examples above.
You can also pass a default value as the second arg to OrderedDict.fromkeys(...).

Another simple approach would be to use dictionary get method
>>> from collections import OrderedDict
>>> d = OrderedDict()
>>> d['key'] = d.get('key', 0) + 1
>>> d['key'] = d.get('key', 0) + 1
>>> d
OrderedDict([('key', 2)])
>>>

A simpler version of #zeekay 's answer is:
from collections import OrderedDict
class OrderedDefaultListDict(OrderedDict): #name according to default
def __missing__(self, key):
self[key] = value = [] #change to whatever default you want
return value

A simple and elegant solution building on #NickBread.
Has a slightly different API to set the factory, but good defaults are always nice to have.
class OrderedDefaultDict(OrderedDict):
factory = list
def __missing__(self, key):
self[key] = value = self.factory()
return value

I created slightly fixed and more simplified version of the accepted answer, actual for python 3.7.
from collections import OrderedDict
from copy import copy, deepcopy
import pickle
from typing import Any, Callable
class DefaultOrderedDict(OrderedDict):
def __init__(
self,
default_factory: Callable[[], Any],
*args,
**kwargs,
):
super().__init__(*args, **kwargs)
self.default_factory = default_factory
def __getitem__(self, key):
try:
return super().__getitem__(key)
except KeyError:
return self.__missing__(key)
def __missing__(self, key):
self[key] = value = self.default_factory()
return value
def __reduce__(self):
return type(self), (self.default_factory, ), None, None, iter(self.items())
def copy(self):
return self.__copy__()
def __copy__(self):
return type(self)(self.default_factory, self)
def __deepcopy__(self, memo):
return type(self)(self.default_factory, deepcopy(tuple(self.items()), memo))
def __repr__(self):
return f'{self.__class__.__name__}({self.default_factory}, {OrderedDict(self).__repr__()})'
And, that may be even more important, provided some tests.
a = DefaultOrderedDict(list)
# testing default
assert a['key'] == []
a['key'].append(1)
assert a['key'] == [1, ]
# testing repr
assert repr(a) == "DefaultOrderedDict(<class 'list'>, OrderedDict([('key', [1])]))"
# testing copy
b = a.copy()
assert b['key'] is a['key']
c = copy(a)
assert c['key'] is a['key']
d = deepcopy(a)
assert d['key'] is not a['key']
assert d['key'] == a['key']
# testing pickle
saved = pickle.dumps(a)
restored = pickle.loads(saved)
assert restored is not a
assert restored == a
# testing order
a['second_key'] = [2, ]
a['key'] = [3, ]
assert list(a.items()) == [('key', [3, ]), ('second_key', [2, ])]

Inspired by other answers on this thread, you can use something like,
from collections import OrderedDict
class OrderedDefaultDict(OrderedDict):
def __missing__(self, key):
value = OrderedDefaultDict()
self[key] = value
return value
I would like to know if there're any downsides of initializing another object of the same class in the missing method.

i tested the default dict and discovered it's also sorted!
maybe it was just a coincidence but anyway you can use the sorted function:
sorted(s.items())
i think it's simpler

How to handle python exception inside if statement

For example I got this if statement:
if user.address.streetname == 'Target':
pass
elif:
(...)
else:
(...)
But not all users have enough money to have an address so it could raise an exception
In my case a django DoesNotExist exception. In this case it should assume false.
How to handle exception in this place without breaking if elif else flow?

If user.address is a model instance, you can do
if user and user.address and user.address.streetname and user.address.streetname == 'Target':
#Do something/
Or, you can also do:
address = getattr(user, 'address', None) if user else None
if address and getattr(address, 'streetname', '') == 'Target':
#do something

Use duck typing and create a sentinel object that is guaranteed to have a non-matching streetname attribute to use in place of an unavailable user.address.
poor_user_address = type('', (), {'streetname': None})()
if getattr(user, 'address', poor_user_address).streetname == "Target":
...
The call to type creates a minimal class with a class variable streetname; the rest of the details of the class are irrelevant. With duck typing, it doesn't matter that poor_user_address is an instance of a different class, as long as it exhibits the same behavior. In this case, the only expected behavior is to have a streetname attribute that can be compared to "Target".

Here's another stackoverflow question that answers this
The relevant part would be: hasattr(user, 'address')
If you add that to your if before accessing the property you can maintain the if/else flow.

try:
streetname = user.address.streetname
except DoesNotExist:
streetname = None
# or:
# streetname = NonexistenceSentinel()
if streetname == 'Target':
pass
elif:
...
else:
...
But probably what you are really looking for is some syntactic sugar to allow you to not put this in everywhere. Here's a recipe that lets you do that:
# 'Stupid' object that just returns itself.
# Any attribute will just return itself.
class SentinelObject(object):
__slots__ = []
def __init__(self):
pass
def __getattr__(self, key):
return self
def __nonzero__(self):
return False
def delegate_specials(specials):
specialnames = ['__%s__'%s for s in specials.split()]
def wrapit(cls, method):
return lambda self, *args, **kwargs: getattr(self._original_obj, method)(*args, **kwargs)
def dowrap(cls):
for n in specialnames:
setattr(cls, n,
wrapit(cls, n))
return cls
return dowrap
#delegate_specials('getitem setitem iter add sub mul div repr str len')
class SafeLookupObject(object):
__slots__ = ['_original_obj', '_sentinel_default']
__original_names__ = ['_original_obj', '_sentinel_default']
def __init__(self, original_obj, sentinel_default=SentinelObject()):
self._original_obj = original_obj
self._sentinel_default = sentinel_default
def __getattr__(self, key):
if key in self.__original_names__:
return object.__getattr__(self, key)
else:
try:
val = getattr(self._original_obj, key)
if callable(val):
return val
else:
return SafeLookupObject(val, self._sentinel_default)
except AttributeError:
return self._sentinel_default
def __setattr__(self, key, value):
if key in self.__original_names__:
return object.__setattr__(self, key, value)
else:
return setattr(self._original, key, value)
May not be perfect, looks OK at a first pass.
What this does: You pass in an original object, and a default val. The default val is a special SentinelObject (more on that in a minute). Only for getattr, if it doesn't exist, it returns the sentinel value. If it does exist, it checks to see if it's callable or not. If it's callable (i.e. a function), it returns it directly. If not, it wraps it in a SafeLookupObject and returns it.
The intention is that if you want to lookup x.y.z, you can just wrap x in the SafeLookupObject, and then x.y will automatically be wrapped as well, all the way down, so if it fails anywhere in the list, that value will be replaced with the sentinel object.
Therefore the special SentinelObject, which returns itself for whatever attribute you pass in. This with the above makes it fully recursive.
I.e. let's say you look up a.b.c.d with a as safe. a.b is OK, but then c does not exist in b. With a default of None, a.b.c returns None, but then a.b.c.d raises an exception.
If you use the SentinelObject as the default, then a.b.c instead returns a SentinelObject which both is boolean False, and can be matched against to determine a non-existent attribute. a.b.c.d also returns the same SentinelObject, and so it's now completely safe.

Immutable dictionary, only use as a key for another dictionary

I had the need to implement a hashable dict so I could use a dictionary as a key for another dictionary.
A few months ago I used this implementation: Python hashable dicts
However I got a notice from a colleague saying 'it is not really immutable, thus it is not safe. You can use it, but it does make me feel like a sad Panda'.
So I started looking around to create one that is immutable. I have no need to compare the 'key-dict' to another 'key-dict'. Its only use is as a key for another dictionary.
I have come up with the following:
class HashableDict(dict):
"""Hashable dict that can be used as a key in other dictionaries"""
def __new__(self, *args, **kwargs):
# create a new local dict, that will be used by the HashableDictBase closure class
immutableDict = dict(*args, **kwargs)
class HashableDictBase(object):
"""Hashable dict that can be used as a key in other dictionaries. This is now immutable"""
def __key(self):
"""Return a tuple of the current keys"""
return tuple((k, immutableDict[k]) for k in sorted(immutableDict))
def __hash__(self):
"""Return a hash of __key"""
return hash(self.__key())
def __eq__(self, other):
"""Compare two __keys"""
return self.__key() == other.__key() # pylint: disable-msg=W0212
def __repr__(self):
"""#see: dict.__repr__"""
return immutableDict.__repr__()
def __str__(self):
"""#see: dict.__str__"""
return immutableDict.__str__()
def __setattr__(self, *args):
raise TypeError("can't modify immutable instance")
__delattr__ = __setattr__
return HashableDictBase()
I used the following to test the functionality:
d = {"a" : 1}
a = HashableDict(d)
b = HashableDict({"b" : 2})
print a
d["b"] = 2
print a
c = HashableDict({"a" : 1})
test = {a : "value with a dict as key (key a)",
b : "value with a dict as key (key b)"}
print test[a]
print test[b]
print test[c]
which gives:
{'a': 1}
{'a': 1}
value with a dict as key (key a)
value with a dict as key (key b)
value with a dict as key (key a)
as output
Is this the 'best possible' immutable dictionary that I can use that satisfies my requirements? If not, what would be a better solution?

If you are only using it as a key for another dict, you could go for frozenset(mutabledict.items()). If you need to access the underlying mappings, you could then use that as the parameter to dict.
mutabledict = dict(zip('abc', range(3)))
immutable = frozenset(mutabledict.items())
read_frozen = dict(immutable)
read_frozen['a'] # => 1
Note that you could also combine this with a class derived from dict, and use the frozenset as the source of the hash, while disabling __setitem__, as suggested in another answer. (#RaymondHettinger's answer for code which does just that).

The Mapping abstract base class makes this easy to implement:
import collections
class ImmutableDict(collections.Mapping):
def __init__(self, somedict):
self._dict = dict(somedict) # make a copy
self._hash = None
def __getitem__(self, key):
return self._dict[key]
def __len__(self):
return len(self._dict)
def __iter__(self):
return iter(self._dict)
def __hash__(self):
if self._hash is None:
self._hash = hash(frozenset(self._dict.items()))
return self._hash
def __eq__(self, other):
return self._dict == other._dict

I realize this has already been answered, but types.MappingProxyType is an analogous implementation for Python 3.3. Regarding the original question of safety, there is a discussion in PEP 416 -- Add a frozendict builtin type on why the idea of a frozendict was rejected.

In order for your immutable dictionary to be safe, all it needs to do is never change its hash. Why don't you just disable __setitem__ as follows:
class ImmutableDict(dict):
def __setitem__(self, key, value):
raise Exception("Can't touch this")
def __hash__(self):
return hash(tuple(sorted(self.items())))
a = ImmutableDict({'a':1})
b = {a:1}
print b
print b[a]
a['a'] = 0
The output of the script is:
{{'a': 1}: 1}
1
Traceback (most recent call last):
File "ex.py", line 11, in <module>
a['a'] = 0
File "ex.py", line 3, in __setitem__
raise Exception("Can't touch this")
Exception: Can't touch this

Here is a link to pip install-able implementation of #RaymondHettinger's answer: https://github.com/pcattori/icicle
Simply pip install icicle and you can from icicle import FrozenDict!
Update: icicle has been deprecated in favor of maps: https://github.com/pcattori/maps (documentation, PyPI).

It appears I am late to post. Not sure if anyone else has come up with ideas. But here is my take on it. The Dict is immutable and hashable. I made it immutable by overriding all the methods, magic and otherwise, with a custom '_readonly' function that raises an Exception. This is done when the object is instantiated. To get around the problem of not being able to apply the values I set the 'hash' under '__new__'. I then I override the '__hash__'function. Thats it!
class ImmutableDict(dict):
_HASH = None
def __new__(cls, *args, **kwargs):
ImmutableDict._HASH = hash(frozenset(args[0].items()))
return super(ImmutableDict, cls).__new__(cls, args)
def __hash__(self):
return self._HASH
def _readonly(self, *args, **kwards):
raise TypeError("Cannot modify Immutable Instance")
__delattr__ = __setattr__ = __setitem__ = pop = update = setdefault = clear = popitem = _readonly
Test:
immutabled1 = ImmutableDict({"This": "That", "Cheese": "Blarg"})
dict1 = {immutabled1: "Yay"}
dict1[immutabled1]
"Yay"
dict1
{{'Cheese': 'Blarg', 'This': 'That'}: 'Yay'}

Variation of Raymond Hettinger's answer by wrapping the self._dict with types.MappingProxyType.
class ImmutableDict(collections.Mapping):
"""
Copies a dict and proxies it via types.MappingProxyType to make it immutable.
"""
def __init__(self, somedict):
dictcopy = dict(somedict) # make a copy
self._dict = MappingProxyType(dictcopy) # lock it
self._hash = None
def __getitem__(self, key):
return self._dict[key]
def __len__(self):
return len(self._dict)
def __iter__(self):
return iter(self._dict)
def __hash__(self):
if self._hash is None:
self._hash = hash(frozenset(self._dict.items()))
return self._hash
def __eq__(self, other):
return self._dict == other._dict
def __repr__(self):
return str(self._dict)

You can use an enum:
import enum
KeyDict1 = enum.Enum('KeyDict1', {'InnerDictKey1':'bla', 'InnerDictKey2 ':2})
d = { KeyDict1: 'whatever', KeyDict2: 1, ...}
You can access the enums like you would a dictionary:
KeyDict1['InnerDictKey2'].value # This is 2
You can iterate over the names, and get their values... It does everything you'd expect.

You can try using https://github.com/Lightricks/freeze
It provides recursively immutable and hashable dictionaries
from freeze import FDict
a_mutable_dict = {
"list": [1, 2],
"set": {3, 4},
}
a_frozen_dict = FDict(a_mutable_dict)
print(a_frozen_dict)
print(hash(a_frozen_dict))
# FDict: {'list': FList: (1, 2), 'set': FSet: {3, 4}}
# -4855611361973338606