So I'm writing a class that extends a dictionary which right now uses a method "dictify" to transform itself into a dict. What I would like to do instead though is change it so that calling dict() on the object results in the same behavior, but I don't know which method to override. Is this not possible, or I am I missing something totally obvious? (And yes, I know the code below doesn't work but I hope it illustrates what I'm trying to do.)
from collections import defaultdict
class RecursiveDict(defaultdict):
'''
A recursive default dict.
>>> a = RecursiveDict()
>>> a[1][2][3] = 4
>>> a.dictify()
{1: {2: {3: 4}}}
'''
def __init__(self):
super(RecursiveDict, self).__init__(RecursiveDict)
def dictify(self):
'''Get a standard dictionary of the items in the tree.'''
return dict([(k, (v.dictify() if isinstance(v, dict) else v))
for (k, v) in self.items()])
def __dict__(self):
'''Get a standard dictionary of the items in the tree.'''
print [(k, v) for (k, v) in self.items()]
return dict([(k, (dict(v) if isinstance(v, dict) else v))
for (k, v) in self.items()])
EDIT: To show the problem more clearly:
>>> b = RecursiveDict()
>>> b[1][2][3] = 4
>>> b
defaultdict(<class '__main__.RecursiveDict'>, {1: defaultdict(<class '__main__.RecursiveDict'>, {2: defaultdict(<class '__main__.RecursiveDict'>, {3: 4})})})
>>> dict(b)
{1: defaultdict(<class '__main__.RecursiveDict'>, {2: defaultdict(<class '__main__.RecursiveDict'>, {3: 4})})}
>>> b.dictify()
{1: {2: {3: 4}}}
I want dict(b) to be same as b.dictify()
Nothing wrong with your approach, but this is similar to the Autovivification feature of Perl, which has been implemented in Python in this question. Props to #nosklo for this.
class RecursiveDict(dict):
"""Implementation of perl's autovivification feature."""
def __getitem__(self, item):
try:
return dict.__getitem__(self, item)
except KeyError:
value = self[item] = type(self)()
return value
>>> a = RecursiveDict()
>>> a[1][2][3] = 4
>>> dict(a)
{1: {2: {3: 4}}}
EDIT
As suggested by #Rosh Oxymoron, using __missing__ results in a more concise implementation. Requires Python >= 2.5
class RecursiveDict(dict):
"""Implementation of perl's autovivification feature."""
def __missing__(self, key):
value = self[key] = type(self)()
return value
Do you want just to print it like a dict ? use this:
from collections import defaultdict
class RecursiveDict(defaultdict):
'''
A recursive default dict.
>>> a = RecursiveDict()
>>> a[1][2][3] = 4
>>> a.dictify()
{1: {2: {3: 4}}}
>>> dict(a)
{1: {2: {3: 4}}}
'''
def __init__(self):
super(RecursiveDict, self).__init__(RecursiveDict)
def dictify(self):
'''Get a standard dictionary of the items in the tree.'''
return dict([(k, (v.dictify() if isinstance(v, dict) else v))
for (k, v) in self.items()])
def __dict__(self):
'''Get a standard dictionary of the items in the tree.'''
print [(k, v) for (k, v) in self.items()]
return dict([(k, (dict(v) if isinstance(v, dict) else v))
for (k, v) in self.items()])
def __repr__(self):
return repr(self.dictify())
Maybe you are looking for __missing__ :
class RecursiveDict(dict):
'''
A recursive default dict.
>>> a = RecursiveDict()
>>> a[1][2][3] = 4
>>> a
{1: {2: {3: 4}}}
>>> dict(a)
{1: {2: {3: 4}}}
'''
def __missing__(self, key):
self[key] = self.__class__()
return self[key]
edit: As ironchefpython pointed out in comments, this isn't actually doing what I thought it did, as in my example b[1] is still a RecursiveDict. This may still be useful, as you essentially get an object pretty similar Rob Cowie's answer, but it is built on defaultdict.
You can get the behavior you want (or something very similar) by overriding __repr__, check this out:
class RecursiveDict(defaultdict):
def __init__(self):
super(RecursiveDict, self).__init__(RecursiveDict)
def __repr__(self):
return repr(dict(self))
>>> a = RecursiveDict()
>>> a[1][2][3] = 4
>>> a # a looks like a normal dict since repr is overridden
{1: {2: {3: 4}}}
>>> type(a)
<class '__main__.RecursiveDict'>
>>> b = dict(a)
>>> b # dict(a) gives us a normal dictionary
{1: {2: {3: 4}}}
>>> b[5][6] = 7 # obviously this won't work anymore
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 5
>>> type(b)
<type 'dict'>
There may be a better way to get to a normal dictionary view of the defaultdict than dict(self) but I couldn't find one, comment if you know how.
You can't do it.
I deleted my previous answer, because I found after looking at the source code, that if you call dict(d) on a d that is a subclass of dict, it makes a fast copy of the underlying hash in C, and returns a new dict object.
Sorry.
If you really want this behavior, you'll need to create a RecursiveDict class that doesn't inherit from dict, and implement the __iter__ interface.
You need to override __iter__.
def __iter__(self):
return iter((k, (v.dictify() if isinstance(v, dict) else v))
for (k, v) in self.items())
Instead of self.items(), you should use self.iteritems() on Python 2.
Edit: OK, This seems to be your problem:
>>> class B(dict): __iter__ = lambda self: iter(((1, 2), (3, 4)))
...
>>> b = B()
>>> dict(b)
{}
>>> class B(list): __iter__ = lambda self: iter(((1, 2), (3, 4)))
...
>>> b = B()
>>> dict(b)
{1: 2, 3: 4}
So this method doesn't work if the object you're calling dict() on is a subclass of dict.
Edit 2: To be clear, defaultdict is a subclass of dict. dict(a_defaultdict) is still a no-op.
Once you have your dictify function working just do
dict = dictify
Update:
Here is a short way to have this recursive dict:
>>> def RecursiveDict():
... return defaultdict(RecursiveDict)
Then you can:
d[1][2][3] = 5
d[1][2][4] = 6
>>> d
defaultdict(<function ReturnsRecursiveDict at 0x7f3ba453a5f0>, {1: defaultdict(<function ReturnsRecursiveDict at 0x7f3ba453a5f0>, {2: defaultdict(<function ReturnsRecursiveDict at 0x7f3ba453a5f0>, {3: 5, 4: 6})})})
I don't see a neat way to implement dictify.
Related
Let's say I have 2 "empty" dicts:
keys_in = ["in1","in2"]
dict2 ={key: None for key in keys_in}
keys_out = ["out1","out2"]
dict3 ={key: None for key in keys_out}
And I want to "map" one of the values from a dict to the other:
dict3["out1"] = dict2["in1"]
So that if the value of dict2["in1"] is set, I get the same value in dict3["out1"]
Is this possible? tried this and the value is not changed:
keys_in = ["in1","in2"]
dict2 ={key: None for key in keys_in}
keys_out = ["out1","out2"]
dict3 ={key: None for key in keys_out}
dict3["out1"] = dict2["in1"]
dict2["in1"] = 4
print(dict3["out1"]) #keeps being None
print(dict2["in1"]) # this of course is 4
I got this idea thinking on C++ pointers, I'm not sure if I'm in the right direction:s
No, this is not how assignment works in Python. Please watch this or read this. tl;dr: names are reassigned independently.
Without any further context, you could create a mutable object and use it in the values of both dicts. I have never seen a use case for this and we're probably dealing with an XY-problem here, but an example might be educational.
class MutableValue:
def __init__(self, value):
self.value = value
def __repr__(self):
return repr(self.value)
Demo:
>>> dict1 = {'in1': MutableValue(4)}
>>> dict2 = {'out1': dict1['in1']}
>>> dict1
{'in1': 4}
>>> dict2
{'out1': 4}
>>> dict1['in1'].value = 5
>>> dict1
{'in1': 5}
>>> dict2
{'out1': 5}
Just for fun, we can take this further:
from collections import UserDict
class MutableValueDict(UserDict):
def __setitem__(self, key, item):
if key in self:
self[key].value = item
else:
super().__setitem__(key, MutableValue(item))
Demo:
>>> dict1 = MutableValueDict({'in1': 4})
>>> dict2 = MutableValueDict({'out1': dict1['in1']})
>>> dict1
{'in1': 4}
>>> dict2
{'out1': 4}
>>> dict1['in1'] = 5 # no need to access .value
>>> dict1
{'in1': 5}
>>> dict2
{'out1': 5}
A tricky way to reach a similar result is to set a callable function to each value of the dependent dictionary, dict3, which has access to the value stored in the original dictionary, dict2. As drawbrack to access to a value in dict3 an empty call should de done, i.e. dict3['out1']().
Natural assumption: both dictionary have the same amount of items (and the right order).
# default initialisation
keys_in = ["in1","in2"]
dict2 = dict.fromkeys(keys_in, None)
# create a dictionary with callable values
keys_out = ["out1","out2"]
dict3 = {k3: lambda: dict2[k2] for k2, k3 in zip(keys_in, keys_out)}
# check content
print(dict3['out1']())
# add content to first dict
dict2['in1'] = 'dict2 -> in1'
dict2['in2'] = 'dict2 -> in2'
# check the updates
print(dict3['out1']())
#'dict2 -> in1'
print(dict3['out2']())
#'dict2 -> in2'
Here an functional programming way to access to the values of the dict3 just to stress that all of its values are callables:
from operator import methodcaller
vs = list(map(methodcaller('__call__'), dict3.values()))
print(vs)
I have a dictionary where key is string and value is list.
Now while adding a value associated with given key, I always have to check if there is any list yet, otherwise I have to initialize as empty list somewhat like following snippet:
if not k in myDict:
myDict[k] = []
myDict[k].append(v)
I am wondering if there is any way to combine these steps into single one in python 3.7.
There are at least three ways:
Use dict.setdefault
>>> data = {}
>>> data.setdefault('foo', []).append(42)
>>> data
{'foo': [42]}
Use defaultdict, which unlike .setdefault, takes a callable:
>>> from collections import defaultdict
>>> data = defaultdict(list)
>>> data
defaultdict(<class 'list'>, {})
>>> data['foo'].append(42)
>>> data
defaultdict(<class 'list'>, {'foo': [42]})
Finally, subclass dict and implement __missing__:
>>> class MyDict(dict):
... def __missing__(self, key):
... self[key] = value = []
... return value
...
>>> data = MyDict()
>>> data['foo'].append(42)
>>> data
{'foo': [42]}
Note, you can think of the last one as the most flexible, you have access to the actual key that's missing when you deal with it. defaultdict is a class factory, and it generates a subclass of dict as well. But, the callable is not passed any arguments, nevertheless, it is sufficient for most needs.
Further, note that the defaultdict and __missing__ approaches will keep the default behavior, this may be undesirable after you create your data structure, you probably want a KeyError usually, or at least, you don't want mydict[key] to add a key anymore.
In both cases, you can just create a regular dict from the dict subclasses, e.g. dict(data). This should generally be very fast, even for large dict objects, especially if it is a one-time cost. For defaultdict, you can also set the default_factory to None and the old behavior returns:
>>> data = defaultdict(list)
>>> data
defaultdict(<class 'list'>, {})
>>> data['foo']
[]
>>> data['bar']
[]
>>> data
defaultdict(<class 'list'>, {'foo': [], 'bar': []})
>>> data.default_factory = None
>>> data['baz']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'baz'
>>>
You can use dict.setdefault:
>>> d = {}
>>> d.setdefault('k', []).append(1)
>>> d
{'k': [1]}
>>> d.setdefault('k', []).append(2)
>>> d
{'k': [1, 2]}
Help on method_descriptor in dict:
dict.setdefault = setdefault(...)
D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D
is there any way to dynamically create missing keys if i want to want to set a variable in a subdictionary.
essentially I want to create any missing keys and set my value.
self.portdict[switchname][str(neighbor['name'])]['local']['ports'] = []
currently i'm doing it but its messy:
if not switchname in self.portdict:
self.portdict[switchname] = {}
if not str(neighbor['name']) in self.portdict[switchname]:
self.portdict[switchname][str(neighbor['name'])] = {}
if not 'local' in self.portdict[switchname][str(neighbor['name'])]:
self.portdict[switchname][str(neighbor['name'])]['local'] = {}
if not 'ports' in self.portdict[switchname][str(neighbor['name'])]['local']:
self.portdict[switchname][str(neighbor['name'])]['local']['ports'] = []
Is there any way to do this in one or two lines instead?
This is easier to do without recursion:
def set_by_path(dct, path, value):
ipath = iter(path)
p_last = next(ipath)
try:
while True:
p_next = next(ipath)
dct = dct.setdefault(p_last, {})
p_last = p_next
except StopIteration:
dct[p_last] = value
And a test case:
d = {}
set_by_path(d, ['foo', 'bar', 'baz'], 'qux')
print d # {'foo': {'bar': {'baz': 'qux'}}}
If you want to have it so you don't need a function, you can use the following defaultdict factory which allows you to nest things arbitrarily deeply:
from collections import defaultdict
defaultdict_factory = lambda : defaultdict(defaultdict_factory)
d = defaultdict_factory()
d['foo']['bar']['baz'] = 'qux'
print d
Use collections.defaultdict
self.portdict = defaultdict(lambda: defaultdict(lambda: defaultdict(lambda: defaultdict(lambda: []))))
I've run into a similar problem in the past. I found that defaultdict was the right answer for me—but writing the super long definitions (like the one in #o11c's answer or #Apero's answer) was no good. Here's what I came up with instead:
from collections import defaultdict
from functools import partial
def NestedDefaultDict(levels, baseFn):
def NDD(lvl):
return partial(defaultdict, NDD(lvl-1)) if lvl > 0 else baseFn
return defaultdict(NDD(levels-1))
This creates a dictionary with levels of nested dictionaries. So if you have levels=3, then you need 3 keys to access the bottom-level value. The second argument is a function which is used to create the bottom-level values. Something like list or lambda: 0 or even dict would work well.
Here's an example of using the "automatic" keys with 4 levels, and list as the default function:
>>> x = NestedDefaultDict(4, list)
>>> x[1][2][3][4].append('hello')
>>> x
defaultdict(<functools.partial object at 0x10b5c22b8>, {1: defaultdict(<functools.partial object at 0x10b5c2260>, {2: defaultdict(<functools.partial object at 0x10b5c2208>, {3: defaultdict(<type 'list'>, {4: ['hello']})})})})
I think that's basically what you'd want for the case in your question. Your 4 "levels" are switch-name, neighbor-name, local, & ports—and it looks like you want a list at the bottom-level to store your ports.
Another example using 2 levels and lambda: 0 as the default:
>>> y = NestedDefaultDict(2, lambda: 0)
>>> y['foo']['bar'] += 7
>>> y['foo']['baz'] += 10
>>> y['foo']['bar'] += 1
>>> y
defaultdict(<functools.partial object at 0x1021f1310>, {'foo': defaultdict(<function <lambda> at 0x1021f3938>, {'baz': 10, 'bar': 8})})
Have a close look to collections.defaultdict:
from collections import defaultdict
foo = defaultdict(dict)
foo['bar'] = defaultdict(dict)
foo['bar']['baz'] = defaultdict(dict)
foo['bar']['baz']['aaa'] = 1
foo['bor'] = 0
foo['bir'] = defaultdict(list)
foo['bir']['biz'].append(1)
foo['bir']['biz'].append(2)
print foo
defaultdict(<type 'dict'>, {'bir': defaultdict(<type 'list'>, {'biz': [1, 2]}), 'bor': 0, 'bar': defaultdict(<type 'dict'>, {'baz': defaultdict(<type 'dict'>, {'aaa': 1})})})
I know you can use setdefault(key, value) to set default value for a given key, but is there a way to set default values of all keys to some value after creating a dict ?
Put it another way, I want the dict to return the specified default value for every key I didn't yet set.
You can replace your old dictionary with a defaultdict:
>>> from collections import defaultdict
>>> d = {'foo': 123, 'bar': 456}
>>> d['baz']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'baz'
>>> d = defaultdict(lambda: -1, d)
>>> d['baz']
-1
The "trick" here is that a defaultdict can be initialized with another dict. This means
that you preserve the existing values in your normal dict:
>>> d['foo']
123
Use defaultdict
from collections import defaultdict
a = {}
a = defaultdict(lambda:0,a)
a["anything"] # => 0
This is very useful for case like this,where default values for every key is set as 0:
results ={ 'pre-access' : {'count': 4, 'pass_count': 2},'no-access' : {'count': 55, 'pass_count': 19}
for k,v in results.iteritems():
a['count'] += v['count']
a['pass_count'] += v['pass_count']
In case you actually mean what you seem to ask, I'll provide this alternative answer.
You say you want the dict to return a specified value, you do not say you want to set that value at the same time, like defaultdict does. This will do so:
class DictWithDefault(dict):
def __init__(self, default, **kwargs):
self.default = default
super(DictWithDefault, self).__init__(**kwargs)
def __getitem__(self, key):
if key in self:
return super(DictWithDefault, self).__getitem__(key)
return self.default
Use like this:
d = DictWIthDefault(99, x=5, y=3)
print d["x"] # 5
print d[42] # 99
42 in d # False
d[42] = 3
42 in d # True
Alternatively, you can use a standard dict like this:
d = {3: 9, 4: 2}
default = 99
print d.get(3, default) # 9
print d.get(42, default) # 99
defaultdict can do something like that for you.
Example:
>>> from collections import defaultdict
>>> d = defaultdict(list)
>>> d
defaultdict(<class 'list'>, {})
>>> d['new'].append(10)
>>> d
defaultdict(<class 'list'>, {'new': [10]})
Is this what you want:
>>> d={'a':1,'b':2,'c':3}
>>> default_val=99
>>> for k in d:
... d[k]=default_val
...
>>> d
{'a': 99, 'b': 99, 'c': 99}
>>>
>>> d={'a':1,'b':2,'c':3}
>>> from collections import defaultdict
>>> d=defaultdict(lambda:99,d)
>>> d
defaultdict(<function <lambda> at 0x03D21630>, {'a': 1, 'c': 3, 'b': 2})
>>> d[3]
99
Not after creating it, no. But you could use a defaultdict in the first place, which sets default values when you initialize it.
You can use the following class. Just change zero to any default value you like. The solution was tested in Python 2.7.
class cDefaultDict(dict):
# dictionary that returns zero for missing keys
# keys with zero values are not stored
def __missing__(self,key):
return 0
def __setitem__(self, key, value):
if value==0:
if key in self: # returns zero anyway, so no need to store it
del self[key]
else:
dict.__setitem__(self, key, value)
I have a large list like:
[A][B1][C1]=1
[A][B1][C2]=2
[A][B2]=3
[D][E][F][G]=4
I want to build a multi-level dict like:
A
--B1
-----C1=1
-----C2=1
--B2=3
D
--E
----F
------G=4
I know that if I use recursive defaultdict I can write table[A][B1][C1]=1, table[A][B2]=2, but this works only if I hardcode those insert statement.
While parsing the list, I don't how many []'s I need beforehand to call table[key1][key2][...].
You can do it without even defining a class:
from collections import defaultdict
nested_dict = lambda: defaultdict(nested_dict)
nest = nested_dict()
nest[0][1][2][3][4][5] = 6
Your example says that at any level there can be a value, and also a dictionary of sub-elements. That is called a tree, and there are many implementations available for them. This is one:
from collections import defaultdict
class Tree(defaultdict):
def __init__(self, value=None):
super(Tree, self).__init__(Tree)
self.value = value
root = Tree()
root.value = 1
root['a']['b'].value = 3
print root.value
print root['a']['b'].value
print root['c']['d']['f'].value
Outputs:
1
3
None
You could do something similar by writing the input in JSON and using json.load to read it as a structure of nested dictionaries.
I think the simplest implementation of a recursive dictionary is this. Only leaf nodes can contain values.
# Define recursive dictionary
from collections import defaultdict
tree = lambda: defaultdict(tree)
Usage:
# Create instance
mydict = tree()
mydict['a'] = 1
mydict['b']['a'] = 2
mydict['c']
mydict['d']['a']['b'] = 0
# Print
import prettyprint
prettyprint.pp(mydict)
Output:
{
"a": 1,
"b": {
"a": 1
},
"c": {},
"d": {
"a": {
"b": 0
}
}
}
I'd do it with a subclass of dict that defines __missing__:
>>> class NestedDict(dict):
... def __missing__(self, key):
... self[key] = NestedDict()
... return self[key]
...
>>> table = NestedDict()
>>> table['A']['B1']['C1'] = 1
>>> table
{'A': {'B1': {'C1': 1}}}
You can't do it directly with defaultdict because defaultdict expects the factory function at initialization time, but at initialization time, there's no way to describe the same defaultdict. The above construct does the same thing that default dict does, but since it's a named class (NestedDict), it can reference itself as missing keys are encountered. It is also possible to subclass defaultdict and override __init__.
This is equivalent to the above, but avoiding lambda notation. Perhaps easier to read ?
def dict_factory():
return defaultdict(dict_factory)
your_dict = dict_factory()
Also -- from the comments -- if you'd like to update from an existing dict, you can simply call
your_dict[0][1][2].update({"some_key":"some_value"})
In order to add values to the dict.
Dan O'Huiginn posted a very nice solution on his journal in 2010:
http://ohuiginn.net/mt/2010/07/nested_dictionaries_in_python.html
>>> class NestedDict(dict):
... def __getitem__(self, key):
... if key in self: return self.get(key)
... return self.setdefault(key, NestedDict())
>>> eggs = NestedDict()
>>> eggs[1][2][3][4][5]
{}
>>> eggs
{1: {2: {3: {4: {5: {}}}}}}
You may achieve this with a recursive defaultdict.
from collections import defaultdict
def tree():
def the_tree():
return defaultdict(the_tree)
return the_tree()
It is important to protect the default factory name, the_tree here, in a closure ("private" local function scope). Avoid using a one-liner lambda version, which is bugged due to Python's late binding closures, and implement this with a def instead.
The accepted answer, using a lambda, has a flaw where instances must rely on the nested_dict name existing in an outer scope. If for whatever reason the factory name can not be resolved (e.g. it was rebound or deleted) then pre-existing instances will also become subtly broken:
>>> nested_dict = lambda: defaultdict(nested_dict)
>>> nest = nested_dict()
>>> nest[0][1][2][3][4][6] = 7
>>> del nested_dict
>>> nest[8][9] = 10
# NameError: name 'nested_dict' is not defined
To add to #Hugo To have a max depth:
l=lambda x:defaultdict(lambda:l(x-1)) if x>0 else defaultdict(dict)
arr = l(2)
A slightly different possibility that allows regular dictionary initialization:
from collections import defaultdict
def superdict(arg=()):
update = lambda obj, arg: obj.update(arg) or obj
return update(defaultdict(superdict), arg)
Example:
>>> d = {"a":1}
>>> sd = superdict(d)
>>> sd["b"]["c"] = 2
You could use a NestedDict.
from ndicts.ndicts import NestedDict
nd = NestedDict()
nd[0, 1, 2, 3, 4, 5] = 6
The result as a dictionary:
>>> nd.to_dict()
{0: {1: {2: {3: {4: {5: 6}}}}}}
To install ndicts
pip install ndicts