Is any way to load JSON code into the recursive defaultdict to avoid KeyError, using Python 2.7?
For example:
from __future__ import print_function
from collections import defaultdict
import json
S = '{"a": [{"b": "TEST1", "p": "TEST2"}, {"b": "TEST3", "p": "TEST4"}]}'
d = json.loads(S)
nd = lambda: defaultdict(nd)
ni = nd()
print('1: ', ni['a'][0]['b'])
ni.update(d)
print('2: ', ni['a'][0]['b'])
print('3: ', ni['a'][1]['p'])
print('4: ', ni['a'][1]['c'])
Result
1: defaultdict(<function <lambda> at 0x0266F530>, {})
2: TEST1
3: TEST4
Traceback (most recent call last):
File "C:/...", line 16, in <module>
print('4: ', ni['a'][1]['c'])
KeyError: 'c'
It looks like after ni.update(d) ni works as dict, and not like recursive defaultdict. Is any way to add dict into a recursive defaultdict and keep its properties?
I would expect result on the step 4 similar to the following:
4: defaultdict(<function <lambda> at 0x0266F530>, {})
It's not enough to create a recursive dict. You have to load the JSON object into your dict recursively.
What happens in your code is that ni['a'] is a normal dict, not a dict of type nd. When you run ni.update(d), update isn't clever enough to traverse d, writing objects on each level into objects of similar type. Instead, it just copies the first-level values and keys of d into ni. The first-level values are just ordinary dicts or whatever.
To initialize ni correctly, you need to write a recursive function which determines whether each value is a dict, array or scalar. If it's a dict, it has to call nd, and then fill the result with values for which it will act in the same recursive way.
OR, you can use the argument object_hook to json.loads. See https://docs.python.org/2/library/json.html#json.load for details. Whatever function you pass here will be called on each dict created at any level of parsing. So the function
def defaultdict_from_dict(d):
nd = lambda: defaultdict(nd)
ni = nd()
ni.update(d)
return ni
or something similar would probably do the trick for you.
Related
my python code is about generating a sequence number from a dict keys, and my dict keys are defined with a range using cycle package in itertools module.
working example:
from itertools import cycle
e = {'Apple': cycle(range(1,999)),'Orange': cycle(range(1,999)),'Banana': cycle(range(1,999))}
def SequenceNum(f):
return f'{next(e[f])}'.zfill(3)
X = SequenceNum('Apple')
print(X)
output
001 --> it keeps incrementing in the range specified above in dict `e`
Challenge:
My requirement is to convert this dict of e into a json file. So it will load keys and values by parsing json file.
cat test.json
{
"DATA": {
"Apple": "cycle(range(1,999))",
"Orange": "cycle(range(1,999))",
"Banana": "cycle(range(1,999))"
}
}
(i had to put the dict values inside double quotes to avoid json file loading error.)
code
import json
from itertools import cycle
with open('test.json') as f:
FromJson = json.load(f)
d = FromJson['DATA']
print(d)
def SequenceNum(f):
return f'{next(d[f])}'.zfill(3)
X = SequenceNum('Apple')
i = 1
while i <= 10:
print(i, SequenceNum('Apple'))
i += 1
here new dict is d that loads json file and it will load the values in single quotes.
output
{'Apple': 'cycle(range(1,999))', 'Orange': 'cycle(range(1,999))', 'Banana': 'cycle(range(1,999))'} #THIS IS OUTPUT of 'd' after loading json file
Traceback (most recent call last):
File "c:\Users\chandu\Documents\test.py", line 14, in <module>
print(i, SequenceNum('Apple'))
File "c:\Users\chandu\Documents\test.py", line 12, in SequenceNum
return f'{next(d[f])}'.zfill(3)
TypeError: 'str' object is not an iterator
it is giving error because my dict values are not properly iterable by cycle itertools modules, since they are in quotes. i dont know if there is any other cause for this error.
please help to resolve this error,
Thanks in advance.
If you are sure what each value is, you can do eval with care:
def SequenceNum(f):
return f'{next(eval(d[f]))}'.zfill(3)
Note this is very dangerous to use, because eval evaluates anything that is passed into it and can cause harm.
This also will always fetch first value from iterator as it is evaluated new everytime. To solve, you can:
def SequenceNum(f):
return eval(d[f])
i = 1
seq_iter = SequenceNum('Apple')
while i <= 10:
print(i, f'{next(seq_iter)}'.zfill(3))
i += 1
This question already has answers here:
Is there a standard class for an infinitely nested defaultdict?
(6 answers)
Closed 9 years ago.
I'm creating a dictionary structure that is several levels deep. I'm trying to do something like the following:
dict = {}
dict['a']['b'] = True
At the moment the above fails because key 'a' does not exist. At the moment I have to check at every level of nesting and manually insert an empty dictionary. Is there some type of syntactic sugar to be able to do something like the above can produce:
{'a': {'b': True}}
Without having to create an empty dictionary at each level of nesting?
As others have said, use defaultdict. This is the idiom I prefer for arbitrarily-deep nesting of dictionaries:
def nested_dict():
return collections.defaultdict(nested_dict)
d = nested_dict()
d[1][2][3] = 'Hello, dictionary!'
print(d[1][2][3]) # Prints Hello, dictionary!
This also makes checking whether an element exists a little nicer, too, since you may no longer need to use get:
if not d[2][3][4][5]:
print('That element is empty!')
This has been edited to use a def rather than a lambda for pep8 compliance. The original lambda form looked like this below, which has the drawback of being called <lambda> everywhere instead of getting a proper function name.
>>> nested_dict = lambda: collections.defaultdict(nested_dict)
>>> d = nested_dict()
>>> d[1][2][3]
defaultdict(<function <lambda> at 0x037E7540>, {})
Use defaultdict.
Python: defaultdict of defaultdict?
Or you can do this, since dict() function can handle **kwargs:
http://docs.python.org/2/library/functions.html#func-dict
print dict(a=dict(b=True))
# {'a': {'b' : True}}
If the depth of your data structure is fixed (that is, you know in advance that you need mydict[a][b][c] but not mydict[a][b][c][d]), you can build a nested defaultdict structure using lambda expressions to create the inner structures:
two_level = defaultdict(dict)
three_level = defaultdict(lambda: defaultdict(dict))
four_level = defaultdict(lamda: defaultdict(lambda: defaultdict(dict)))
I'm trying to take a list of objects and rearrange them into a dictionary of nested dictionaries based on the objects' "keys." Each key contains the key of its parent dictionary in a known pattern. The problem I'm running into is being able to access an element like...
hier_data['1']['Assets']['1.2']['Assets']['1.2.3']['Assets']['1.2.3.4']['Assets']
...dynamically so as to add to it. Is there a way to build some sort of recursive function that will traverse down the dictionary based on the key? For example, if I needed to add the object with the key '1.2.3.4.5', is there a way to do...
hier_data['1']['Assets']['1.2']['Assets']['1.2.3']['Assets']['1.2.3.4']['Assets']['1.2.3.4.5'] = {...}
...dynamically and recursively?
I should note that the list I'm starting from is sorted by key so that I should always have the current object's parent's 'Assets' dictionary available for adding to.
You can use a recursive defaultdict:
>>> from collections import defaultdict
>>> l = lambda: defaultdict(l)
>>> d = defaultdict(l)
>>> d['123']['4234']['asd']['dsaf'] = 4
>>> d
defaultdict(<function <lambda> at 0x15f9578>, {'123': defaultdict(<function <lambda> at 0x15f9578>, {'4234': defaultdict(<function <lambda> at 0x15f9578>, {'asd': defaultdict(<function <lambda> at 0x15f9578>, {'dsaf': 4})})})})
Turns out what I was having trouble with was simpler than I thought. All I needed to do was something like this:
hier_data = {}
for id in sorted(data.iterkeys()):
key = id.split('.')
data[id]['Assets'] = {}
insert_point = hier_data
for i in range(len(key)/2-1):
insert_point = insert_point['.'.join(key[0:2*i+2])]['Assets']
insert_point[id] = data[id]
return hier_data
I thought getting keys from dictionaries (e.g. hier_data[...]) would return a copy of the object at that point in the dictionary, not a pointer to the object. Turns out all I needed was to iterate over my broken-up key to move my insert_point cursor to the correct spot to add in my object.
I need to add a key with a value that increases by one for every item in the nested dictionary. I have been trying to use the dict['key']='value' syntax but can't get it to work for a nested dictionary. I'm sure it's a very simple.
My Dictionary:
mydict={'a':{'result':[{'key1':'value1','key2':'value2'},
{'key1':'value3','key2':'value4'}]}}
This is the code that will add the key to the main part of the dictionary:
for x in range(len(mydict)):
number = 1+x
str(number)
mydict[d'index']=number
print mydict
#out: {d'index':d'1',d'a'{d'result':[...]}}
I want to add the new key and value to the small dictionaries inside the square parentheses:
{'a':{'result':[{'key1':'value1',...,'index':'number'}]}}
If I try adding more layers to the last line of the for loop I get a traceback error:
Traceback (most recent call last):
File "C:\Python27\program.py", line 34, in <module>
main()
File "C:\Python27\program.py", line 23, in main
mydict['a']['result']['index']=number
TypeError: list indices must be integers, not unicode
I've tried various different ways of listing the nested items but no joy. Can anyone help me out here?
The problem is that mydict is not simply a collection of nested dictionaries. It contains a list as well. Breaking up the definition helps clarify the internal structure:
dictlist = [{'key1':'value1','key2':'value2'},
{'key1':'value3','key2':'value4'}]
resultdict = {'result':dictlist}
mydict = {'a':resultdict}
So to access the innermost values, we have to do this. Working backwards:
mydict['a']
returns resultdict. Then this:
mydict['a']['result']
returns dictlist. Then this:
mydict['a']['result'][0]
returns the first item in dictlist. Finally, this:
mydict['a']['result'][0]['key1']
returns 'value1'
So now you just have to amend your for loop to iterate correctly over mydict. There are probably better ways, but here's a first approach:
for inner_dict in mydict['a']['result']: # remember that this returns `dictlist`
for key in inner_dict:
do_something(inner_dict, key)
I'm not fully sure what you're trying to do, but I think itertools.count would be able to help here.
>>> c = itertools.count()
>>> c.next()
0
>>> c.next()
1
>>> c.next()
2
>>> c.next()
3
... and so on.
Using this, you can keep incrementing the value that you want to use in your dicts
Hope this helps
I am writing a program that stores data in a dictionary object, but this data needs to be saved at some point during the program execution and loaded back into the dictionary object when the program is run again.
How would I convert a dictionary object into a string that can be written to a file and loaded back into a dictionary object? This will hopefully support dictionaries containing dictionaries.
The json module is a good solution here. It has the advantages over pickle that it only produces plain text output, and is cross-platform and cross-version.
import json
json.dumps(dict)
If your dictionary isn't too big maybe str + eval can do the work:
dict1 = {'one':1, 'two':2, 'three': {'three.1': 3.1, 'three.2': 3.2 }}
str1 = str(dict1)
dict2 = eval(str1)
print(dict1 == dict2)
You can use ast.literal_eval instead of eval for additional security if the source is untrusted.
I use json:
import json
# convert to string
input_ = json.dumps({'id': id_ })
# load to dict
my_dict = json.loads(input_)
Why not to use Python 3's inbuilt ast library's function literal_eval. It is better to use literal_eval instead of eval
import ast
str_of_dict = "{'key1': 'key1value', 'key2': 'key2value'}"
ast.literal_eval(str_of_dict)
will give output as actual Dictionary
{'key1': 'key1value', 'key2': 'key2value'}
And If you are asking to convert a Dictionary to a String then, How about using str() method of Python.
Suppose the dictionary is :
my_dict = {'key1': 'key1value', 'key2': 'key2value'}
And this will be done like this :
str(my_dict)
Will Print :
"{'key1': 'key1value', 'key2': 'key2value'}"
This is the easy as you like.
Use the pickle module to save it to disk and load later on.
Convert dictionary into JSON (string)
import json
mydict = { "name" : "Don",
"surname" : "Mandol",
"age" : 43}
result = json.dumps(mydict)
print(result[0:20])
will get you:
{"name": "Don", "sur
Convert string into dictionary
back_to_mydict = json.loads(result)
In Chinese language you should do the following adjustments:
import codecs
fout = codecs.open("xxx.json", "w", "utf-8")
dict_to_json = json.dumps({'text':"中文"},ensure_ascii=False,indent=2)
fout.write(dict_to_json + '\n')
You may find the json.dumps() method needs help handling some object types.
Credit goes to the top answer of this post for the following:
import json
json.dumps(my_dictionary, indent=4, sort_keys=True, default=str)
I think you should consider using the shelve module which provides persistent file-backed dictionary-like objects. It's easy to use in place of a "real" dictionary because it almost transparently provides your program with something that can be used just like a dictionary, without the need to explicitly convert it to a string and then write to a file (or vice-versa).
The main difference is needing to initially open() it before first use and then close() it when you're done (and possibly sync()ing it, depending on the writeback option being used). Any "shelf" file objects create can contain regular dictionaries as values, allowing them to be logically nested.
Here's a trivial example:
import shelve
shelf = shelve.open('mydata') # open for reading and writing, creating if nec
shelf.update({'one':1, 'two':2, 'three': {'three.1': 3.1, 'three.2': 3.2 }})
shelf.close()
shelf = shelve.open('mydata')
print shelf
shelf.close()
Output:
{'three': {'three.1': 3.1, 'three.2': 3.2}, 'two': 2, 'one': 1}
If you care about the speed use ujson (UltraJSON), which has the same API as json:
import ujson
ujson.dumps([{"key": "value"}, 81, True])
# '[{"key":"value"},81,true]'
ujson.loads("""[{"key": "value"}, 81, true]""")
# [{u'key': u'value'}, 81, True]
I use yaml for that if needs to be readable (neither JSON nor XML are that IMHO), or if reading is not necessary I use pickle.
Write
from pickle import dumps, loads
x = dict(a=1, b=2)
y = dict(c = x, z=3)
res = dumps(y)
open('/var/tmp/dump.txt', 'w').write(res)
Read back
from pickle import dumps, loads
rev = loads(open('/var/tmp/dump.txt').read())
print rev
I figured out the problem was not with my dict object it was the keys and values that were of RubyString type after loading it with RubyMarshl 'loads' method
So i did this:
dic_items = dict.items()
new_dict = {str(key): str(value) for key, value in dic_items}