We Keep Coding

Python throws KeyError: 1 when using a Dictionary restored from a file

In my program, I have certain settings that can be modified by the user, saved on the disk, and then loaded when application is restarted. Some these settings are stored as dictionaries. While trying to implement this, I noticed that after a dictionary is restored, it's values cannot be used to access values of another dictionary, because it throws a KeyError: 1 exception.
This is a minimal code example that ilustrates the issue:
import json
motorRemap = {
1: 3,
2: 1,
3: 6,
4: 4,
5: 5,
6: 2,
}
motorPins = {
1: 6,
2: 9,
3: 10,
4: 11,
5: 13,
6: 22
}
print(motorPins[motorRemap[1]]); #works correctly
with open('motorRemap.json', 'w') as fp:
json.dump(motorRemap, fp)
with open('motorRemap.json', 'r') as fp:
motorRemap = json.load(fp)
print(motorPins[motorRemap[1]]); #throws KeyError: 1
You can run this code as it is. First print statement works fine, but after the first dictionary is saved and restored, it doesn't work anymore. Apparently, saving/restoring somehow breaks that dictionary.
I have tried saving and restoring with json and pickle libraries, and both produce in the same error. I tried printing values of the first dictionary after it is restored directly ( print(motorRemap[1]), and it prints out correct values without any added spaces or anything. KeyError usually means that the specified key doesn't exist in the dictionary, but in this instance print statement shows that it does exist - unless some underlying data types have changed or something. So I am really puzzled as to why this is happening.
Can anyone help me understand what is causing this issue, and how to solve it?

What happens becomes clear when you look at what json.dump wrote into motorRemap.json:
{"1": 3, "2": 1, "3": 6, "4": 4, "5": 5, "6": 2}
Unlike Python, json can only use strings as keys. Python, on the other hand, allows many different types for dictionary keys, including booleans, floats and even tuples:
my_dict = {False: 1,
3.14: 2,
(1, 2): 3}
print(my_dict[False], my_dict[3.14], my_dict[(1, 2)])
# Outputs '1 2 3'
The json.dump function automatically converts some of these types to string when you try to save the dictionary to a json file. False becomes "false", 3.14 becomes "3.14" and, in your example, 1 becomes "1". (This doesn't work for the more complex types such as a tuple. You will get a TypeError if you try to json.dump the above dictionary where one of the keys is (1, 2).)
Note how the keys change when you dump and load a dictionary with some of the Python-specific keys:
import json
my_dict = {False: 1,
3.14: 2}
print(my_dict[False], my_dict[3.14])
with open('my_dict.json', 'w') as fp:
json.dump(my_dict, fp)
# Writes {"false": 1, "3.14": 2} into the json file
with open('my_dict.json', 'r') as fp:
my_dict = json.load(fp)
print(my_dict["false"], my_dict["3.14"])
# And not my_dict[False] or my_dict[3.14] which raise a KeyError
Thus, the solution to your issue is to access the values using strings rather than integers after you load the dictionary from the json file.
print(motorPins[motorRemap["1"]]) instead of your last line will fix your code.
From a more general perspective, it might be worth considering keeping the keys as strings from the beginning if you know you will be saving the dictionary into a json file. You could also convert the values back to integers after loading as discussed here; however, that can lead to bugs if not all the keys are integers and is not a very good idea in bigger scale.
Checkout pickle if you want to save the dictionary keeping the Python format. It is, however, not human-readable unlike json and it's also Python-specific so it cannot be used to transfer data to other languages, missing virtually all the main benefits of json.
If you want to save and load the dictionary using pickle, this is how you would do it:
# import pickle
...
with open('motorRemap.b', 'wb') as fp:
pickle.dump(motorRemap, fp)
with open('motorRemap.b', 'rb') as fp:
motorRemap = pickle.load(fp)
...

since the keys (integers) from a dict will be written to the json file as strings, we can modify the reading of the json file. using a dict comprehension restores the original dict values:
...
with open('motorRemap.json', 'r') as fp:
motorRemap = {int(item[0]):item[1] for item in json.load(fp).items()}
...

Encoding and Serialization Code Gives Different Result C# and Python

I tried to convert this C# code (encoding and serialization) to Python but results are different. Why?
var dictt = new Dictionary<string, object>
{
{ "aaa", "6mjDx3Cya4JvbTLMenPpXA==" },
{ "bbb", "4U5M+V2yoIA7rWj46rdhTBgpEjf1zYK0m11lDM7DRCI="},
};
JavaScriptSerializer serialzr = new JavaScriptSerializer();
return Convert.ToBase64String(Encoding.UTF8.GetBytes(serialzr.Serialize(dictt))); //eyJpdiI6IjZtakR4M0N5YTRKdmJUTE1lblBwWEE9PSIsInZhbHVlIjoiNFU1TStWMnlvSUE3cldqNDZyZGhUQmdwRWpmMXpZSzBtMTFsRE03RFJDST0ifQ==
dictt = {
"aaa": "6mjDx3Cya4JvbTLMenPpXA==",
"bbb": "4U5M+V2yoIA7rWj46rdhTBgpEjf1zYK0m11lDM7DRCI="
}
y = json.dumps(dictt)
#y= {"aaa": "6mjDx3Cya4JvbTLMenPpXA==", "bbb": "4U5M+V2yoIA7rWj46rdhTBgpEjf1zYK0m11lDM7DRCI="}
json_object = json.loads(y) #convert it to json, like serialize in C#
# json_object = {'aaa': '6mjDx3Cya4JvbTLMenPpXA==', 'bbb': '4U5M+V2yoIA7rWj46rdhTBgpEjf1zYK0m11lDM7DRCI='}
json_object_utf8_encoded = str(json_object).encode('utf8') #encode utf8
#json_object_utf8_encoded = b"{'aaa': '6mjDx3Cya4JvbTLMenPpXA==', 'bbb': '4U5M+V2yoIA7rWj46rdhTBgpEjf1zYK0m11lDM7DRCI='}"
json_base64 = base64.b64encode(json_object_utf8_encoded) #convert to base64 string
#json_base64 = "b'eydpdic6ICc2bWpEeDNDeWE0SnZiVExNZW5QcFhBPT0nLCAndmFsdWUnOiAnNFU1TStWMnlvSUE3cldqNDZyZGhUQmdwRWpmMXpZSzBtMTFsRE03RFJDST0nfQ=='"
json_base64_str = json_base64.decode("utf-8")
return json_base64_str
#eydpdic6ICc2bWpEeDNDeWE0SnZiVExNZW5QcFhBPT0nLCAndmFsdWUnOiAnNFU1TStWMnlvSUE3cldqNDZyZGhUQmdwRWpmMXpZSzBtMTFsRE03RFJDST0nfQ==

There's a small difference between the following two python commands, resulting with two different outputs: " vs. '.
While this one:
serialized = json.dumps(dict)
print(serialized)
outputs:
{"aaa": "6mjDx3Cya4JvbTLMenPpXA==", "bbb": "4U5M+V2yoIA7rWj46rdhTBgpEjf1zYK0m11lDM7DRCI="}
This one:
serialized = json.dumps(dict)
deserialized = json.loads(serialized)
print(str(deserialized))
outputs:
{'aaa': '6mjDx3Cya4JvbTLMenPpXA==', 'bbb': '4U5M+V2yoIA7rWj46rdhTBgpEjf1zYK0m11lDM7DRCI='}
You work with the latter one: you're unnecessarily deserializing the json, and then calling str upon it, giving you the different result. Just drop the json.loads step.
EDIT:
The returned values (C# vs python) will still not match, because C# serializer (whether it's JavaScriptSerializer or the more common JsonConvert.Serialize of Newtonsoft) outputs a json with no spaces:
{"aaa":"6mjDx3Cya4JvbTLMenPpXA==","bbb":"4U5M+V2yoIA7rWj46rdhTBgpEjf1zYK0m11lDM7DRCI="}
while python's json.dumps outputs a json with spaces:
{"aaa": "6mjDx3Cya4JvbTLMenPpXA==", "bbb": "4U5M+V2yoIA7rWj46rdhTBgpEjf1zYK0m11lDM7DRCI="}
Even if you'd pass an indentation argument to json.dumps (e.g., json.dumps(dict, indent=2)), you can't do that with JavaScriptSerializer, and although you could do it with Newtonsoft's JsonConvert.Serialize it would still not work, since the latter uses CRLF characters for the indentation, while python doesn't.
That being said, it shouldn't matter, because the logical output is the same: A Base64 encoding of the serialized dictionary.

How to maintain dictionary element order in JSON dump in Python

Due to a very helpful person on this site, I have been able to put together a script which takes user names (inputted by user) and then use a loop to append each of those user names to a specific JSON structure. However, I have noticed that the order within my JSON dump is not maintained. For example, here is what my loop and JSON dump looks like:
list_users = []
for user in users:
list_users.append({"name": user,
"member": 123,
"comment": "this is a comment"})
json_user_list_encoding = json.dumps(list_users, indent=2)
My print looks like this:
({"member": 123,
"comment": "this is a comment"
"name": user
})
I was wondering if it was possible to maintain the same order in "list_users" when I use the JSON dump. I have looked at this site a bit and from what I have read, I will have to assign keys in order to maintain a specific order. However, I am not sure how to do this being that I only have one JSON object. I hope this make sense. Thanks for the help.

If you just want them ordered reliably/repeatably, but don't care about the order itself, then json dumps accepts a sort_keys kwarg that can help:
>>> json.dumps({'z': 3, 'b': 2, 'a': 1})
'{"a": 1, "z": 3, "b": 2}'
>>> json.dumps({'z': 3, 'b': 2, 'a': 1}, sort_keys=True)
'{"a": 1, "b": 2, "z": 3}'
Otherwise, put the users in an OrderedDict instead of a plain dict.
from collections import OrderedDict
list_users.append(OrderedDict([
("name", user),
("member", 123),
("comment", "this is a comment"),
]))
If you need them to deserialize to the same order too, use the object_pairs_hook when loading.

Making pretty json string with python's json

I am using python's json to produce a json file and cannot manage to format it the way I want.
What I have is a dictionary with some keys, and each key has a list of numbers attached to it:
out = {"a": [1,2,3], "b": [4,5,6]}
What I want to do is produce a JSON string where each list is in its own line, like so:
{
"a": [1,2,3],
"b": [4,5,6]
}
However, I can only get
>>> json.dumps(out)
'{"a": [1, 2, 3], "b": [4, 5, 6]}'
which has no new lines, or
>>> print json.dumps(out, indent=2)
{
"a": [
1,
2,
3
],
"b": [
4,
5,
6
]
}
which has waaay to many. Is there a simply way to produce the string I want? I can do it manually, of course, but I am wondering if it is possible with json alone...

You can't do that with the json module, no. It was never the goal for the module to allow this much control over the output.
The indent option is only meant to aid debugging. JSON parsers don't care about how much whitespace is used in-between elements.

Convert a python dict to a string and back

I am writing a program that stores data in a dictionary object, but this data needs to be saved at some point during the program execution and loaded back into the dictionary object when the program is run again.
How would I convert a dictionary object into a string that can be written to a file and loaded back into a dictionary object? This will hopefully support dictionaries containing dictionaries.

The json module is a good solution here. It has the advantages over pickle that it only produces plain text output, and is cross-platform and cross-version.
import json
json.dumps(dict)

If your dictionary isn't too big maybe str + eval can do the work:
dict1 = {'one':1, 'two':2, 'three': {'three.1': 3.1, 'three.2': 3.2 }}
str1 = str(dict1)
dict2 = eval(str1)
print(dict1 == dict2)
You can use ast.literal_eval instead of eval for additional security if the source is untrusted.

I use json:
import json
# convert to string
input_ = json.dumps({'id': id_ })
# load to dict
my_dict = json.loads(input_)

Why not to use Python 3's inbuilt ast library's function literal_eval. It is better to use literal_eval instead of eval
import ast
str_of_dict = "{'key1': 'key1value', 'key2': 'key2value'}"
ast.literal_eval(str_of_dict)
will give output as actual Dictionary
{'key1': 'key1value', 'key2': 'key2value'}
And If you are asking to convert a Dictionary to a String then, How about using str() method of Python.
Suppose the dictionary is :
my_dict = {'key1': 'key1value', 'key2': 'key2value'}
And this will be done like this :
str(my_dict)
Will Print :
"{'key1': 'key1value', 'key2': 'key2value'}"
This is the easy as you like.

Use the pickle module to save it to disk and load later on.

Convert dictionary into JSON (string)
import json
mydict = { "name" : "Don",
"surname" : "Mandol",
"age" : 43}
result = json.dumps(mydict)
print(result[0:20])
will get you:
{"name": "Don", "sur
Convert string into dictionary
back_to_mydict = json.loads(result)

In Chinese language you should do the following adjustments:
import codecs
fout = codecs.open("xxx.json", "w", "utf-8")
dict_to_json = json.dumps({'text':"中文"},ensure_ascii=False,indent=2)
fout.write(dict_to_json + '\n')

You may find the json.dumps() method needs help handling some object types.
Credit goes to the top answer of this post for the following:
import json
json.dumps(my_dictionary, indent=4, sort_keys=True, default=str)

I think you should consider using the shelve module which provides persistent file-backed dictionary-like objects. It's easy to use in place of a "real" dictionary because it almost transparently provides your program with something that can be used just like a dictionary, without the need to explicitly convert it to a string and then write to a file (or vice-versa).
The main difference is needing to initially open() it before first use and then close() it when you're done (and possibly sync()ing it, depending on the writeback option being used). Any "shelf" file objects create can contain regular dictionaries as values, allowing them to be logically nested.
Here's a trivial example:
import shelve
shelf = shelve.open('mydata') # open for reading and writing, creating if nec
shelf.update({'one':1, 'two':2, 'three': {'three.1': 3.1, 'three.2': 3.2 }})
shelf.close()
shelf = shelve.open('mydata')
print shelf
shelf.close()
Output:
{'three': {'three.1': 3.1, 'three.2': 3.2}, 'two': 2, 'one': 1}

If you care about the speed use ujson (UltraJSON), which has the same API as json:
import ujson
ujson.dumps([{"key": "value"}, 81, True])
# '[{"key":"value"},81,true]'
ujson.loads("""[{"key": "value"}, 81, true]""")
# [{u'key': u'value'}, 81, True]

I use yaml for that if needs to be readable (neither JSON nor XML are that IMHO), or if reading is not necessary I use pickle.
Write
from pickle import dumps, loads
x = dict(a=1, b=2)
y = dict(c = x, z=3)
res = dumps(y)
open('/var/tmp/dump.txt', 'w').write(res)
Read back
from pickle import dumps, loads
rev = loads(open('/var/tmp/dump.txt').read())
print rev

I figured out the problem was not with my dict object it was the keys and values that were of RubyString type after loading it with RubyMarshl 'loads' method
So i did this:
dic_items = dict.items()
new_dict = {str(key): str(value) for key, value in dic_items}