I don't have much experience in Python and I've ran into problem converting sql query data which is technically a list containing a JSON string into a Python dictionary. I'm querying the sqlite3 database which is returning a piece of data like this:
def get_answer(self, id):
self.__cur.execute('select answer from some_table where id= %s;' % id)
res_data = self.__cur.fetchall()
return res_data
The data is a single JSON format element which its simplified version looks like this:
[
{"ind": [ {"v": 101}, {"v": 102}, {"v": 65 }]},
{"ind": [ {"v": 33}, {"v": 102}, {"v": 65}]}
]
But when I try to convert the res_data to JSON, with code like this:
temp_json = simplejson.dumps(get_answer(trace_id))
it returns a string and when I get the len(temp_json) it returns the number of characters in res_data instead of the number of objects in res_data. However, if I use Visual Studio's JSON visualizer on what get_answer(trace_id) returns, it properly shows each of the objects res_data.
I also tried to convert the res_data to a dictionary with code like this:
dict_data = ast.literal_eval(Json_data)
or
dict_data = ast.literal_eval(Json_data[0])
and in both cases it throws a "malformed string" exception. I tried to write it to a file and read it back as a JSON but it didn't worked.
Before doing that I had the copy pasted the res_data manually and used:
with open(file_name) as json_file:
Json_data = simplejson.load(json_file)
and it worked like a charm. I've been experimenting different ways stated in SO and elsewhere but although the problem seems very straight forward, I still haven't found a solution so your help is highly appreciated.
OK, I finally found the solution:
states = json.loads(temp_json[0][0])
one confusing issue was that states = json.loads(temp_json[0]) was throwing the "Expected string or buffer" exception and temp_json was a list containing only one element, so I didn't think I will get anything from temp_json[0][0].
I hope it helps others too!
I think you are confusing the data formats. What you supposedly get back from your database wrapper seems to be a list of dictionaries (it is not SQL either - your question title is misleading). But I actually think that sqlite3 would give you a list of tuples.
JSON is a text format or more precisely the serialization of an object to a string. That's why json.dumps (= dump to string) results in a string and json.loads(= load string) would result in a python object, a dictionary or a list of dictionaries).
json.dumps does NOT mean "dump JSON to string". It is rather the .dumps method of the json module which takes a Python object (dict, list) and serializes it to JSON.
I will extend the answer if I understand what exactly you want to achieve but you get JSON with json.dumps(), JSON is a string format. In contrast simplejson.load() gives you a Python list or dict.
Did you try json.loads() just in case the return from your database is actually a string (which you could easily test).
Related
I have a data.json file, which looks like this:
["{\"Day\":\"Today\",\"Event\":\"1\", \"Date\":\"2019-03-20\"}"]
I am trying to get "Event" from this file using python and miserably failing at this.
with open('data.json', 'r') as json_file:
data = json.load(json_file)
print (data['Event'])
I get the following error:
TypeError: list indices must be integers or slices, not str
And even when I try
print (data[0]['Event'])
then I get this error:
TypeError: string indices must be integers
One more thing:
print(type(data))
gives me "list"
I have searched all over and have not found a solution to this. I would really appreciate your suggestions.
You could use the ast module for this:
import ast
mydata = ["{\"Day\":\"Today\",\"Event\":\"1\", \"Date\":\"2019-03-20\"}"]
data = ast.literal_eval(mydata[0])
data
{'Day': 'Today', 'Event': '1', 'Date': '2019-03-20'}
data['Event']
'1'
Edit
Your original code does load the data into a list structure, but only contains a single string entry inside that list, despite proper json syntax. ast, like json, will parse that string entry into a python data structure, dict.
As it sits, when you try to index that list, it's not the same as calling a key in a dict, hence the slices cannot be str:
alist = [{'a':1, 'b':2, 'c':3}]
alist['a']
TypeError
# need to grab the dict entry from the list
adict = alist[0]
adict['a']
1
You need to convert the elements in data to dict using json module.
Ex:
import json
with open(filename) as infile:
data = json.load(infile)
for d in data:
print(json.loads(d)['Event'])
Or:
data = list(map(json.loads, data))
print(data[0]["Event"])
Output:
1
Your problem is that you are parsing it as a list that consists of a single element that is a string.
["{\"Day\":\"Today\",\"Event\":\"1\", \"Date\":\"2019-03-20\"}"]
See how the entire content of the list is surrounded by " on either side and every other " is preceded by a \? The slash generally means to ignore the special meaning the following character might have, but interpret it as purely a string.
If you have control over the file's contents, the easiest solution would be to adjust it. You will want it to be in a format like this:
[{"Day":"Today", "Event": "1", "Date": "2019-03-20"}]
Edit: As others have suggested, you can also parse it in its current state. Granted, cleaning the data is tedious, but oftentimes worth the effort. Though this may not be one of those cases. I'm leaving this answer up anyway because it may help with explaining why OPs initial attempt did not work, and why he received the error messages he got.
Hello all and sorry if the title was worded poorly. I'm having a bit of trouble wrapping my head around how to solve this issue I have encountered. I would have liked to simply pass a dict as the value for this key in my json obj but sadly I have to pass it as a string. So, I have a json dict object that looks like this
data = {"test": "Fuzz", "options": "'{'size':'Regular','connection':'unconnected'}'"}. Obviously, I would prefer that the second dict value weren't a string representation of a dictionary but rather a dictionary. Is the best route here to just strip the second and second to last single quotes for the data[options] or is there a better alternative?
Sorry for any confusion. This is how the json object looks after I perform
json.dump(data, <filename>)
The value for options can be thought of as another variable say x and it's equivalent to '{'size':'Regular','connection':'unconnected'}'
I could do x[1:-1] but I'm not sure if that is the most pythonic way to do things here.
import ast
bad_string_dict = "'{'size':'Regular','connection':'unconnected'}'"
good_string_dict = bad_string_dict.strip("'")
good_dict = ast.literal_eval(good_string_dict)
print(good_dict)
You will have to strip quotation mark, no other way around
Given OP's comments I suggest the following:
Set the environment variable to a known data format (example: json/yaml/...), not a specific language (python)
Use the json module (or the format you've chosen) to load the data
The data should look like this:
raw_data = {"test": "Fuzz", "options": "{\"size\": \"Regular\", \"connection\": \"unconnected\"}"}
And the code should look like this:
raw_options = raw_data['options']
options = json.loads(raw_options)
data = {**raw_data, 'options': options}
I have a simple file (username.json) as shown below:
{"lastname": "doe", "firstname": "john"}
I use the following code to read the file:
with open(filename) as file_obj:
dictionary = json.load(file_obj)
print(dictionary['firstname'])
But when I print the dictionary value for the key "firstname" it prints nothing.
When I print the dictionary I get the following:
{u'lastname': u'doe', u'firstname': u'john'}
I know that "u" stands for unicode but for some reason I am not able to use the firstname and lastname keys.
UPDATE:
For some reason it works now!
json.loads converts a json object to the python equivalent.
This means it uses lists and dicts instead of arrays and objects. You are seeing the representation of the former.
doctionary["firstname"] will get you the value in first name (ie, "doe") while it's still a python object.
If you want to see json again, you'll need to pass it through json.dumps - but of course you won't be able to manipulate it as above when in that format.
A bit lost after much research. My code below parses the JSON to a dictionary I have thought using json load
response = json.load(MSW) # -->Will take a JSON String & Turn it into a python dict
Using the iteration below I return a series like this which is fine
{u'swell': {u'components': {u'primary': {u'direction': 222.5}}}}
{u'swell': {u'components': {u'primary': {u'direction': 221.94}}}}
ourResult = response
for rs in ourResult:
print rs
But how oh how do I access the 222.5 value. The above appears to just be one long string eg response[1] and not a dictionary structure at all.
In short all I need is the numerical value (which I assume is a part of that sting) so I can test conditions in the rest of my code. Is is a dictionary? With thanks as new and lost
You have to use python syntax as follows:
>>> print response['swell']['components']['primary']['direction']
222.5
Just access the nested dictionaries, unwrapping each layer with an additional key:
for rs in ourResult:
print rs['components']['primary']['direction']
When I call the JSON file for Vincent van Gogh's List of Works wikipedia page, using this url,
it obviously returns a huge blob of text which I believe is some sort of dictionary of lists.
Now, someone has already shown me Python's import wikipedia feature, so skip that. How can I decode this JSON? I feel like I have tried everything in Python 3's library, and always get an error, like I get if I try this code for example:
data = urllib.request.urlopen(long_json_url)
stuff = json.load(data) #or json.loads(data)
print(stuff)
it returns
TypeError: the JSON object must be str, not 'bytes'
Or if I try this code:
data = urllib.request.urlopen(longurl)
json_string = data.read().decode('utf-8')
json_data = json.loads(json_string)
print(json_data)
It doesn't return an error, but just what looks like nothing
>>>
>>>
But if I highlight that empty space and paste it, it pastes the same blob of text.
{'warnings': {'main': {'*': "Unrecognized parameter: 'Page'"}}, 'query': {'normalized': [{'from': 'list of works by Vincent van Gogh',... etc
If I try a for loop:
for entry in json_data:
print(entry)
It returns
>>>
query
warnings
>>>
And that's it. So it's not returning an error there, but not really much else, just two values? How would you make the JSON data into a workable Python dict or list? Or at the very least, into a more vertical format that I could actually read?
How would you make the JSON data into a workable Python dict or list?
You're already doing that with
json_data = json.loads(json_string)
This however:
for entry in json_data:
print(entry)
will only print the keys of your dictionaries. If you want to print the values, you need to use:
for entry in json_data:
print(json_data[entry])
if you inspect the data, you'll see that there are two keys for the main dictionary. The ones you already got by iterating over the dict:
{u'query': {...}, u'warnings': {...}}