We Keep Coding

JSON String conversion to Dict

I'm having trouble converting the below JSON string to type dict. json.loads throws an error because of the two sets of squiggles: {}, {}
test = '{"124074": "0.0944", "124111": "0.0809", "124194": "0.0788"}, {"128213": "0.39", "129043": "0.458", "129054": "0.378"}'
Any thoughts on how I fix this? I need to be able to convert to dict so I can iterate through the keys. Should the JSON string be represented in a different way?

Based on the comment from #rv.kvetch, one can get the following result
>>> test = '{"124074": "0.0944", "124111": "0.0809", "124194": "0.0788"}, {"128213": "0.39", "129043": "0.458", "129054": "0.378"}'
>>> dicts = json.loads(f'[{test}]')
>>> dicts
[{'124074': '0.0944', '124111': '0.0809', '124194': '0.0788'}, {'128213': '0.39', '129043': '0.458', '129054': '0.378'}]
>>> dicts[0]
{'124074': '0.0944', '124111': '0.0809', '124194': '0.0788'}
>>> type(dicts[0])
<class 'dict'>
essentially a list of dicts is created instead of a single dict.

remove quotes from a json file using python

the dataframe 'dataset' is automatically generated by PowerBI here is the result of my dataset.head(10).to_clipboard(sep=',', index=False)
coordinates,status
"[143.4865219,-34.7560602]",not started
"[143.4865241,-34.7561332]",not started
"[143.4865264,-34.7562088]",not started
"[143.4865286,-34.7562818]",not started
"[143.4865305,-34.7563453]",not started
"[143.4865327,-34.7564183]",not started
"[143.486535,-34.756494]",not started
"[143.4865371,-34.756567]",not started
"[143.486539,-34.7566304]",not started
"[143.4865412,-34.7567034]",not started
then to get the json
i do this data=dataset.to_json(orient='records')
which give me this results
[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"},
how i get this instead , no quotes on the coordinates values
[{"coordinates":[143.4865219,-34.7560602],"status":"not started"},{"coordinates":[143.4865241,-34.7561332],"status":"not started"},
edit
print(type(data))
<class 'str'>

You could use ast.literal_eval:
Safely evaluate an expression node or a string containing a Python
literal or container display. The string or node provided may only
consist of the following Python literal structures: strings, bytes,
numbers, tuples, lists, dicts, sets, booleans, and None.
This can be used for safely evaluating strings containing Python
values from untrusted sources without the need to parse the values
oneself.[...]
Your data seems to be a string, and not a list as Python would print it (it uses single quotes by default, the double quotes in your data seem to indicate that it is a string, ready to be saved in a json file for example). So, you have to convert it first to a Python object with json.loads:
from ast import literal_eval
import json
data = """[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"}]"""
data = json.loads(data)
for d in data:
d['coordinates'] = literal_eval(d['coordinates'])
print(data)
# [{'coordinates': [143.4865219, -34.7560602], 'status': 'not started'}, {'coordinates': [143.4865241, -34.7561332], 'status': 'not started'}]

import json
s = '[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"}]'
d = json.loads(s)
d[0]['coordinates'] = json.loads(d[0]['coordinates'])
Applying this concept to every value can be done as in
for dic in d:
for key, value in dic.items():
try:
temp = json.loads(value)
if isinstance(temp, list):
dic[key] = temp
except Exception:
pass
or if you are sure there will be a coordinates key in ever dictionary
and that key having a "list" value
for dic in d: dic['coordinates'] = json.loads(dic['coordinates'])

simply u can use eval function.
new =[]
l = '[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"}]'
l=eval(l)
for each_element in l:
temp={}
for k,v in each_element.items():
if k =='coordinates' :
temp[k]=eval(v)
else:
temp[k]=v
new.append(temp)
print(temp)

Converting a dataframe into JSON (in pyspark) and then selecting desired fields

I'm new to Spark. I have a dataframe that contains the results of some analysis. I converted that dataframe into JSON so I could display it in a Flask App:
results = result.toJSON().collect()
An example entry in my json file is below. I then tried to run a for loop in order to get specific results:
{"userId":"1","systemId":"30","title":"interest"}
for i in results:
print i["userId"]
This doesn't work at all and I get errors such as: Python (json) : TypeError: expected string or buffer
I used json.dumps and json.loads and still nothing - I keep on getting errors such as string indices must be integers, as well as the above error.
I then tried this:
print i[0]
This gave me the first character in the json "{" instead of the first line. I don't really know what to do, can anyone tell me where I'm going wrong?
Many Thanks.

If the result of result.toJSON().collect() is a JSON encoded string, then you would use json.loads() to convert it to a dict. The issue you're running into is that when you iterate a dict with a for loop, you're given the keys of the dict. In your for loop, you're treating the key as if it's a dict, when in fact it is just a string. Try this:
# toJSON() turns each row of the DataFrame into a JSON string
# calling first() on the result will fetch the first row.
results = json.loads(result.toJSON().first())
for key in results:
print results[key]
# To decode the entire DataFrame iterate over the result
# of toJSON()
def print_rows(row):
data = json.loads(row)
for key in data:
print "{key}:{value}".format(key=key, value=data[key])
results = result.toJSON()
results.foreach(print_rows)
EDIT: The issue is that collect returns a list, not a dict. I've updated the code. Always read the docs.
collect() Return a list that contains all of the elements in this RDD.
Note This method should only be used if the resulting array is
expected to be small, as all the data is loaded into the driver’s
memory.
EDIT2: I can't emphasize enough, always read the docs.
EDIT3: Look here.

import json
>>> df = sqlContext.read.table("n1")
>>> df.show()
+-----+-------+----+---------------+-------+----+
| c1| c2| c3| c4| c5| c6|
+-----+-------+----+---------------+-------+----+
|00001|Content| 1|Content-article| |2018|
|00002|Content|null|Content-article|Content|2015|
+-----+-------+----+---------------+-------+----+
>>> results = df.toJSON().map(lambda j: json.loads(j)).collect()
>>> for i in results: print i["c1"], i["c6"]
...
00001 2018
00002 2015

Here is what worked for me:
df_json = df.toJSON()
for row in df_json.collect():
#json string
print(row)
#json object
line = json.loads(row)
print(line[some_key])
Keep in mind that using .collect() is not advisable, since it collects the distributed data frames, and defeats the purpose of using data frames.

To get an array of python dicts:
results = df.toJSON().map(json.loads).collect()
To get an array of JSON strings:
results = df.toJSON().collect()
To get a JSON string (i.e. a JSON string of an array):
results = df.toPandas().to_json(orient='records')
and using that to get an array of Python dicts:
results = json.loads(df.toPandas().to_json(orient='records'))

how to create a dictionary from a set of properly formatted tuples in python

Is there a simple way to create a dictionary from a list of formatted tuples. e.g. if I do something like:
d={"responseStatus":"SUCCESS","sessionId":"01234","userId":2000004904}
This creates a dictionary called d. However, if I want to create a dictionary from a string which contains the same string, I can't do that
res=<some command that returns {"responseStatus":"SUCCESS","sessionId":"01234","userId":2000004904}>
print res
# returns {"responseStatus":"SUCCESS","sessionId":"01234","userId":2000004904}
d=dict(res)
This throws an error that says:
ValueError: dictionary update sequence element #0 has length 1; 2 is required

I strongly strongly suspect that you have json on your hands.
import json
d = json.loads('{"responseStatus":"SUCCESS","sessionId":"01234","userId":2000004904}')
would give you what you want.

Use dict(zip(tuples))
>>> u = ("foo", "bar")
>>> v = ("blah", "zoop")
>>> d = dict(zip(u, v))
>>> d
{'foo': 'blah', 'bar': 'zoop'}
Note, if you have an odd number of tuples this will not work.

Based on what you gave is, res is
# returns {"responseStatus":"SUCCESS","sessionId":"01234","userId":2000004904}
So the plan is to grab the string starting at the curly brace to the end and use json to decode it:
import json
# Discard the text before the curly brace
res = res[res.index('{'):]
# Turn that text into a dictionary
d = json.loads(res)

All you need to do in your particular case is
d = eval(res)
And please keep security in mind when using eval, especially if you're mixing it with ajax/json.
UPDATE
Since others pointed out you might be getting this data over the web and it isn't just a "how to make this work" question, use this:
import json
json.loads(res)

How can I parse a dictionary string?

I am trying to convert a string to a dictionary with dict function, like this
import json
p = "{'id':'12589456'}"
d = dict(p)
print d['id']
But I get the following error
ValueError: dictionary update sequence element #0 has length 1; 2 is required
Why does it fail? How can I fix this?

What you have is a string, but dict function can only iterate over tuples (key-value pairs) to construct a dictionary. See the examples given in the dict's documentation.
In this particular case, you can use ast.literal_eval to convert the string to the corresponding dict object, like this
>>> p = "{'id':'12589456'}"
>>> from ast import literal_eval
>>> d = literal_eval(p)
>>> d['id']
'12589456'

Since p is a string containing JSON (ish), you have to load it first to get back a Python dictionary. Then you can access items within it:
p = '{"id":"12589456"}'
d = json.loads(p)
print d["id"]
However, note that the value in p is not actually JSON; JSON demands (and the Python json module enforces) that strings are quoted with double-quotes, not single quotes. I've updated it in my example here, but depending on where you got your example from, you might have more to do.