I hope to use the dictionary load by json file. However, each item contains the character 'u'. I need to remove the 'u's.
I tried dumps, but it does not work.
import ast
import json
data= {u'dot',
u'dog',
u'fog',
u'eeee'}
res = eval(json.dumps(data))
print res
I hope to get: {
'dot',
'dog',
'fog,
'eeee'
}
But the error is:
TypeError: set([u'eeee', u'fog', u'dog', u'dot']) is not JSON serializable
The strings that start with u are unicode strings.
In your case, this has nothing to do with the problem:
data= {u'dot',
u'dog',
u'fog',
u'eeee'}
This creates a set and stores the results in the data variable. The json serializer can't handle sets since the json spec makes no mention of them. If you change this to be a list, the serializer can handle the data:
res = set(eval(json.dumps(list(data))))
Here I'm converting the data variable to a list to serialize it, then converting it back to a set to store the result in the res variable.
Alternatively, you can directly ask Python to convert the unicode strings to strings, using something like this:
res = {x.encode("utf-8") for x in data}
print(res)
Related
I am using python 3.10.4. I have a string which is as follows:
str_obj = "[(190229461780583, 'main'), (302030571463836, 'feature/BRANCH_livestream'), (1071064128966159, 'feature/BRANCH_sensitive'), (1137211553786277, 'main'), (1283366887974580, 'feature/add_sql_vm'), (1492751472739439, 'feature/BRANCH-2977'), (2662272892040840, 'main'), (4078787696930326, 'main')]"
I need to convert it to a json object. I need it in json format for further data extraction. Following this link I used json.loads() to convert it from string to json:
json_obj = json.loads(str_obj)
However I am getting the error:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I checked this link but it is not really related to my problem.
I cannot simply drop "" from str_obj as this is the value I am receiving from an environment variable sent to me from a DevOps pipeline.
I would like to know what the root cause problem is and how I can resolve it.
Probably something like this would help you if you wanted to make json out of this string.
Surely the best option would be to fix the input data
str_obj = "[(190229461780583, 'main'), (302030571463836, 'feature/BRANCH_livestream'), (1071064128966159, 'feature/BRANCH_sensitive'), (1137211553786277, 'main'), (1283366887974580, 'feature/add_sql_vm'), (1492751472739439, 'feature/BRANCH-2977'), (2662272892040840, 'main'), (4078787696930326, 'main')]"
# clear the input string
str_obj = str_obj.replace('[','').replace(']','').replace('(','').replace(')','').replace('\'','').replace(' ','').split(',')
# create output string
output = {}
# Create pairs of key:value
str_to_list = [str_obj[i:i+2] for i in range(0, len(str_obj), 2)]
# append to output dict
for e in str_to_list:
output[e[0]] = e[1]
# You can easly create json form dict
json_object = json.dumps(output)
That isn't json, json uses key value pairs. Did you want ast.literal_eval?
import ast
ast.literal_eval(str_obj)
# [(190229461780583, 'main'), (302030571463836, 'feature/BRANCH_livestream'), (1071064128966159, 'feature/BRANCH_sensitive'), (1137211553786277, 'main'), (1283366887974580, 'feature/add_sql_vm'), (1492751472739439, 'feature/BRANCH-2977'), (2662272892040840, 'main'), (4078787696930326, 'main')]
The value of your str_obj is simply not a valid json format at all. You can use a JSON parser online to verify the syntax you need to turn into a valid JSON : Online JSON parser
I have pulled JSON data from a url. The result is a dictionary. How can I transform this dictionary so metric is a column, and the time is the index for each value
Thanks in advance
time------------------------AdrActCnt-----BlkCnt------BlkSizeByte
2021-01-28T00:00:00.000Z----1097896.0-----145.0-------190568423.0
2021-01-29T00:00:00.000Z----1208741.0-----152.0-------199725189.0
2021-01-29T00:00:00.000Z----1087755.0-----136.0-------177349536.0
Output:
{"metricData":{"metrics":["AdrActCnt","BlkCnt","BlkSizeByte"],"series":
[{"time":"2021-01-28T00:00:00Z","values"["1097896.0","145.0","190568423.0"]},
{"time":"2021-01-29T00:00:00Z","values":["1208741.0","152.0","199725189.0"]},
{"time":"2021-01-30T00:00:00Z","values":["1087755.0","136.0","177349536.0"]}
You may be looking for a dict comprehension, which is similar to a list comprehension, just creates a dictionary at the end:
liststuff = [{"time":"2021-01-28T00:00:00.000Z","values":["1097896.0","145.0","190568423.0"]},{"time":"2021-01-29T00:00:00.000Z","values":["1208741.0","152.0","199725189.0"]},{"time":"2021-01-30T00:00:00.000Z","values":["1087755.0","136.0","177349536.0"]}]
dictstuff = {item['time']:item['values'] for item in liststuff}
print(dictstuff)
{'2021-01-28T00:00:00.000Z': ['1097896.0', '145.0', '190568423.0'], '2021-01-29T00:00:00.000Z': ['1208741.0', '152.0', '199725189.0'], '2021-01-30T00:00:00.000Z': ['1087755.0', '136.0', '177349536.0']}
liststuff is your data, just needed [] wrapping (I assume that's a typo in the question, it's not valid JSON without the brackets). If you need help with parsing the string, use json.loads() (from the json module) to make it actual Python data:
import json
jsonstuff = '[{"time":"2021-01-28T00:00:00.000Z","values":["1097896.0","145.0","190568423.0"]},{"time":"2021-01-29T00:00:00.000Z","values":["1208741.0","152.0","199725189.0"]},{"time":"2021-01-30T00:00:00.000Z","values":["1087755.0","136.0","177349536.0"]}]'
liststuff = json.loads(jsonstuff)
(here jsonstuff is the string you've downloaded)
the dataframe 'dataset' is automatically generated by PowerBI here is the result of my dataset.head(10).to_clipboard(sep=',', index=False)
coordinates,status
"[143.4865219,-34.7560602]",not started
"[143.4865241,-34.7561332]",not started
"[143.4865264,-34.7562088]",not started
"[143.4865286,-34.7562818]",not started
"[143.4865305,-34.7563453]",not started
"[143.4865327,-34.7564183]",not started
"[143.486535,-34.756494]",not started
"[143.4865371,-34.756567]",not started
"[143.486539,-34.7566304]",not started
"[143.4865412,-34.7567034]",not started
then to get the json
i do this data=dataset.to_json(orient='records')
which give me this results
[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"},
how i get this instead , no quotes on the coordinates values
[{"coordinates":[143.4865219,-34.7560602],"status":"not started"},{"coordinates":[143.4865241,-34.7561332],"status":"not started"},
edit
print(type(data))
<class 'str'>
You could use ast.literal_eval:
Safely evaluate an expression node or a string containing a Python
literal or container display. The string or node provided may only
consist of the following Python literal structures: strings, bytes,
numbers, tuples, lists, dicts, sets, booleans, and None.
This can be used for safely evaluating strings containing Python
values from untrusted sources without the need to parse the values
oneself.[...]
Your data seems to be a string, and not a list as Python would print it (it uses single quotes by default, the double quotes in your data seem to indicate that it is a string, ready to be saved in a json file for example). So, you have to convert it first to a Python object with json.loads:
from ast import literal_eval
import json
data = """[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"}]"""
data = json.loads(data)
for d in data:
d['coordinates'] = literal_eval(d['coordinates'])
print(data)
# [{'coordinates': [143.4865219, -34.7560602], 'status': 'not started'}, {'coordinates': [143.4865241, -34.7561332], 'status': 'not started'}]
import json
s = '[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"}]'
d = json.loads(s)
d[0]['coordinates'] = json.loads(d[0]['coordinates'])
Applying this concept to every value can be done as in
for dic in d:
for key, value in dic.items():
try:
temp = json.loads(value)
if isinstance(temp, list):
dic[key] = temp
except Exception:
pass
or if you are sure there will be a coordinates key in ever dictionary
and that key having a "list" value
for dic in d: dic['coordinates'] = json.loads(dic['coordinates'])
simply u can use eval function.
new =[]
l = '[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"}]'
l=eval(l)
for each_element in l:
temp={}
for k,v in each_element.items():
if k =='coordinates' :
temp[k]=eval(v)
else:
temp[k]=v
new.append(temp)
print(temp)
I'm working with flask. I have a flask function that receives some posted json and looks like:
#app.route('/index',methods=['POST'])
def index():
import json
json = request.get_json(force=True) # receives request from php
for j in json:
print str(j)
The printed json looks like:
{u'token': u'146bf00b2cb96e6c425c2ab997637', u'a': u'aaa'}
{u'token': u'146bf00b2cb96e6c425c2ab3f7417', u'a': u'bbb'}
{u'token': u'146bf00b2cb96e6c425c2ab3f5692', u'a': u'ccc'}
The number of json records that come in are variable. I need to load the values into a list of dictionaries that looks like:
[{u'token': u'146bf00b2cb96e6c425c2ab997637', u'a': u'aaa'},{u'token': u'146bf00b2cb96e6c425c2ab3f7417', u'a': u'bbb'} ....]
I've tried:
test = [{'a': j[i]['a'], 'token':j[i]['token']} for i in len(json)-1]
However, this is giving me the error listed above. What am I doing wrong?
You don't need to produce a list object because you already have that list. the json variable references it, which is why your for j in json: loop works in the first place.
Your error stems from trying to loop over an integer len(json) - 1:
test = [{'a': j[i]['ad'], 'token':j[i]['token']} for i in len(json)-1]
# ^^^^^^^^^^^
Perhaps you wanted to use range(len(json)) there? Not that there is a need to produce integers here, you could just loop over the json list again to extract what you want:
test = [{'a': entry['a'], 'token': entry['token']} for entry in json]
but this just creates copies of the already existing dictionaries, unless they have more keys you didn't include in your question.
Edit: Turns out my whole problem was my Json had already been converted to a dictionary previously, without me realising. And I was using json.dumps() instead of .loads() by mistake.
Original Q:I have a piece of Json code I have turned into a dictionary. I am retrieving this Json from a server. When I try access the inner values of the new dictionary to change a value, I get a:
TypeError: string indices must be integers, not str
This is my code:
corner_data = json.dumps(r.json())
print corner_data
corner_data["geometry"]["corners"] = '{"bottom_right": {"y": "575.531616", "x": "690.547363"}, "top_left": {"y": "-146.739075", "x": "-109.105957"}}'
Turn the JSON string into a Python dict using json.loads (not dumps!):*
corner_data = json.loads(r.json())
Modify the data, assigning a Python dict (not a string!):
corner_data['geometry']['corners'] = {'bottom_right': {'y': '575.531616', 'x': '690.547363'}, 'top_left': {'y': '-146.739075', 'x': '-109.105957'}}
Encode it all back into JSON:
print json.dumps(corner_data)
* I'm assuming here that r.json() returns a string. It's entirely possible that r.json() already decodes a JSON string and returns a dict, that depends on what r is. If that's the case, you don't need json.loads here, you can simply omit it.