I'm getting facedetection data from an API in this form:
{"id":1,"ageMin":0,"ageMax":100,"faceConfidence":66.72220611572266,"emotion":"ANGRY","emotionConfidence":50.0'
b'2540969848633,"eyeglasses":false,"eyeglassesConfidence":50.38102722167969,"eyesOpen":true,"eyesOpenConfidence":50.20328140258789'
b',"gender":"Male","genderConfidence":50.462989807128906,"smile":false,"smileConfidence":50.15522384643555,"sunglasses":false,"sun'
b'glassesConfidence":50.446510314941406}]'
I'd like to save this to a csv-file like this:
id ageMin ageMax faceConfidence
1 0 100 66
... and so on.
I tried to do it this way:
response = requests.get(url, headers=headers)
with open('detections.csv', 'w') as f:
writer = csv.writer(f)
for item in response:
writer.writerow(str(item))
That puts every char in its own cell. I've also tried to use item.id, but that gives an error: AttributeError: 'bytes' object has no attribute 'id'.
Could someone point me to the right direction?
Maybe an overkill for a small task, but you can do the following:
convert JSON response (do not forget to check exceptions, etc.) to python dictionary
dic = response.json()
Create a dataframe, for example using pandas:
df = pandas.DataFrame(dic)
Save to csv omitting index:
df.to_csv('detections.csv', index=False, sep="\t")
You can do this relatively easily with the pandas and json libraries.
import pandas as pd
import json
response = """{
"id": 1,
"ageMin": 0,
"ageMax": 100,
"faceConfidence": 66.72220611572266,
"emotion": "ANGRY",
"emotionConfidence": 50.0,
"eyeglasses": false,
"eyeglassesConfidence": 50.38102722167969,
"eyesOpen": true,
"eyesOpenConfidence": 50.20328140258789,
"gender": "Male",
"genderConfidence": 50.462989807128906,
"smile": false,
"smileConfidence": 50.15522384643555,
"sunglasses": false,
"glassesConfidence":50.446510314941406
}"""
file = json.loads(doc)
json = pd.DataFrame({"data": file})
json.to_csv("response.csv")
This is the response formatted to csv.
,data
ageMax,100
ageMin,0
emotion,ANGRY
emotionConfidence,50.0
eyeglasses,False
eyeglassesConfidence,50.38102722167969
eyesOpen,True
eyesOpenConfidence,50.20328140258789
faceConfidence,66.72220611572266
gender,Male
genderConfidence,50.462989807128906
glassesConfidence,50.446510314941406
id,1
smile,False
smileConfidence,50.15522384643555
sunglasses,False
Related
I have a JSON file and I need to convert that into CSV. But my JSON file contains JSON object which is an array and my all attributes are in that array but the code I am trying converts the first object into a single value but in actual I want all those attributes from JSON object.
JSON file content
{
"leads": [
{
"id": "31Y2V29CH0X82",
"product_type": "prelist"
},
{
"id": "2N649TAJBA50Z",
"product_type": "prelist"
}
],
"has_next_page": true,
"next_cursor": "2022-07-27T20:02:13.856000-07:00"
}
Python code
import pandas as pd
df = pd.read_json (r'C:\Users\Ron\Desktop\Test\Product_List.json')
df.to_csv (r'C:\Users\Ron\Desktop\Test\New_Products.csv', index = None)
The output I am getting is as following
And the output I want
I want the attributes as CSV content with headers?
I think you'll have to do this row by row.
data = {"leads": [{"id": "31Y2V29CH0X82", "product_type": "prelist"}, {"id": "2N649TAJBA50Z", "product_type": "prelist"}], "has_next_page": True,
"next_cursor": "2022-07-27T20:02:13.856000-07:00"}
headers = data.copy()
del headers['leads']
rows = []
for row in data['leads']:
row.update( headers )
rows.append( row )
import pandas as pd
df = pd.DataFrame( rows )
print(df)
Output:
id product_type has_next_page next_cursor
0 31Y2V29CH0X82 prelist True 2022-07-27T20:02:13.856000-07:00
1 2N649TAJBA50Z prelist True 2022-07-27T20:02:13.856000-07:00
I am trying to convert dictionary to json, and one of dictionary values is from dataframe.to_json, and I got some strange output as following:
Here is the code
import json
import pandas as pd
my_dict = {}
my_dict["ClassName"] = "First class"
# get student list
df = pd.read_csv("./test.csv")
my_dict["StudentList"] = df.to_json(orient='records')
# output
with open("./output.json", 'w') as fp:
json.dump(my_dict, fp, indent=4)
Here is the input file ./test.csv
Name,Age
Joe,20
Emily,22
John,21
Peter,23
Here is the output file ./output.json
{
"ClassName": "First class",
"StudentList": "[{\"Name\":\"Joe\",\"Age\":20},{\"Name\":\"Emily\",\"Age\":22},{\"Name\":\"John\",\"Age\":21},{\"Name\":\"Peter\",\"Age\":23}]"
}
Here is what I need:
{
"ClassName": "First class",
"StudentList": [{"Name":"Joe","Age":20},{"Name":"Emily","Age":22},{"Name":"John","Age":21},{"Name":"Peter","Age":23}]
}
Thanks for any help.
Use df.to_dict instead of df.to_json:
my_dict["StudentList"] = df.to_dict(orient='records')
to_json just returns a string representation of the JSON, while to_dict returns an actual JSON object.
I had a code, which gave me an empty DataFrame with no saved tweets.
I tried to debug it by putting print(line) under the for line in json file: and json_data = json.loads(line).
That resulted a KeyError.
How do I fix it?
Thank you.
list_df = list()
# read the .txt file, line by line, and append the json data in each line to the list
with open('tweet_json.txt', 'r') as json_file:
for line in json_file:
print(line)
json_data = json.loads(line)
print(line)
tweet_id = json_data['tweet_id']
fvrt_count = json_data['favorite_count']
rtwt_count = json_data['retweet_count']
list_df.append({'tweet_id': tweet_id,
'favorite_count': fvrt_count,
'retweet_count': rtwt_count})
# create a pandas DataFrame using the list
df = pd.DataFrame(list_df, columns = ['tweet_id', 'favorite_count', 'retweet_count'])
df.head()
Your comment says you're trying to save to a file, but your code kind of says that you're trying to read from a file. Here are examples of how to do both:
Writing to JSON
import json
import pandas as pd
content = { # This just dummy data, in the form of a dictionary
"tweet1": {
"id": 1,
"msg": "Yay, first!"
},
"tweet2": {
"id": 2,
"msg": "I'm always second :("
}
}
# Write it to a file called "tweet_json.txt" in JSON
with open("tweet_json.txt", "w") as json_file:
json.dump(content, json_file, indent=4) # indent=4 is optional, it makes it easier to read
Note the w (as in write) in open("tweet_json.txt", "w"). You're using r (as in read), which doesn't give you permission to write anything. Also note the use of json.dump() rather than json.load(). We then get a file that looks like this:
$ cat tweet_json.txt
{
"tweet1": {
"id": 1,
"msg": "Yay, first!"
},
"tweet2": {
"id": 2,
"msg": "I'm always second :("
}
}
Reading from JSON
Let's read the file that we just wrote, using pandas read_json():
import pandas as pd
df = pd.read_json("tweet_json.txt")
print(df)
Output looks like this:
>>> df
tweet1 tweet2
id 1 2
msg Yay, first! I'm always second :(
I have below json which I've got from some url:
{
"abc":
{
"123":[45600,null,3567],
"378":[78689,2345,5678],
"343":[23456,null,null]
}
}
it is stored in json_obj object:
json_obj = response.json()
I need to convert this json into dataframe, My code should be something like this :
df = pd.read_json(response,orient='columns')
so, that result should be :
abc
123 [45600,null,3567]
378 [78689,2345,5678]
343 [23456,null,null]
But with above code I get error :
date_unit).parse()
self._parse_no_numpy()
loads(json, precise_float=self.precise_float), dtype=None)
TypeError: Expected String or Unicode
If I replace response with url in the above code. it will work fine. But, I need to pass json_object instead of url.
Please give suggestions.
df = pd.read_json(response,orient='columns')
read_json() takes in JSON data. "response", i believe stores the API response for some request you're making.
response.json() would give you the python dictionary of the body of the response.
You need to convert that to JSON.
Try this:
import json
df = pd.read_json(json.dumps(response.json()),orient='columns')
I have a JSON file that looks like this:
[
"{'_filled': False,\n 'affiliation': u'Postdoctoral Scholar, University of California, Berkeley',\n 'citedby': 113,\n 'email': u'#berkeley.edu',\n 'id': u'4bahYMkAAAAJ',\n 'interests': [u'3D Shape',\n u'Shape from Texture',\n u'Shape from Shading',\n u'Naive Physics',\n u'Haptics'],\n 'name': u'Steven A. Cholewiak',\n 'url_citations': u'/citations?user=4bahYMkAAAAJ&hl=en',\n 'url_picture': u'/citations?view_op=view_photo&user=4bahYMkAAAAJ&citpid=1'}",
"\n"]
I am using python to extract the value of citedby. However, I am not able to figure.
Here is my code:
import json
json_data = open("output.json")
data = json.load(json_data)
print data[]
Now I know data would take an integer value whereas I would want to have it as a dictionary where in I could search using the key.
Is there any way I can achieve this?
import json
import ast
json_data = open("output.json")
data = json.load(json_data)
print ast.literal_eval(data[0])['citedby']