We Keep Coding

Reading a text file of dictionaries stored in one line

Question
I have a text file that records metadata of research papers requested with SemanticScholar API. However, when I wrote requested data, I forgot to add "\n" for each individual record. This results in something looks like
{<metadata1>}{<metadata2>}{<metadata3>}...
and this should be if I did add "\n".
{<metadata1>}
{<metadata2>}
{<metadata3>}
...
Now, I would like to read the data. As all the metadata is now stored in one line, I need to do some hacks
First I split the cluttered dicts using "{".
Then I tried to convert the string line back to dict. Note that I do consider line might not be in a proper JSON format.
import json
with open("metadata.json", "r") as f:
for line in f.readline().split("{"):
print(json.loads("{" + line.replace("\'", "\"")))
However, there is still an error message
JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
I am wondering what should I do to recover all the metadata I collected?
MWE
Note, in order to get metadata.json file I use, use the following code, it should work out of the box.
import json
import urllib
import requests
baseURL = "https://api.semanticscholar.org/v1/paper/"
paperIDList = ["200794f9b353c1fe3b45c6b57e8ad954944b1e69",
"b407a81019650fe8b0acf7e4f8f18451f9c803d5",
"ff118a6a74d1e522f147a9aaf0df5877fd66e377"]
for paperID in paperIDList:
response = requests.get(urllib.parse.urljoin(baseURL, paperID))
metadata = response.json()
record = dict()
record["title"] = metadata["title"]
record["abstract"] = metadata["abstract"]
record["paperId"] = metadata["paperId"]
record["year"] = metadata["year"]
record["citations"] = [item["paperId"] for item in metadata["citations"] if item["paperId"]]
record["references"] = [item["paperId"] for item in metadata["references"] if item["paperId"]]
with open("metadata.json", "a") as fileObject:
fileObject.write(json.dumps(record))

The problem is that when you do the split("{") you get a first item that is empty, corresponding to the opening {. Just ignore the first element and everything works fine (I added an r in your quote replacements so python considers then as strings literals and replace them properly):
with open("metadata.json", "r") as f:
for line in f.readline().split("{")[1:]:
print(json.loads("{" + line).replace(r"\'", r"\""))
As suggested in the comments, I would actually recommend recreating the file or saving a new version where you replace }{ by }\n{:
with open("metadata.json", "r") as f:
data = f.read()
data_lines = data.replace("}{","}\n{")
with open("metadata_mod.json", "w") as f:
f.write(data_lines)
That way you will have the metadata of a paper per line as you want.

How to read a json file, find a key and bring the related data as object on Python?

I'm kinda new to Python and hence learning by doing, so here I am asking for some heads up.
I need to store values in a JSON file, and later in the next execution check first if the user already requested access into the system to provide the proper returns.
So basically my json file will be:
{instance_id : 1111 {'username':'Dummy_User','email':'dummy#user.com','domain':'local','role':'admin','instance':'production','group':'administrators','state':'failed','description':'User Dummy_User is already a member of group administrators'}
},
{instance_id : 2222 {'username':'Dummy_User1','email':'dummy#user.com','domain':'local','role':'admin','instance':'production','group':'administrators','state':'success','description':'User Dummy_User1 added to the administrators group'}
}
I need to be able to query it by instance_id and get as return all of the fields, or a particular one.
Example:
result = checkJson(1111)
print (result.state + " - " + result.description)
>> failed - User Dummy_User is already a member of group administrators
What I'm currently doing: I'm writing the file using "with open" funcion but it is not separating the "objects" by comma in the file, hence it is not being recognizable as a valid json later
def write_json(self, data, filename):
with open(filename,'r+') as f:
json_data = json.load(f)
json_data.update(data)
f.seek(0)
json.dump(json_data, f)
Any thoughts on the best way to do that?
Thank you in advance!

I figured. As per my example, I could get it done by:
Writing in the json:
userInput = {id : {"username":username,"adgroup":adgroup}}
filename = "path\to\file.json"
with open(filename,"r+") as f:
json_data = json.load(f)
json_data.update(data)
f.seek(0)
json.dump(json_data, f)
This will create an object of userInput and add to the json within the agregate structure.
Later, to read this json I used:
jsonFile = "path\to\file.json"
with open(jsonFile) as access:
data = json.load(access)
instance = data[id]['username']
name = data[id]['adgroup']
where ID is the ID stored in the JSON to identify that object you saved before.

How to read data from a CSV into a tuple in Python? [duplicate]

This question already has answers here:
Reading a CSV file using Python
(2 answers)
Closed 3 years ago.
I'm attempting to load a CSV which looks like the following and create a tuple out of it, so that I can later access account1, account2, etc in code:
user,password
johndoe#gmail.com,password
janesmith#gmail.com,password2
So far, all I've been able to do is import the CSV.
import csv
# our csv file name
filename = "resources/accounts.csv"
with open(filename) as infile:
reader = csv.reader(infile)
I'm looking for something like so:
{("johndoe#gmail.com", "password"), ("janesmith#gmail.com", "password")}
Could someone advise on what to do from here?

import csv # our csv file name
filename = "resources/accounts.csv"
with open(filename) as infile:
reader = csv.reader(infile)
accounts = {"accounts" : []}
for line in reader:
accounts["accounts"].append({"user":line[0], "pass": line[1]})
print(accounts["accounts"][1]["user"])
Sorry I did this on mobile hope it shows up

Reading a Json in python [duplicate]

This question already has answers here:
Reading JSON from a file [duplicate]
(7 answers)
Closed 4 years ago.
I need to get a particular key:value from a json file.
Code used to open json file :
import json
data = repr(open('file.json', 'rb').read())
print(data)
Output( Json file) :
b'b\'{"image":{"width":750,"height":1127,"orientation":1},"objects":[{"type":"face","boundingBox":{"x":132,"y":180,"height":619,"width":513},"landmarks":{"faceContour":[[162,375],[162,440],[165,504],[175,568],[193,628],[226,686],[265,740],[313,784],[373,799],[436,793],[494,758],[543,712],[586,660],[616,601],[631,534],[639,468],[645,400]],"noseBridge":[[387,396],[385,441],[381,485],[378,532]],"noseBall":[[322,547],[350,560],[380,572],[412,563],[442,554]],"eyebrowRight":[[419,348],[466,329],[516,325],[564,339],[595,377]],"eyebrowLeft":[[197,358],[224,325],[269,315],[316,325],[358,347]],"eyeRight":[[448,419],[479,404],[510,405],[539,416],[511,425],[480,424]],"eyeRightCenter":[[495,416]],"eyeLeft":[[242,402],[269,393],[299,396],[329,412],[297,414],[267,413]],"eyeLeftCenter":[[284,405]],"mouthOuter":[[269,596],[310,598],[349,598],[379,606],[411,601],[456,606],[502,609],[456,663],[412,684],[378,687],[345,680],[307,652]],"mouthInner":[[284,603],[349,610],[379,615],[411,613],[487,614],[412,652],[379,656],[348,648]]},"attributes":{"gender":"male","genderConfidence":0.8385,"age":24,"ageConfidence":0.8419,"emotion":"happiness","emotionConfidence":1.0,"emotionsAll":{"neutral":0.0,"sadness":0.0,"disgust":0.0,"anger":0.0,"surprise":0.0,"fear":0.0,"happiness":1.0},"pose":{"pitch":-5.7105,"roll":0.2941,"yaw":1.8646},"race":{"asian":0.0002,"black":0.0001,"white":0.9989},"eyewear":{"sunglasses":0.0,"glasses":0.0},"hair":{"color":{"blond":0.0,"black":0.9994,"brown":0.0},"cut":{"short":0.9999,"long":0.0,"bald":0.0},"facial":{"beard":0.0,"mustache":0.0,"stubble":0.2543}},"frontal":true}},{"type":"person","boundingBox":{"x":20,"y":60,"height":1053,"width":696}}],"requestId":"1239be0c782440bd828cee21dbc2baa1"}\'\r\n'
From this Json file, I need to extract a particular key:value. For example, I need to get the emotionsAll (Key) from the file.
I tried using :
print(data['emotionsAll'][0]['neutral'])
error i got :
Traceback (most recent call last): File "C:\Users\HP\Downloads\Final Project\getfile.py", line 6, in <module>
print(data['emotionsAll'][0]['neutral']) TypeError: string indices must be integers
So, Is there any other way to get a particular key:value ?

Try using the json package
Contents of file.json
{"image":{"width":750,"height":1127,"orientation":1},"objects":[{"type":"face","boundingBox":{"x":132,"y":180,"height":619,"width":513},"landmarks":{"faceContour":[[162,375],[162,440],[165,504],[175,568],[193,628],[226,686],[265,740],[313,784],[373,799],[436,793],[494,758],[543,712],[586,660],[616,601],[631,534],[639,468],[645,400]],"noseBridge":[[387,396],[385,441],[381,485],[378,532]],"noseBall":[[322,547],[350,560],[380,572],[412,563],[442,554]],"eyebrowRight":[[419,348],[466,329],[516,325],[564,339],[595,377]],"eyebrowLeft":[[197,358],[224,325],[269,315],[316,325],[358,347]],"eyeRight":[[448,419],[479,404],[510,405],[539,416],[511,425],[480,424]],"eyeRightCenter":[[495,416]],"eyeLeft":[[242,402],[269,393],[299,396],[329,412],[297,414],[267,413]],"eyeLeftCenter":[[284,405]],"mouthOuter":[[269,596],[310,598],[349,598],[379,606],[411,601],[456,606],[502,609],[456,663],[412,684],[378,687],[345,680],[307,652]],"mouthInner":[[284,603],[349,610],[379,615],[411,613],[487,614],[412,652],[379,656],[348,648]]},"attributes":{"gender":"male","genderConfidence":0.8385,"age":24,"ageConfidence":0.8419,"emotion":"happiness","emotionConfidence":1.0,"emotionsAll":{"neutral":0.0,"sadness":0.0,"disgust":0.0,"anger":0.0,"surprise":0.0,"fear":0.0,"happiness":1.0},"pose":{"pitch":-5.7105,"roll":0.2941,"yaw":1.8646},"race":{"asian":0.0002,"black":0.0001,"white":0.9989},"eyewear":{"sunglasses":0.0,"glasses":0.0},"hair":{"color":{"blond":0.0,"black":0.9994,"brown":0.0},"cut":{"short":0.9999,"long":0.0,"bald":0.0},"facial":{"beard":0.0,"mustache":0.0,"stubble":0.2543}},"frontal":true}},{"type":"person","boundingBox":{"x":20,"y":60,"height":1053,"width":696}}],"requestId":"1239be0c782440bd828cee21dbc2baa1"}
Code:
import json
with open('file.json', 'rb') as fp:
json_data = json.load(fp)
print(json_data['objects'][0]['attributes']['emotionsAll']['neutral'])

A couple things:
1 - the emotionsAll key is within the objects key, first element in the list [0], attributes key
2 - Your json file was written with the bytes prefix, so when it's being read, it starts your string with b'. You can either a) have that file written without that mode by decoding/encoding, or just manipulate that string.
import json
data = repr(open('file.json', 'rb').read())
data = data.split('{', 1)[-1]
data = data.rsplit('}', 1)[0]
data = ''.join(['{', data, '}'])
jsonObj = json.loads(data)
print(jsonObj['objects'][0]['attributes']['emotionsAll']['neutral'])
Output:
print(jsonObj['objects'][0]['attributes']['emotionsAll']['neutral'])
0.0

How to read and write dictionaries to external files in python? [duplicate]

This question already has answers here:
Why can't Python parse this JSON data? [closed]
(3 answers)
Closed 5 years ago.
I have a dictionary in python. I want to modify that dictionary and then save the dictionary to an external file so that when I load up the python program again it grabs the dictionary data from the external file.
class Data:
"""
Data handling class to save
and receive json data, parent
of User for data purposes.
"""
def saveData(data, file):
with open(file, 'r+') as dataFile:
dataFile.write(json.dumps(data))
def getData(file):
with open(file, 'r+') as dataFile:
return json.loads(dataFile.readline())
def deleteContent(file):
file.seek(0)
file.truncate()
But when I write to the file and then try to read it reads it as a string and I can't use the read data to set a dictionary. How can I get data in a dictionary from an external JSON file as dictionary data, not string data?
data = Data.getData("chatbotData.json")
dataDict = data
dataDict["age"] = 2
Here is what I want to do with the data and I get this error:
TypeError: 'str' object does not support item assignment

Let's create a dictionary:
>>> d = {'guitar':'Jerry', 'drums':'Mickey' }
Now, let's dump it to a file:
>>> import json
>>> json.dump(d, open('1.json', 'w'))
Now, let's read it back in:
>>> json.load(open('1.json', 'r'))
{'guitar': 'Jerry', 'drums': 'Mickey'}
Taking better care of file handles
The above illustrates the json module but was sloppy about closing files. Better:
>>> with open('1.json', 'w') as f:
... json.dump(d, f)
...
>>> with open('1.json') as f:
... json.load(f)
...
{'guitar': 'Jerry', 'drums': 'Mickey'}