Related
I'm able to load and parse a json file with Python by referring to list items by name. My users.json data file:
{
"joe": {
"secret": "abc123.321def"
},
"sarah": {
"secret": "led789.321plo"
},
"dave": {
"secret": "ghi532.765dlmn"
}
}
My code - to output the 'secret' value associated with a specific user (e.g. Dave):
import json
with open('users_sample.json') as f:
users = json.load(f)
# f.close()
print(users['dave']['secret'])
This outputs Dave's secret:
ghi532.765dlmn
That's easy enough when I can predict or know the user names, but how do I iterate through each user in the users.json file and output each user's 'secret' value?
Thanks in advance!
I would encapsulate the logic to print each user and their associated function into a helper function:
def print_users(users_dict: dict, header='Before'):
print(f'-- {header}')
for u in users_dict:
print(f' {u}: {users_dict[u].get("secret", "<empty>")}')
Then, upon loading the users object initially via json.load, you can then call the function like so:
print_users(users)
To replace the secret for each user, in this case to replace every occurrence of a dot . with a plus +, a simple approach could be to use a for loop to update the users object in place:
for name, user in users.items():
if 'secret' in user:
user['secret'] = user['secret'].replace('.', '+')
Then print the result after the replacements are carried out:
print_users(users, 'After')
Finally, we can write the result users object back out to a file:
with open('users_sample_UPDATED.json', 'w') as out_file:
json.dump(users, out_file)
The output of the above code, in this case would be:
-- Before
joe: abc123.321def
sarah: led789.321plo
dave: ghi532.765dlmn
-- After
joe: abc123+321def
sarah: led789+321plo
dave: ghi532+765dlmn
The full code:
import json
def main():
with open('users_sample.json') as f:
users = json.load(f)
print_users(users)
new_users = {name: {'secret': user['secret'].replace('.', '+')}
for name, user in users.items()}
print_users(new_users, 'After')
with open('users_sample_UPDATED.json', 'w') as out_file:
json.dump(new_users, out_file)
def print_users(users_dict: dict, header='Before'):
print(f'-- {header}')
for u in users_dict:
print(f' {u}: {users_dict[u].get("secret", "<empty>")}')
if __name__ == '__main__':
main()
Iterate the dictionary using a for loop
code that works:
import json
with open('users_sample.json') as f:
users = json.load(f)
for user in users:
print(f"user name: {user} secret: {users[user]['secret']}")
You have a nested dictionary - i.e., each value associated with a top-level key is also a dictionary. You can iterate over those dictionaries with the built-in values() function. This leads to:
print(*[e.get('secret') for e in users.values()], sep='\n')
I am trying to write a new row into a CSV file and I can't because I get an error in Python Shell.
Below is the code I am using (I am reading JSON from API and want to put data into CSV file)
# import urllib library
from urllib.request import Request, urlopen
c=1
# import json
import json
# store the URL in url as
# parameter for urlopen
import pandas as pd
import csv
headerList = ['name','id','order','height','weight','speed','special_defense','special_attack','defense','attack','hp']
# open CSV file and assign header
with open("pokemon_stats.csv", 'w') as file:
dw = csv.DictWriter(file, delimiter=',',
fieldnames=headerList)
dw.writeheader()
# display csv file
fileContent = pd.read_csv("pokemon_stats.csv")
for r in range(1,3):
req = Request('https://pokeapi.co/api/v2/pokemon/'+str(r)+'/', headers={'User-Agent': 'Chrome/32.0.1667.0'})
# store the response of URL
response = urlopen(req)
# storing the JSON response
# from url in data
data_json = json.loads(response.read())
#print(data_json)
for key, value in data_json.items():
if key=='name':
name=value
elif key=='id':
id=value
elif key=='order':
order=value
elif key=='height':
height=value
elif key=='weight':
weight=value
elif key == 'stats':
for sub in data_json['stats']:
for i in sub:
if i=='base_stat':
base_stat=sub[i]
if i=='stat':
for j in sub[i]:
if j=='name':
stat_name=sub[i][j]
if stat_name=='hp':
hp=base_stat
elif stat_name=='attack':
attack=base_stat
elif stat_name=='defense':
defense=base_stat
elif stat_name=='special-attack':
special_attack=base_stat
elif stat_name=='special-defense':
special_defense=base_stat
elif stat_name=='speed':
speed=base_stat
data = [name,id,order,height,weight,speed,special_defense,special_attack,defense,attack,hp]
dw.writerow(data)
After I try the execution of this code I get an error as it follows:
Traceback (most recent call last):
File "C:/Users/sbelcic/Desktop/NANOBIT_API.py", line 117, in <module>
dw.writerow(data)
File "C:\Users\sbelcic\AppData\Local\Programs\Python\Python37\lib\csv.py", line 155, in writerow
return self.writer.writerow(self._dict_to_list(rowdict))
File "C:\Users\sbelcic\AppData\Local\Programs\Python\Python37\lib\csv.py", line 148, in _dict_to_list
wrong_fields = rowdict.keys() - self.fieldnames
AttributeError: 'list' object has no attribute 'keys'*
Can somebody pls help and tell me what I am doing wrong.
I don't have working experience of manipulating JSON response with Python so any comments are welcome. If someone sees a better way to do this he is welcome to share.
Since dw is a DictionaryWriter, data needs to be a dictionary (currently it's a list) as seen in the documentation.
Convert data to a dictionary with your headers
data = [name,id,order,height,weight,speed,special_defense,special_attack,defense,attack,hp]
data = dict(zip(headerList, data))
dw.writerow(data)
Check the example for using the DictWriter. You need to pass a dictionary to writerow instead of a list, so your last line should be
data =['name':name,'id': id,'order':order,'height': height,'weight':weight,'speed':speed,'special_defense':special_defense,'special_attack':special_attack,'defense':defense,'attack':attack,'hp':hp]
dw.writerow(data)
Note that your whole code can also be simplified if you populate the data dictionary instead of all your if/else:
data={} #empty dictionary
#First extract everything that is on the main level of your dict
for key in ("name", "id", "order", "height", "weight":
if key in data_json:
data[key]=data_json[key]
#Check if the "stats" dict exists in your JSON data
if 'stats' in data_json:
if 'base_stat' in data_json['stats']:
data['base_stat']=data_json['stats']['base_stat']
if 'stat' in data_json['stats']:
statDict = data_json['stats']['stat']
for key in ['hp', 'attack', 'defense', 'special-attack', 'special-defense', 'speed']:
if key in statDict:
data[key]=statDict[key]
Notes:
I did not test this code, check it carefully, but I hope you get the idea
You could add else to all if key in checks to include an error message if a stat is missing
If you are sure that all keys will always be present, then you can skip a few of the if checks
I'm going to ignore the actual error that got you here, and instead propose a radical restructure: I think your code will be simpler and easier to reason about.
I've looked at the JSON returned from that Pokemon API and I can see why you started down the path you did: there's a lot of data, and you only need a small subset of it. So, you're going through a lot of effort to pick out exactly what you want.
The DictWriter interface can really help you here. Consider this really small example:
header = ['name', 'id', 'order']
with open('output.csv', 'w', newline='') as f:
writer = csv.DictWriter(f, fieldnames=header)
writer.writeheader()
writer.writerow({'name': 'bulbasaur', 'id': 1, 'order': 1, 'species': {}})
Maybe you've run something like this before and got this error:
ValueError: dict contains fields not in fieldnames: 'species'
because the JSON you loaded has keys you didn't include when you created your writer... because you don't want them. And then, maybe you figured, "well, that means I've got to be very selective about what I put in the dict before passing to writerow()?
Since you've already defined which keys you care about for the header, use those keys to pull out what you want from the JSON:
header = ['name', 'id', 'order', 'height', 'weight',
'speed', 'special-defense', 'special-attack',
'defense', 'attack', 'hp']
all_data = json.load(open('1.json')) # bulbasaur, I downloaded this from the API URL
my_data = {}
for key in header:
my_data[key] = all_data.get(key) # will return None for sub-stats keys, which is okay for now
writer = csv.DictWriter(sys.stdout, fieldnames=header)
writer.writeheader()
writer.writerow(my_data)
The get(key_name) method on a dict (the JSON data) will try to find that key in the dict and return that key's value. If the key isn't found, None is returned. Running that I get the following CSV (the sub-stat columns are empty, as expected):
name,id,order,height,weight,speed,special_defense,special_attack,defense,attack,hp
bulbasaur,1,1,7,69,,,,,,
This has the same effect as your "if this key, then this value" statements, but it's driven by the data (header names) you already defined.
On to the sub-stats...
I think it's safe to assume that if there is a stats key in the JSON, each "stat object" in the list of stats will have the data you want. It's important to make sure you're only copying the stats you've specified in header; and again, you can use your data to drive the process:
for stat in all_data['stats']:
stat_name = stat['stat']['name']
if stat_name not in header:
continue # skip this sub-stat, no column for it in the CSV
base_stat = stat['base_stat']
my_data[stat_name] = base_stat
When I insert that loop, I now get this for my CSV output:
name,id,order,height,weight,speed,special_defense,special_attack,defense,attack,hp
bulbasaur,1,1,7,69,45,,,49,49,45
Some stats are populated, but some, the "special" stats are blank? That's because in your header you've named them like special_attack (with underscore) but in reality they're like special-attack (with hyphen). I fixed your header, and now I get:
name,id,order,height,weight,speed,special-defense,special-attack,defense,attack,hp
bulbasaur,1,1,7,69,45,65,65,49,49,45
Those are all the pieces you need. To put it together, I recommend the following structure... I'm a big fan of breaking up a process like this into distinct tasks: get all the data, then process all the data, then write all the processed data. It makes debugging easier, and less indentation of code:
# Make all API calls and record their JSON
all_datas = []
# loop over your API calls:
# make the request
# get the JSON data
# append JSON data to all_datas
# Process/transform the API JSON into what you want
my_data_rows = []
for all_data in all_datas:
my_data_row = {}
for key in header:
my_data_row[key] = all_data.get(key)
for stat in all_data['stats']:
stat_name = stat['stat']['name']
if stat_name not in header:
continue # skip this sub-stat
base_stat = stat['base_stat']
my_data[stat_name] = base_stat
# Write your transformed data to CSV
writer = csv.DictWriter(sys.stdout, fieldnames=header)
writer.writeheader()
writer.writerows(my_data_rows)
I have this method that writes json data to a file. The title is based on books and data is the book publisher,date,author, etc. The method works fine if I wanted to add one book.
Code
import json
def createJson(title,firstName,lastName,date,pageCount,publisher):
print "\n*** Inside createJson method for " + title + "***\n";
data = {}
data[title] = []
data[title].append({
'firstName:', firstName,
'lastName:', lastName,
'date:', date,
'pageCount:', pageCount,
'publisher:', publisher
})
with open('data.json','a') as outfile:
json.dump(data,outfile , default = set_default)
def set_default(obj):
if isinstance(obj,set):
return list(obj)
if __name__ == '__main__':
createJson("stephen-king-it","stephen","king","1971","233","Viking Press")
JSON File with one book/one method call
{
"stephen-king-it": [
["pageCount:233", "publisher:Viking Press", "firstName:stephen", "date:1971", "lastName:king"]
]
}
However if I call the method multiple times , thus adding more book data to the json file. The format is all wrong. For instance if I simply call the method twice with a main method of
if __name__ == '__main__':
createJson("stephen-king-it","stephen","king","1971","233","Viking Press")
createJson("william-golding-lord of the flies","william","golding","1944","134","Penguin Books")
My JSON file looks like
{
"stephen-king-it": [
["pageCount:233", "publisher:Viking Press", "firstName:stephen", "date:1971", "lastName:king"]
]
} {
"william-golding-lord of the flies": [
["pageCount:134", "publisher:Penguin Books", "firstName:william","lastName:golding", "date:1944"]
]
}
Which is obviously wrong. Is there a simple fix to edit my method to produce a correct JSON format? I look at many simple examples online on putting json data in python. But all of them gave me format errors when I checked on JSONLint.com . I have been racking my brain to fix this problem and editing the file to make it correct. However all my efforts were to no avail. Any help is appreciated. Thank you very much.
Simply appending new objects to your file doesn't create valid JSON. You need to add your new data inside the top-level object, then rewrite the entire file.
This should work:
def createJson(title,firstName,lastName,date,pageCount,publisher):
print "\n*** Inside createJson method for " + title + "***\n";
# Load any existing json data,
# or create an empty object if the file is not found,
# or is empty
try:
with open('data.json') as infile:
data = json.load(infile)
except FileNotFoundError:
data = {}
if not data:
data = {}
data[title] = []
data[title].append({
'firstName:', firstName,
'lastName:', lastName,
'date:', date,
'pageCount:', pageCount,
'publisher:', publisher
})
with open('data.json','w') as outfile:
json.dump(data,outfile , default = set_default)
A JSON can either be an array or a dictionary. In your case the JSON has two objects, one with the key stephen-king-it and another with william-golding-lord of the flies. Either of these on their own would be okay, but the way you combine them is invalid.
Using an array you could do this:
[
{ "stephen-king-it": [] },
{ "william-golding-lord of the flies": [] }
]
Or a dictionary style format (I would recommend this):
{
"stephen-king-it": [],
"william-golding-lord of the flies": []
}
Also the data you are appending looks like it should be formatted as key value pairs in a dictionary (which would be ideal). You need to change it to this:
data[title].append({
'firstName': firstName,
'lastName': lastName,
'date': date,
'pageCount': pageCount,
'publisher': publisher
})
I've been searching the internet and couldn't find a simple example to encode and decode a custom object using JSON in python.
Let's say I have the following class:
class Test:
def __init__(self, name=None, grade=None):
self.name = name
self.grade = grade
and also have a list of Test objects:
t1 = Test("course1", 80)
t2 = Test("course2", 90)
list_of_tests = [t1, t2]
How can I serialize the class Test and the object list_of_tests
using JSON? I want to be able to write it to a file and read it from a file, using python.
To be honest the easiest thing to do here is to manually create a list of dictionaries from your objects. Then you can pass that directly to the JSON functions.
data = [{'name': x.name, 'grade': x.grade} for x in list_of_tests]
with open('output.json', 'w') as out:
json.dump(data, out)
and read it back:
with open('output.json') as inp:
data = json.load(inp)
list_of_tests = [Test(x['name'], x['grade']) for x in data]
You can control how an unrecognised object is serialised by dumps(default=converter_function). For it to be valid JSON you'd have to return a plain dict with the fields you want plus some tag field identifying that it is to be treated specially by loads.
Then have another converter function to reverse the process passed to loads() as object_hook.
I am fairly new to code and i have a problem in reading a text file.
For my code i need to ask the user to type in a specific name code in order to proceed to the code. However, there are various name codes the user could use and i don't know how to make it so if you type either code in, you can proceed.
For example the text file looks like this
john123,x,x,x
susan233,x,x,x
conor,x,x,x
What i need to do is accept the name tag despite what one it is and be able to print it after. All the name tags are in one column.
file = open("paintingjobs.txt","r")
details = file.readlines()
for line in details:
estimatenum = input ("Please enter the estimate number.")
if estimatenum = line.split
This is my code so far, but i do not know what to do in terms of seeing if the name tag is valid to let the user proceed.
Here is another solution, without pickle. I'm assuming that your credentials are stored one per line. If not, you need to tell me how they are separated.
name = 'John'
code = '1234'
with open('file.txt', 'r') as file:
possible_match = [line.replace(name, '') for line in file if name in line]
authenticated = False
for item in possible_match:
if code in tmp: # Or, e.g. int(code) == int(tmp)
authenticated = True
break
You can use a module called pickle. This is a Python 3.0 internal library. In Python 2.0, it is called: cPickle; everything else is the same in both.
Be warned that the way you're doing this is not a secure approach!
from pickle import dump
credentials = {
'John': 1234,
'James': 4321,
'Julie': 6789
}
dump(credentials, open("credentials.p", "wb"))
This saves a file entitled credentials.p. You can the load this as follows:
from pickle import load
credentials = load(open("credentials.p", "rb"))
print(credentials)
Here are a couple of tests:
test_name = 'John'
test_code = 1234
This will amount to:
print('Test: ', credentials[test_name] == test_code)
which displays: {'John': 1234, 'James': 4321, 'Julie': 6789}
Displays: Test: True
test_code = 2343
print('Test:', credentials[test_name] == test_code)
Displays: Test: False