I'm able to load and parse a json file with Python by referring to list items by name. My users.json data file:
{
"joe": {
"secret": "abc123.321def"
},
"sarah": {
"secret": "led789.321plo"
},
"dave": {
"secret": "ghi532.765dlmn"
}
}
My code - to output the 'secret' value associated with a specific user (e.g. Dave):
import json
with open('users_sample.json') as f:
users = json.load(f)
# f.close()
print(users['dave']['secret'])
This outputs Dave's secret:
ghi532.765dlmn
That's easy enough when I can predict or know the user names, but how do I iterate through each user in the users.json file and output each user's 'secret' value?
Thanks in advance!
I would encapsulate the logic to print each user and their associated function into a helper function:
def print_users(users_dict: dict, header='Before'):
print(f'-- {header}')
for u in users_dict:
print(f' {u}: {users_dict[u].get("secret", "<empty>")}')
Then, upon loading the users object initially via json.load, you can then call the function like so:
print_users(users)
To replace the secret for each user, in this case to replace every occurrence of a dot . with a plus +, a simple approach could be to use a for loop to update the users object in place:
for name, user in users.items():
if 'secret' in user:
user['secret'] = user['secret'].replace('.', '+')
Then print the result after the replacements are carried out:
print_users(users, 'After')
Finally, we can write the result users object back out to a file:
with open('users_sample_UPDATED.json', 'w') as out_file:
json.dump(users, out_file)
The output of the above code, in this case would be:
-- Before
joe: abc123.321def
sarah: led789.321plo
dave: ghi532.765dlmn
-- After
joe: abc123+321def
sarah: led789+321plo
dave: ghi532+765dlmn
The full code:
import json
def main():
with open('users_sample.json') as f:
users = json.load(f)
print_users(users)
new_users = {name: {'secret': user['secret'].replace('.', '+')}
for name, user in users.items()}
print_users(new_users, 'After')
with open('users_sample_UPDATED.json', 'w') as out_file:
json.dump(new_users, out_file)
def print_users(users_dict: dict, header='Before'):
print(f'-- {header}')
for u in users_dict:
print(f' {u}: {users_dict[u].get("secret", "<empty>")}')
if __name__ == '__main__':
main()
Iterate the dictionary using a for loop
code that works:
import json
with open('users_sample.json') as f:
users = json.load(f)
for user in users:
print(f"user name: {user} secret: {users[user]['secret']}")
You have a nested dictionary - i.e., each value associated with a top-level key is also a dictionary. You can iterate over those dictionaries with the built-in values() function. This leads to:
print(*[e.get('secret') for e in users.values()], sep='\n')
Related
Here's the code I'm trying to run. Works when executed from PyCharm. I set up a cronjob and it worked wonders for weeks. It's now giving a KeyError out of the bloom. Can't find what's wrong since I haven't touched anything in the cronjob.
import csv
import json
import os
import random
import time
from urllib.parse import urlencode
import requests
API_URL = "https://community.koodomobile.com/widget/pointsLeaderboard?"
LEADERBOARD_FILE = "leaderboard_data.json"
def get_leaderboard(period: str = "allTime", max_results: int = 20) -> list:
payload = {"period": period, "maxResults": max_results}
return requests.get(f"{API_URL}{urlencode(payload)}").json()
def dump_leaderboard_data(leaderboard_data: dict) -> None:
with open("leaderboard_data.json", "w") as jf:
json.dump(leaderboard_data, jf, indent=4, sort_keys=True)
def read_leaderboard_data(data_file: str) -> dict:
with open(data_file) as f:
return json.load(f)
def parse_leaderboard(leaderboard: list) -> dict:
return {
item["name"]: {
"id": item["id"],
"score_data": {
time.strftime("%Y-%m-%d %H:%M"): item["points"],
},
"rank": item["leaderboardPosition"],
} for item in leaderboard
}
def update_leaderboard_data(target: dict, new_data: dict) -> dict:
for player, stats in new_data.items():
target[player]["rank"] = stats["rank"]
target[player]["score_data"].update(stats["score_data"])
return target
def leaderboard_to_csv(leaderboard: dict) -> None:
data_rows = [
[player] + list(stats["score_data"].values())
for player, stats in leaderboard.items()
]
random_player = random.choice(list(leaderboard.keys()))
dates = list(leaderboard[random_player]["score_data"])
with open("the_data.csv", "w") as output:
w = csv.writer(output)
w.writerow([""] + dates)
w.writerows(data_rows)
def script_runner():
if os.path.isfile(LEADERBOARD_FILE):
fresh_data = update_leaderboard_data(
target=read_leaderboard_data(LEADERBOARD_FILE),
new_data=parse_leaderboard(get_leaderboard()),
)
leaderboard_to_csv(fresh_data)
dump_leaderboard_data(fresh_data)
else:
dump_leaderboard_data(parse_leaderboard(get_leaderboard()))
if __name__ == "__main__":
script_runner()
Here's the error taht it gives me. Seems like there's a problem with dictionary hence the KeyError.
File "/Users/Rob/PycharmProjects/Koodo/TEST.Json.py", line 75, in <module>
script_runner()
File "/Users/Rob/PycharmProjects/Koodo/TEST.Json.py", line 64, in script_runner
fresh_data = update_leaderboard_data(
File "/Users/Rob/PycharmProjects/Koodo/TEST.Json.py", line 44, in update_leaderboard_data
target[player]["rank"] = stats["rank"]
KeyError: 'triggered123'
Here's the data in JSON file : https://pastebin.com/HQyL4Kyx
It looks to me like the first time your code is run, it caches the leaderboard.
On subsequent runs, it will update the existing leaderboard with the new values, by looping over each of the new entries, finding their username, and looking up that key in the older dictionary. However, if there's a new user, that key will not exist in the old dictionary.
You're getting the error because the player triggered123 is a new user to the leaderboard, and was listed after you first ran the script.
You need to update update_leaderboard_data to handle this case (by checking if the key exists in the dictionary before attempting to access it).
That's because the key named triggered123 is NOT present in the dictionary target.
Source: https://wiki.python.org/moin/KeyError
The problem is that in update_leaderboard_data you iterate over the new data and use them to lookup in the old data. If a new key does not already exists in the old data, you'll get the KeyError.
Try to print the dict target to see if the key triggered123 is in it.
Or use the get method of the dict with a default value: so no error wil be raised, and you'll be able to search this default value in your output.
I made a pickled dictionary, now I only want to retrieve the values associated with a user-inputted country.
import pickle
choice = input("Choose a country: ")
choice.capitalize()
file_name = "nationDict.dat"
fileObject = open(file_name, 'rb')
countries = pickle.load(fileObject)
for choice in countries:
print(choice)
and I use this classmethod to create the dictionary
#classmethod
def dictMaker(cls):
dictCont = {}
dictPop = {}
dictArea = {}
dictDensity = {}
for i in range(193):
dictCont[Nation.country[i]] = Nation.continent[i]
dictPop[Nation.country[i]] = Nation.population[i]
dictArea[Nation.country[i]] = Nation.area[i]
dictDensity[Nation.country[i]] = Nation.density[i]
with open("nationDict.dat", 'wb') as pickUN:
pickle.dump((dictCont, dictPop, dictArea, dictDensity), pickUN, protocol=pickle.HIGHEST_PROTOCOL)
I want to get data only for the country of choice, but I don't understand how. I end up getting the data for every country, I do get the 4 different sets of info I want though, but I want it for only 1 country. Everything I look up is about printing entire dictionaries, but I can't find anything talking about individual values only. I've tried just about every keyword to find things on this site.
I would consider storing your country data in a different form, such as nested dictionary:
import pickle
countries = {
Nation.country[i]: {
"continent": Nation.continent[i],
"population": Nation.population[i],
"area": Nation.area[i],
"density": Nation.density[i],
}
for i in range(193)
}
# Now you can pickle only one object:
with open("nation_dict.dat", "wb") as fh:
pickle.dump(countries, fh, protocol=pickle.HIGHEST_PROTOCOL)
And your script becomes:
import pickle
choice = input("Choose a country: ")
choice.capitalize()
file_name = "nationDict.dat"
with (file_name, 'rb') as fh:
countries = pickle.load(fileObject)
print(countries.get(choice))
# {"continent": "Europe", "population": 123456789, "area": 12345, "density": 12345}
Once your script is working I recommend posting on Code Review.
for countryDict in countries:
print(countryDict[choice])
Should do the trick. The variable that you have defined as countries is actually a tuple of dictionaries (dictCont, dictPop, dictArea, dictDensity). So the for loop iterates over each of those dicts and then gets the country of choice from them. In this case, countries is a poor name choice. I had read it and assumed it was a single dictionary with an array of values, as I was too lazy to read your second code block. As a rule of thumb, always assume other coders are lazy. Trust me.
I'm trying to create a dict that contains a list of users and their ssh-keys.
The list of users and the ssh-keys are stored in different yaml files which need to grab the info from. The files are "admins" and "users" and they look like:
Admins file:
admins:
global:
- bob
- john
- jimmy
- hubert
SSH key file:
users:
bob:
fullname: Bob McBob
ssh-keys:
ssh-rsa "thisismysshkey"
john:
fullname: John McJohn
ssh-keys:
ssh-rsa "thisismysshkey"
So far i have this code:
import yaml
#open admins list as "f"
f = open("./admins.sls", 'r')
#creates "admins" list
admins = yaml.load(f)
#grab only needed names and make a list
admins = admins['admins']['global']
#convert back to dict with dummy values of 0
admin_dict = dict.fromkeys(admins, 0)
So at this point I have this dict:
print(admin_dict)
{'bob': 0, 'john': 0}
Now i want to loop through the list of names in "admins" and update the key (currently set to 0) with their ssh-key from the other file.
So i do:
f = open("./users.sls", 'r')
ssh_keys = yaml.load(f)
for i in admins:
admin_dict[k] = ssh_keys['users'][i]['ssh-keys']
but when running that for loop, only one value is getting updated.
Kinda stuck here, i'm way out of my python depth... am i on the right track here?
edit:
changed that last loop to be:
for i in admins:
for key, value in admin_dict.items():
admin_dict[key] = ssh_keys['users'][i]['ssh-keys']
and things look better. Is this valid?
With an admin.yaml file like:
admins:
global:
- bob
- john
- jimmy
- hubert
And a ssh_key.yaml like so:
users:
bob:
fullname: Bob McBob
ssh-keys:
ssh-rsa: "bob-rsa-key"
john:
fullname: John McJohn
ssh-keys:
ssh-rsa: "john-rsa-key"
jimmy:
fullname: Jimmy McGill
ssh-keys:
ssh-rsa: "jimmy-rsa-key"
ssh-ecdsa: "jimmy-ecdsa-key"
You could do something like this asssuming you want to know which type of ssh key each user has (if not just go index one level deeper for the specific name of the key type in the dictionary comprehension):
import yaml
import pprint
def main():
with open('admin.yaml', 'r') as f:
admins_dict = yaml.load(f, yaml.SafeLoader)
admins_list = admins_dict['admins']['global']
with open('ssh_keys.yaml', 'r') as f:
ssh_dict = yaml.load(f, yaml.SafeLoader)
users_dict = ssh_dict['users']
admins_with_keys_dict = {
admin: users_dict[admin]['ssh-keys'] if admin in users_dict else None
for admin in admins_list
}
pp = pprint.PrettyPrinter(indent=2)
pp.pprint(admins_with_keys_dict)
if __name__ == '__main__':
main()
Output:
{ 'bob': {'ssh-rsa': 'bob-rsa-key'},
'hubert': None,
'jimmy': {'ssh-ecdsa': 'jimmy-ecdsa-key', 'ssh-rsa': 'jimmy-rsa-key'},
'john': {'ssh-rsa': 'john-rsa-key'}}
Alternative Output if you only want the rsa keys:
{ 'bob': 'bob-rsa-key',
'hubert': None,
'jimmy': 'jimmy-rsa-key',
'john': 'john-rsa-key'}
Above output achieved making the following change to the dictionary comprehension:
admin: users_dict[admin]['ssh-keys']['ssh-rsa'] if admin in users_dict else None
^^^^^^^^^^^
I am planning to use JSON file as a simple database, i am trying to append to it new entries and try to take my entries later.
This is the code i have:
import json
import time
try:
with open('json_database.json', 'r') as json_database:
profiles = json.load(json_database)
except FileNotFoundError:
profiles = []
while True:
answer = input('list info (l), write info (w), new info (a)').lower()
if answer == 'w':
break
elif answer == 'l':
print(profiles)
else:
username = input('username: ')
email = input('Email: ')
rating = input('Rating: ')
lichess_profiles.append({
'profile':{
'username': lichess_username,
'email': email,
'rating': rating
}
})
with open('json_database.json', 'w') as json_database:
json.dump(profiles, json_database)
Now i want to call the info from the JSON info ! thats what i added :
with open('json_database.json') as json_1:
result = json.load(json_1)
print(result['profile']['email'])
what is the reason of that ? what shall i add ?
i tried that code but it raise this error :
TypeError: list indices must be integers or slices, not str
The base item you are writing to the json file is a list, but you're treating it like a dictionary. It contains dictionaries, but you have to access it like a list:
print(result[0]['profile']['email'])
print(result[1]['profile']['email'])
# etc.
I have this method that writes json data to a file. The title is based on books and data is the book publisher,date,author, etc. The method works fine if I wanted to add one book.
Code
import json
def createJson(title,firstName,lastName,date,pageCount,publisher):
print "\n*** Inside createJson method for " + title + "***\n";
data = {}
data[title] = []
data[title].append({
'firstName:', firstName,
'lastName:', lastName,
'date:', date,
'pageCount:', pageCount,
'publisher:', publisher
})
with open('data.json','a') as outfile:
json.dump(data,outfile , default = set_default)
def set_default(obj):
if isinstance(obj,set):
return list(obj)
if __name__ == '__main__':
createJson("stephen-king-it","stephen","king","1971","233","Viking Press")
JSON File with one book/one method call
{
"stephen-king-it": [
["pageCount:233", "publisher:Viking Press", "firstName:stephen", "date:1971", "lastName:king"]
]
}
However if I call the method multiple times , thus adding more book data to the json file. The format is all wrong. For instance if I simply call the method twice with a main method of
if __name__ == '__main__':
createJson("stephen-king-it","stephen","king","1971","233","Viking Press")
createJson("william-golding-lord of the flies","william","golding","1944","134","Penguin Books")
My JSON file looks like
{
"stephen-king-it": [
["pageCount:233", "publisher:Viking Press", "firstName:stephen", "date:1971", "lastName:king"]
]
} {
"william-golding-lord of the flies": [
["pageCount:134", "publisher:Penguin Books", "firstName:william","lastName:golding", "date:1944"]
]
}
Which is obviously wrong. Is there a simple fix to edit my method to produce a correct JSON format? I look at many simple examples online on putting json data in python. But all of them gave me format errors when I checked on JSONLint.com . I have been racking my brain to fix this problem and editing the file to make it correct. However all my efforts were to no avail. Any help is appreciated. Thank you very much.
Simply appending new objects to your file doesn't create valid JSON. You need to add your new data inside the top-level object, then rewrite the entire file.
This should work:
def createJson(title,firstName,lastName,date,pageCount,publisher):
print "\n*** Inside createJson method for " + title + "***\n";
# Load any existing json data,
# or create an empty object if the file is not found,
# or is empty
try:
with open('data.json') as infile:
data = json.load(infile)
except FileNotFoundError:
data = {}
if not data:
data = {}
data[title] = []
data[title].append({
'firstName:', firstName,
'lastName:', lastName,
'date:', date,
'pageCount:', pageCount,
'publisher:', publisher
})
with open('data.json','w') as outfile:
json.dump(data,outfile , default = set_default)
A JSON can either be an array or a dictionary. In your case the JSON has two objects, one with the key stephen-king-it and another with william-golding-lord of the flies. Either of these on their own would be okay, but the way you combine them is invalid.
Using an array you could do this:
[
{ "stephen-king-it": [] },
{ "william-golding-lord of the flies": [] }
]
Or a dictionary style format (I would recommend this):
{
"stephen-king-it": [],
"william-golding-lord of the flies": []
}
Also the data you are appending looks like it should be formatted as key value pairs in a dictionary (which would be ideal). You need to change it to this:
data[title].append({
'firstName': firstName,
'lastName': lastName,
'date': date,
'pageCount': pageCount,
'publisher': publisher
})