I'm building a quiz app in Python 3. The users are shown a bunch of clues and they have to guess the sports star. My current backend is using SQLite, however I want to have a live database using Firebase and so need to change this to JSON format.
The table currently looks like this:
Player Name Difficulty Year Club Apps (Goals)
player_1 easy 2014 - 2017 club x (y)
player_1 easy 2017 - 2019 club_2 x (y)
player_2 medium 2019 - club x (y)
Currently, users select the difficulty. I then run a SQL query that gives me a list of all players with that difficulty. The app then displays to the user the last three columns, which are the clues they need to use to guess the player.
To put in into JSON format, I was thinking it would look something like this
{
"users":
{
"player_1_id":
{
"name": "player_1",
"difficulty": "difficulty",
"year": [year_1, year_2, ..., year_n],
"club": [club_in_year_1, club_in_year_2, ...., club_in_year_n],
"apps": [apps_in_year_1, apps_in_year_2, ..., apps_in_year_n]
},
"player_2_id":
"name": "player_2",
...
}
}
So there can be different number of values within each list of year/club/apps, per player name. I created and ran the following code:
import sqlite3
def dict_factory(cursor, row):
d = {}
for idx, col in enumerate(cursor.description):
d[col[0]] = row[idx]
return d
connection = sqlite3.connect("player_database.db")
connection.row_factory = dict_factory
cursor = connection.cursor()
cursor.execute("select * from player_history")
results = cursor.fetchall()
print(results)
connection.close()
but this prints each new row with a separate ID, which isn't quite what I want.
How would I edit this to get my desired outpout?
Here is a basic solution to map your data.
# define a function that takes a data array as argument
def mapData(data = []):
# dict that is used to store aggregated rows
results = {}
# iterate through the data array
for item in data:
# aggreagate values of there is already an entry
# in the results dict for given name
if item["name"] in results.keys():
prev = results[item["name"]]
prev["year"].append(item["year"])
prev["club"].append(item["club"])
prev["apps"].append(item["apps"])
# continue will skip to next iteration
continue
# create a new entry if there is no existing key
# for given name in the results dict
results[item["name"]] = {
**item,
"year": [item["year"]],
"club": [item["club"]],
"apps": [item["apps"]],
}
# return the result
return results
# some fake data that looks like the data from sql
sqlData= [
{
"name": "player_1",
"difficulty": "difficulty",
"year": "2009",
"club": "2003",
"apps": "2005",
},
{
"name": "player_1",
"difficulty": "difficulty",
"year": "1999",
"club": "1998",
"apps": "2008",
},
{
"name": "player_2",
"difficulty": "difficulty",
"year": "1999",
"club": "1998",
"apps": "2008",
}
]
# pass the data into the function and assign the result
# to a new variable called aggregatedData
aggregatedData = mapData(sqlData)
# print the result
print(aggregatedData)
# result:
# {
# "player_1":{
# "name":"player_1",
# "difficulty":"difficulty",
# "year":[
# "2009",
# "1999"
# ],
# "club":[
# "2003",
# "1998"
# ],
# "apps":[
# "2005",
# "2008"
# ]
# },
# "player_2":{
# "name":"player_2",
# "difficulty":"difficulty",
# "year":[
# "1999"
# ],
# "club":[
# "1998"
# ],
# "apps":[
# "2008"
# ]
# }
# }
Related
I have json file which has a list of ids and date. How to write a python program to print all the ids for a particular month from the json file
Below is the sample json data
{
"entities": [
{
"Fields": [
{
"Name": "version",
"values": [
{
"value": "Cycle 1"
}
]
},
{
"Name": "subject",
"values": [
{
"value": "1008"
}
]
},
{
"Name": "project",
"values": [
{}
]
},
{
"Name": "linkage",
"values": [
{
"value": "N"
}
]
},
{
"Name": "cycle-id",
"values": []
},
{
"Name": "creation-time",
"values": [
{
"value": "2016-07-12"
}
]
},
{
"Name": "id",
"values": [
{
"value": "1"
}
]
}]}]}
I have just tried to load the json file from below code.
import json
f = open('defects-export-0-100.json')
data = json.load(f)
print(data)
# month = str("MM")
month = '09'
defect_items = []
defectIDs = []
for item in data["entities"]:
for container in item["Fields"]:
if container["Name"] == "creation-time":
if container["values"][0]["value"].split("-")[1] == month:
defect_items.append(item)
for item in defect_items:
for container in item["Fields"]:
if container["Name"] == "id":
defectIDs.append(container["values"][0]["value"])
My desired output: All the IDs from the one particular month of creation date.
The biggest issue is how you're referencing keys in a dictionary. You can get the value at a particular key with:
x = {"key": value}
x["key"]
# value
I've made some assumptions about your data set, but this code works with the sample you gave.
import json
with open("data.txt", "r") as f:
data = json.load(f)
#month = str("MM")
month = "07"
defect_items = []
defectIDs = []
# Loop through each entity
for item in data["entities"]:
# Loop through each field
for container in item["Fields"]:
# Find the field with the name "creation-item"
if container["Name"] == "creation-time":
# Check if the value matches with the desired date
# Assuming there can only be one value
if container["values"][0]["value"].split("-")[1] == month:
defect_items.append(item)
# Loop through the defective items
for item in defect_items:
# Loop through the fields
for container in item["Fields"]:
# Find the field with the name "id"
if container["Name"] == "id":
# Grab the value
# Assuming there can only be one value
defectIDs.append(container["values"][0]["value"])
Once the data is loaded, you can interact with it as you would any Python object. Get all the items with:
items = data['entities']
For the code below to work, create a variable month and set it to a string with the format MM (where M is a digit of the month: e.g. month='01' for January) so it exactly matches the correct month format of the data.
Then, run the following loop to collect the IDs:
ids = []
for item in items.keys():
id = None
time = False
for container in items[item].keys():
if items[item][container]['Name'] == 'creation-time':
if items[item][container]['values']['value'].split('-')[1] == month:
time = True
if items[item][container]['Name'] == 'id':
id = items[item][container]['values']['value']
if time and id: ids.append(id)
I have functions witch generate data witch I add t dict the think is there I want my json file to look like this 1.to have multiple data not only one key value pair like in my code:
{"data":[
{"key":"Shyam", "value":10.4},
{"key":"Bob", "value":12.5},
{"key":"Jai", "value":24.2}
]}
This is how is look like the moment only one key value pair is added:
{
"key": "Amadm",
"value": 14.5
}
This is my code to assign to dict before json dumps.
:
def gen_dict(key, value, ts):
data = {
"name": key,
"value": value,
"ts": ts
}
return data
json_object = json.dumps(gen_dict(gen_key(), gen_value()), indent = 4)
So my question is how to assign more than one key value pair in the dict and later to transform to json obj like in the example I show in the example.
You are creating a single dictionary, what you want is a list of dictionaries:
Assuming that you each call to gen_key() and gen_value() generates a single instance of the data, you can use:
# Some random key
def gen_key():
return ''.join((random.choice(string.ascii_lowercase) for x in range(5)))
# Some random value
def gen_value():
return random.choice(range(1000))
s = json.dumps({"data": [ {
"name": gen_key(), "value": gen_value()} for i in range(3)] }, indent = 4)
output:
{
"data": [
{
"name": "rrqct",
"value": 162
},
{
"name": "vbuyq",
"value": 422
},
{
"name": "kfyqt",
"value": 7
}
]
}
I am working with a JSON request.get that returns a list. I will like to save each individual object in the response to my models so I did this:
in views.py:
def save_ram_api(request):
r = requests.get('https://ramb.com/ciss-api/v1/')
# data = json.loads(r)
data = r.json()
for x in data:
title = x["title"]
ramyo_donotuse = x["ramyo"]
date = x["date"]
thumbnail = x["thumbnail"]
home_team_name = x["side1"]["name"]
away_team_name = x["side2"]["name"]
competition_name = x["tournament"]["name"]
ramAdd = ramSample.objects.create(title=title, ramyo_donotuse=ramyo_donotuse, date=date, thumbnail=thumbnail, home_team_name=home_team_name, away_team_name=away_team_name, competition_name=competition_name)
ramAdd.save()
return HttpResponse("Successfully submitted!")
This works fine except that it would only save the last objects on the list.
the JSON response list (as a random 60 objects at any time) would look something like:
[
{
"title": "AY - BasketMouth",
"ramyo": "AY de comedian"
"side1": {
"name": "Comedy Central",
"url": "https:\/\/www.rabithole.com\/laugh\/dave-chappel\/"
},
"side2": {
"name": "Basket Mouth",
"url": "https:\/\/www.rabithole.com\/laugh\/chris-rockie\/"
},
"tournament": {
"name": "Night of a thousand laugh",
"id": 15,
"url": "https:\/\/www.rabithole.com\/laugh\/chris-rockie\/"
},
"points": [
{
"nature": "Gentle",
"phrase": "Just stay"
},
{
"nature": "Sarcastic",
"phrase": "Help me"
}
]
},
{
"title": "Dave - Chris",
"ramyo": "Dave Chapelle"
"side1": {
"name": "Comedy Central",
"url": "https:\/\/www.rabithole.com\/laugh\/dave-chappel\/"
},
"side2": {
"name": "Chris Rockie",
"url": "https:\/\/www.rabithole.com\/laugh\/chris-rockie\/"
},
"tournament": {
"name": "Tickle me",
"id": 15,
"url": "https:\/\/www.rabithole.com\/laugh\/chris-rockie\/"
},
"points": [
{
"nature": "Rogue",
"phrase": "Just stay"
}
]
}
]
In this case my views.py will only save the last dictionary on the list, ignoring the other 59.
My question would be:
How do I get the views.py to save the entire objects on the list?
Notice that the "points" is also a list that contains one or more dictionaries, any help how to save this as well?
Your code is saving only the last object in the list because you are creating and saving the object outside of the loop. Try this,
def save_ram_api(request):
r = requests.get('https://ramb.com/ciss-api/v1/')
# data = json.loads(r)
data = r.json()
for x in data:
title = x["title"]
ramyo_donotuse = x["ramyo"]
date = x["date"]
thumbnail = x["thumbnail"]
home_team_name = x["side1"]["name"]
away_team_name = x["side2"]["name"]
competition_name = x["tournament"]["name"]
ramAdd = ramSample.objects.create(title=title, ramyo_donotuse=ramyo_donotuse, date=date, thumbnail=thumbnail, home_team_name=home_team_name, away_team_name=away_team_name, competition_name=competition_name)
ramAdd.save()
return HttpResponse("Successfully submitted!")
How do I get the views.py to save the entire objects on the list?
Notice that the "points" is also a list that contains one or more
dictionaries, any help how to save this as well?
Regarding your those questions
If you are using PostgreSQL as a database then you can use Django's built is JSONField and ArrayField for PostgreSQL database.
And if your database is not PostgreSQL you can use jsonfield library.
I am trying to create a particular nested dictionary from a DataFrame in Pandas conditions, in order to then visualize.
dat = pd.DataFrame({'cat_1' : ['marketing', 'marketing', 'marketing', 'communications'],
'child_cat' : ['marketing', 'social media', 'marketing', 'communications],
'skill' : ['digital marketing','media marketing','research','seo'],
'value' : ['80', '101', '35', '31']
and I would like to turn this into a dictionary that looks a bit like this:
{
"name": "general skills",
"children": [
{
"name": "marketing",
"children": [
{
"name": "marketing",
"children": [
{
"name": "digital marketing",
"value": 80
},
{
"name": "research",
"value": 35
}
]
},
{
"name": "social media", // notice that this is a sibling of the parent marketing
"children": [
{
"name": "media marketing",
"value": 101
}
]
}
]
},
{
"name": "communications",
"children": [
{
"name": "communications",
"children": [
{
"name": "seo",
"value": 31
}
]
}
]
}
]
}
So cat_1 is the parent node, child_cat is its children, and skill is its child too. I am having trouble with creating the additional children lists. Any help?
With a lot of inefficiencies I came up with this solution. Probably highly sub-optimal
final = {}
# control dict to get only one broad category
contrl_dict = {}
contrl_dict['dummy'] = None
final['name'] = 'variants'
final['children'] = []
# line is the values of each row
for idx, line in enumerate(df_dict.values):
# parent categories dict
broad_dict_1 = {}
print(line)
# this takes every value of the row minus the value in the end
for jdx, col in enumerate(line[:-1]):
# look into the broad category first
if jdx == 0:
# check in our control dict - does this category exist? if not add it and continue
if not col in contrl_dict.keys():
# if it doesn't it appends it
contrl_dict[col] = 'added'
# then the broad dict parent takes the name
broad_dict_1['name'] = col
# the children are the children broad categories which will be populated further
broad_dict_1['children'] = []
# go to broad categories 2
for ydx, broad_2 in enumerate(list(df_dict[df_dict.broad_categories == col].broad_2.unique())):
# sub categories dict
prov_dict = {}
prov_dict['name'] = broad_2
# children is again a list
prov_dict['children'] = []
# now isolate the skills and values of each broad_2 category and append them
for row in df_dict[df_dict.broad_2 == broad_2].values:
prov_d_3 = {}
# go to each row
for xdx, direct in enumerate(row):
# in each row, values 2 and 3 are name and value respectively add them
if xdx == 2:
prov_d_3['name'] = direct
if xdx == 3:
prov_d_3['size'] = direct
prov_dict['children'].append(prov_d_3)
broad_dict_1['children'].append(prov_dict)
# if it already exists in the control dict then it moves on
else:
continue
final['children'].append(broad_dict_1)
I'm in over my head, trying to parse JSON for my first time and dealing with a multi dimensional array.
{
"secret": "[Hidden]",
"minutes": 20,
"link": "http:\/\/www.1.com",
"bookmark_collection": {
"free_link": {
"name": "#free_link#",
"bookmarks": [
{
"name": "1",
"link": "http:\/\/www.1.com"
},
{
"name": "2",
"link": "http:\/\/2.dk"
},
{
"name": "3",
"link": "http:\/\/www.3.in"
}
]
},
"boarding_pass": {
"name": "Boarding Pass",
"bookmarks": [
{
"name": "1",
"link": "http:\/\/www.1.com\/"
},
{
"name": "2",
"link": "http:\/\/www.2.com\/"
},
{
"name": "3",
"link": "http:\/\/www.3.hk"
}
]
},
"sublinks": {
"name": "sublinks",
"link": [
"http:\/\/www.1.com",
"http:\/\/www.2.com",
"http:\/\/www.3.com"
]
}
}
}
This is divided into 3 parts, the static data on my first dimension (secret, minutes, link) Which i need to get as seperate strings.
Then I need a dictionary per "bookmark collection" which does not have fixed names, so I need the name of them and the links/names of each bookmark.
Then there is the seperate sublinks which is always the same, where I need all the links in a seperate dictionary.
I'm reading about parsing JSON but most of the stuff I find is a simple array put into 1 dictionary.
Does anyone have any good techniques to do this ?
After you parse the JSON, you will end up with a Python dict. So, suppose the above JSON is in a string named input_data:
import json
# This converts from JSON to a python dict
parsed_input = json.loads(input_data)
# Now, all of your static variables are referenceable as keys:
secret = parsed_input['secret']
minutes = parsed_input['minutes']
link = parsed_input['link']
# Plus, you can get your bookmark collection as:
bookmark_collection = parsed_input['bookmark_collection']
# Print a list of names of the bookmark collections...
print bookmark_collection.keys() # Note this contains sublinks, so remove it if needed
# Get the name of the Boarding Pass bookmark:
print bookmark_collection['boarding_pass']['name']
# Print out a list of all bookmark links as:
# Boarding Pass
# * 1: http://www.1.com/
# * 2: http://www.2.com/
# ...
for bookmark_definition in bookmark_collection.values():
# Skip sublinks...
if bookmark_definition['name'] == 'sublinks':
continue
print bookmark_definition['name']
for bookmark in bookmark_definition['bookmarks']:
print " * %(name)s: %(link)s" % bookmark
# Get the sublink definition:
sublinks = parsed_input['bookmark_collection']['sublinks']
# .. and print them
print sublinks['name']
for link in sublinks['link']:
print ' *', link
Hmm, doesn't json.loads do the trick?
For example, if your data is in a file,
import json
text = open('/tmp/mydata.json').read()
d = json.loads(text)
# first level fields
print d['minutes'] # or 'secret' or 'link'
# the names of each of bookmark_collections's items
print d['bookmark_collection'].keys()
# the sublinks section, as a dict
print d['bookmark_collection']['sublinks']
The output of this code (given your sample input above) is:
20
[u'sublinks', u'free_link', u'boarding_pass']
{u'link': [u'http://www.1.com', u'http://www.2.com', u'http://www.3.com'], u'name': u'sublinks'}
Which, I think, gets you what you need?