I have two lists:
managers = ['john', 'karl', 'ricky']
sites = ['site1', 'site2', 'site3']
I also have a json file for each manager:
{
"results": [
{
"id": "employee1",
"user": {
"location": "site1"
}
},
{
"id": "employee2",
"user": {
"location": "site1"
}
},
{
"id": "employee3",
"user": {
"location": "site2"
}
},
{
"id": "employee4",
"user": {
"location": "site3"
}
}
]
}
So now, I want to create a dictionary by to categorize data by site! which is making it challenging for me. The output I need is to show employees for each site.
Here is the code I have:
myDict = dict()
for j in managers:
for i in range(len(sites)):
with open(""+j+"-data.json", 'r') as jasonRead:
json_object = json.loads(jasonRead.read())
for result in json_object['results']:
site = (result['user']['location'])
for employee, manager in map(str.split, str(site), managers[i]):
myDict.setdefault(managers[i], []).append(site)
The error I get:
File "test.py", line 25, in <module>
for employee, manager in map(str.split, str(site), managers[i]):
ValueError: need more than 1 value to unpack
Instead of looping over sites, just test if the site of the employee is in the list.
The keys of the resulting dictionary should be the site names, not the manager names.
mydict = {}
for manager in managers:
with open(f'{manager}-data.json') as jasonRead:
json_object = json.load(jasonRead)
for result in json_object['results']:
site = result['user']['location']
if site in sites:
mydict.setdefault(site, []).append(result['id'])
print(mydict)
Related
If I have a heavy json file that have 30m entries like that
{"id":3,"price":"231","type":"Y","location":"NY"}
{"id":4,"price":"321","type":"N","city":"BR"}
{"id":5,"price":"354","type":"Y","city":"XE","location":"CP"}
--snip--
{"id":30373779,"price":"121","type":"N","city":"SR","location":"IU"}
{"id":30373780,"price":"432","type":"Y","location":"TB"}
{"id":30373780,"price":"562","type":"N","city":"CQ"}
how I can only abstract the location and the city and parse it into one json like that in python:
{
"orders":{
3:{
"location":"NY"
},
4:{
"city":"BR"
},
5:{
"city":"XE",
"location":"CP"
},
30373779:{
"city":"SR",
"location":"IU"
},
30373780:{
"location":"TB"
},
30373780:{
"city":"CQ"
}
}
}
P.S: beatufy the syntax is not necessary.
Assuming your input file is actually in jsonlines format, then you can read each line, extract the city and location keys from the dict and then append those to a new dict:
import json
from collections import defaultdict
orders = { 'orders' : defaultdict(dict) }
with open('orders.txt', 'r') as f:
for line in f:
o = json.loads(line)
id = o['id']
if 'location' in o:
orders['orders'][id]['location'] = o['location']
if 'city' in o:
orders['orders'][id]['city'] = o['city']
print(orders)
Output for your sample data (note it has two 30373780 id values, so the values get merged into one dict):
{
"orders": {
"3": {
"location": "NY"
},
"4": {
"city": "BR"
},
"5": {
"location": "CP",
"city": "XE"
},
"30373779": {
"location": "IU",
"city": "SR"
},
"30373780": {
"location": "TB",
"city": "CQ"
}
}
}
As you've said that your file is pretty big and you probably don't want to keep all entries in memory here is the way to consume source file line by line and write output immediately:
import json
with open(r"in.jsonp") as i_f, open(r"out.json", "w") as o_f:
o_f.write('{"orders":{')
for i in i_f:
i_obj = json.loads(i)
o_f.write(f'{i_obj["id"]}:')
o_obj = {}
if location := i_obj.get("location"):
o_obj["location"] = location
if city := i_obj.get("city"):
o_obj["city"] = city
json.dump(o_obj, o_f)
o_f.write(",")
o_f.write('}}')
It will generate semi-valid JSON object in same format you've provided in your question.
I have the following data:
data={
"locations": [
{
"id": "27871f2d-101c-449e-87ad-36a663b144fe",
"switch_id": 20,
"switch_port": 16,
"vlan_id": 101,
},
{
"id": "94b1d7a2-7ff2-4ba3-8259-5eb7ddd09fe1",
"switch_id": 6,
"switch_port": 24,
"vlan_id": 203,
},
]
}
And what I want to do is extract 'id' and 'vlan_id' into a new dictionary with a list of sub dictionaries, like this:
new_data={
"connections": [
{
"id": "27871f2d-101c-449e-87ad-36a663b144fe",
"vlan_id": 101,
},
{
"id": "94b1d7a2-7ff2-4ba3-8259-5eb7ddd09fe1",
"vlan_id": 203,
},
]
}
My initial thoughts were as a dictionary comprehension like this:
new_data = {"connections": [some code here]}
But not sure of the some code bit yet.
Try:
new_data = {"connections": [{'id': d['id'], 'vlan_id': d['vlan_id']} for d in data['locations']]}
{'connections': [{'id': '27871f2d-101c-449e-87ad-36a663b144fe', 'vlan_id': 101}, {'id': '94b1d7a2-7ff2-4ba3-8259-5eb7ddd09fe1', 'vlan_id': 203}]}
You can create the new_data variable accesing the first dictionary data like this:
new_data={
"connections": [
{
"id": data['locations'][0]['id'],
"vlan_id": data['locations'][0]['vlan_id'],
},
{
"id": data['locations'][1]['id'],
"vlan_id": data['locations'][1]['vlan_id'],
},
]
}
edit:
You can get a more dynamic approach by reading every object in the list with a forloop like this:
new_data={
"connections": []
}
for object in data['locations']:
new_dict = {
"id": object["id"],
"vlan_id": object["vlan_id"]
}
new_data['connections'].append(new_dict)
Following Marc's answer here, you could modify it to
new_data = {}
for i in range(len(data['locations'])):
if "connections" not in new_data.keys():
new_data['connections'] = [{"id": data['locations'][i]['id'],"vlan_id": data['locations'][i]['vlan_id']}]
else:
new_data['connections'].append({"id": data['locations'][i]['id'],"vlan_id": data['locations'][i]['vlan_id']})
The Answers here are good but you can make the code more dynamic
keys_to_extract = ['id', 'vlan_id']
locations = data['locations']
connections = { key: val for key, val in locations.items() if key in keys_to_extract }
new_data = {'connections': connections}
Now you can change the keys you need on the fly
I am working with a JSON request.get that returns a list. I will like to save each individual object in the response to my models so I did this:
in views.py:
def save_ram_api(request):
r = requests.get('https://ramb.com/ciss-api/v1/')
# data = json.loads(r)
data = r.json()
for x in data:
title = x["title"]
ramyo_donotuse = x["ramyo"]
date = x["date"]
thumbnail = x["thumbnail"]
home_team_name = x["side1"]["name"]
away_team_name = x["side2"]["name"]
competition_name = x["tournament"]["name"]
ramAdd = ramSample.objects.create(title=title, ramyo_donotuse=ramyo_donotuse, date=date, thumbnail=thumbnail, home_team_name=home_team_name, away_team_name=away_team_name, competition_name=competition_name)
ramAdd.save()
return HttpResponse("Successfully submitted!")
This works fine except that it would only save the last objects on the list.
the JSON response list (as a random 60 objects at any time) would look something like:
[
{
"title": "AY - BasketMouth",
"ramyo": "AY de comedian"
"side1": {
"name": "Comedy Central",
"url": "https:\/\/www.rabithole.com\/laugh\/dave-chappel\/"
},
"side2": {
"name": "Basket Mouth",
"url": "https:\/\/www.rabithole.com\/laugh\/chris-rockie\/"
},
"tournament": {
"name": "Night of a thousand laugh",
"id": 15,
"url": "https:\/\/www.rabithole.com\/laugh\/chris-rockie\/"
},
"points": [
{
"nature": "Gentle",
"phrase": "Just stay"
},
{
"nature": "Sarcastic",
"phrase": "Help me"
}
]
},
{
"title": "Dave - Chris",
"ramyo": "Dave Chapelle"
"side1": {
"name": "Comedy Central",
"url": "https:\/\/www.rabithole.com\/laugh\/dave-chappel\/"
},
"side2": {
"name": "Chris Rockie",
"url": "https:\/\/www.rabithole.com\/laugh\/chris-rockie\/"
},
"tournament": {
"name": "Tickle me",
"id": 15,
"url": "https:\/\/www.rabithole.com\/laugh\/chris-rockie\/"
},
"points": [
{
"nature": "Rogue",
"phrase": "Just stay"
}
]
}
]
In this case my views.py will only save the last dictionary on the list, ignoring the other 59.
My question would be:
How do I get the views.py to save the entire objects on the list?
Notice that the "points" is also a list that contains one or more dictionaries, any help how to save this as well?
Your code is saving only the last object in the list because you are creating and saving the object outside of the loop. Try this,
def save_ram_api(request):
r = requests.get('https://ramb.com/ciss-api/v1/')
# data = json.loads(r)
data = r.json()
for x in data:
title = x["title"]
ramyo_donotuse = x["ramyo"]
date = x["date"]
thumbnail = x["thumbnail"]
home_team_name = x["side1"]["name"]
away_team_name = x["side2"]["name"]
competition_name = x["tournament"]["name"]
ramAdd = ramSample.objects.create(title=title, ramyo_donotuse=ramyo_donotuse, date=date, thumbnail=thumbnail, home_team_name=home_team_name, away_team_name=away_team_name, competition_name=competition_name)
ramAdd.save()
return HttpResponse("Successfully submitted!")
How do I get the views.py to save the entire objects on the list?
Notice that the "points" is also a list that contains one or more
dictionaries, any help how to save this as well?
Regarding your those questions
If you are using PostgreSQL as a database then you can use Django's built is JSONField and ArrayField for PostgreSQL database.
And if your database is not PostgreSQL you can use jsonfield library.
Have this section in one large JSON file
"UserDetailList": [
{
"UserName": "citrix-xendesktop-ec2-provisioning",
"GroupList": [],
"CreateDate": "2017-11-07T14:20:14Z",
"UserId": "AIDAI2YJINPRUEM3XHKXO",
"Path": "/",
"AttachedManagedPolicies": [
{
"PolicyName": "AmazonEC2FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
],
"Arn": "arn:aws:iam::279052847476:user/citrix-xendesktop-ec2-provisioning"
},
Need to extract AttachedManagedPolicy.Policy name for user
Desired output:
"citrix-xendesktop-ec2-provisioning","AmazonEC2FullAccess"
"citrix-xendesktop-ec2-provisioning","AmazonS3FullAccess"
Some users don't have any policy at all so need some checking mechanism to avoid errors
with open('1.json') as file:
data = json.load(file)
for element in data['UserDetailList']:
s = element['UserName'], element['AttachedManagedPolicies']
print s
And getting
(u'citrix-xendesktop-ec2-provisioning', [{u'PolicyName': u'AmazonEC2FullAccess', u'PolicyArn': u'arn:aws:iam::aws:policy/AmazonEC2FullAccess'}, {u'PolicyName': u'AmazonS3FullAccess', u'PolicyArn': u'arn:aws:iam::aws:policy/AmazonS3FullAccess'}])
When added element['AttachedManagedPolicies']['PolicyName']
got: TypeError: list indices must be integers, not str
You are getting error because element['AttachedManagedPolicies'] is list not dictionary you need to iterate over element['AttachedManagedPolicies'] and then access key as below:
[i['PolicyName'] for i in element['AttachedManagedPolicies']]
this will construct list of values for key PolicyName
As you said you have very large JSON structure you might have empty values or not values and for that you can proceed as below:
d = {
"UserDetailList": [
{
"UserName": "citrix-xendesktop-ec2-provisioning",
"GroupList": [],
"CreateDate": "2017-11-07T14:20:14Z",
"UserId": "AIDAI2YJINPRUEM3XHKXO",
"Path": "/",
"AttachedManagedPolicies": [
{
"PolicyName": "AmazonEC2FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
],
"Arn": "arn:aws:iam::279052847476:user/citrix-xendesktop-ec2-provisioning"
}
]
}
user_list = d.get("UserDetailList", None) # if unable to fetch key then it will return None
if user_list:
for user_detail in user_list:
username = user_detail.get("UserName", None)
policies = [i.get('PolicyName') for i in user_detail.get('AttachedManagedPolicies', []) if i.get('PolicyName', None)] # empty list constructed if no policy exist
print(username, policies)
I'm using the following python code to connect to a jsonrpc server and nick some song information. However, I can't work out how to get the current title in to a variable to print elsewhere. Here is the code:
TracksInfo = []
for song in playingSongs:
data = { "id":1,
"method":"slim.request",
"params":[ "",
["songinfo",0,100, "track_id:%s" % song, "tags:GPASIediqtymkovrfijnCYXRTIuwxN"]
]
}
params = json.dumps(data, sort_keys=True, indent=4)
conn.request("POST", "/jsonrpc.js", params)
httpResponse = conn.getresponse()
data = httpResponse.read()
responce = json.loads(data)
print json.dumps(responce, sort_keys=True, indent=4)
TrackInfo = responce['result']["songinfo_loop"][0]
TracksInfo.append(TrackInfo)
This brings me back the data in json format and the print json.dump brings back:
pi#raspberrypi ~/pithon $ sudo python tom3.py
{
"id": 1,
"method": "slim.request",
"params": [
"",
[
"songinfo",
"0",
100,
"track_id:-140501481178464",
"tags:GPASIediqtymkovrfijnCYXRTIuwxN"
]
],
"result": {
"songinfo_loop": [
{
"id": "-140501481178464"
},
{
"title": "Witchcraft"
},
{
"artist": "Pendulum"
},
{
"duration": "253"
},
{
"tracknum": "1"
},
{
"type": "Ogg Vorbis (Spotify)"
},
{
"bitrate": "320k VBR"
},
{
"coverart": "0"
},
{
"url": "spotify:track:2A7ZZ1tjaluKYMlT3ItSfN"
},
{
"remote": 1
}
]
}
}
What i'm trying to get is result.songinfoloop.title (but I tried that!)
The songinfo_loop structure is.. peculiar. It is a list of dictionaries each with just one key.
Loop through it until you have one with a title:
TrackInfo = next(d['title'] for d in responce['result']["songinfo_loop"] if 'title' in d)
TracksInfo.append(TrackInfo)
A better option would be to 'collapse' all those dictionaries into one:
songinfo = reduce(lambda d, p: d.update(p) or d,
responce['result']["songinfo_loop"], {})
TracksInfo.append(songinfo['title'])
songinfo_loop is a list not a dict. That means you need to call it by position, or loop through it and find the dict with a key value of "title"
positional:
responce["result"]["songinfo_loop"][1]["title"]
loop:
for info in responce["result"]["songinfo_loop"]:
if "title" in info.keys():
print info["title"]
break
else:
print "no song title found"
Really, it seems like you would want to have the songinfo_loop be a dict, not a list. But if you need to leave it as a list, this is how you would pull the title.
The result is really a standard python dict, so you can use
responce["result"]["songinfoloop"]["title"]
which should work