Accessing Nested JSON [AWS Metadata] with Python - python

I'm using Lambda to run through my AWS account, returning a list of all instances. I need to be able to print out all of the 'VolumeId' values, but I can't work out how to access them as they are nested. I am able to print out the first VolumeId for each instance, however, some of the instances have several volumes, and some only have one. I think I know why I get these results, but I can't work out what to do to get all of them back.
Here's a snippet of what the JSON for one instance looks like:
{
'Groups':[],
'Instances':[
{
'AmiLaunchIndex':0,
'ImageId':'ami-0',
'InstanceId':'i-0123',
'InstanceType':'big',
'KeyName':'nonprod',
'LaunchTime':'date',
'Monitoring':{
'State':'disabled'
},
'Placement':{
'AvailabilityZone':'world',
'GroupName':'',
'Tenancy':'default'
},
'PrivateDnsName':'secret',
'PrivateIpAddress':'1.2.3.4',
'ProductCodes':[
],
'PublicDnsName':'',
'State':{
'Code':80,
'Name':'stopped'
},
'StateTransitionReason':'User initiated',
'SubnetId':'subnet-1',
'VpcId':'vpc-1',
'Architecture':'yes',
'BlockDeviceMappings':[
{
'DeviceName':'/sda',
'Ebs':{
'AttachTime':'date',
'DeleteOnTermination':True,
'Status':'attached',
'VolumeId':'vol-1'
}
},
{
'DeviceName':'/sdb',
'Ebs':{
'AttachTime':'date'),
'DeleteOnTermination':False,
'Status':'attached',
'VolumeId':'vol-2'
}
}
],
This is what I'm doing to get the first VolumeId:
ec2client = boto3.client('ec2')
ec2 = ec2client.describe_instances()
for reservation in ec2["Reservations"]:
for instance in reservation["Instances"]:
instanceid = instance["InstanceId"]
volumes = instance["BlockDeviceMappings"][0]["Ebs"]["VolumeId"]
print("The associated volume IDs for this instance are: ",(volumes))
I think the reason that I'm getting just the first ID is because I'm referencing the first element within "BlockDeviceMappings", but I can't work out how to get the other ones. If I try it without specifying the [0], I get the list indices must be integers or slices, not str error. I tried to use a dictionary instead of a list too, but felt like I was barking up the wrong tree with that one. Any suggestions/help would be appreciated!

One possible answer, not particularly pythonic
...
id_list = []
volumes_data = instance["BlockDeviceMappings"]
for element in volumes_data:
id_list.append(element["Ebs"]["VolumeId"])
Or else use json.loads and then iterate though json using .get syntax like the final answer in this

Related

How to change the value of a JSON list?

I know for fact that this question has been asked before. I looked at the following posts:
Python replace values in unknown structure JSON file
How to find and replace a part of a value in json file
https://pretagteam.com/question/how-to-replace-a-value-in-json-file-using-python
How to find and replace a part of a value in json file (Somehow this is what I am looking for but it does not fit my problem with the links being different and stored all under "latest")
and much more contributions, however I am still stuck.
I have a simple JSON structure like this:
{
"guild1": {
"latest": [
"link1",
"link2"
],
"channel": channel_here
},
"guild2": {
"latest": [
"link"
],
"channel": channel_here
}
}
I found a way to iterate over the entries and print the list in which the condition is found, for example:
# searching for link 1
["link1", "link2", "link3"] # here link 1 for example is found in one of the list entries
["link5"] -> This for example does not match what I am looking for, I can ignore this
If found, I simply want to replace it (just link1) with another before defined value. I tried to iterate over the keys of the found list but was not able to .replace it or something else. The key which matches my search criteria is printed out, but I can't do anything else with it. Is it even possible to change the entry in a list that easy?
Mostly I also got the following error which I also looked up but did not get any did not get any wiser:
TypeError: string indices must be integers
Here is how I search for the list end the "entries":
if defined_link in data[searchloop]['latest']: # data is my JSON file, searchloop the for-loop
for key in data[searchloop]['latest']: # Get all the matching keys
if key == defined_link: # Get the key I am looking for
Maybe someone else can help me here!
This code would change one value in a list (here 'link1') to another:
L = ["link1", "link2", "link3"]
L[L.index('link1')] = 'someOtherValue'
After this you would see that L changed to ["someOtherValue", "link2", "link3"]
You first find at which index is 'link1', let's say you get back 3. So, you know you have to change the value of L[3]. Now it's simple. Do L[3] = 'someOtherValue'.
Iterate over each guild in the json structure.
Iterate over each link in that guild's "latest" list.
If the link matches, assign that list item to something else.
for guild in jsonstructure:
for i in range(len(jsonstructure[guild]['latest'])):
if jsonstructure[guild]['latest'][i] == 'something':
jsonstructure[guild]['latest'][i] = 'new thing'
I think you should try
list.append(i)
It will work I faced same problem
For example:
string = "Hello World!"
list = []
for i in string:
list.append(string[i])
print(list)

How to iterate over a python dictionary, setting the key of the dictionary as another dictionary's value

I come from a C++ background, I am new to Python, and I suspect this problem has something to do with [im]mutability.
I am building a JSON representation in Python that involves several layers of nested lists and dictionaries in one "object". My goal is to call jsonify on the end result and have it look like nicely structured data.
I hit a problem while building out an object:
approval_groups_list = list()
approval_group_dict = dict()
for groupMemKey, groupvals in groupsAndMembersDict.items():
approval_group_dict["group_name"] = groupMemKey
approval_group_dict["name_dot_numbers"] = groupvals # groupvals is a list of strings
approval_groups_list.append(approval_group_dict)
entity_approval_unit["approval_groups"] = approval_groups_list
The first run does as expected, but after, whatever groupMemkey is touched last, that is what all other objects mirror.
groupsAndMembersDict= {
'Art': ['string.1', 'string.2', 'string.3'],
'Math': ['string.10', 'string.20', 'string.30']
}
Expected result:
approval_groups:
[
{
"group_name": "Art",
"name_dot_numbers": ['string.1', 'string.2', 'string.3']
},
{
"group_name": "Math",
"name_dot_numbers": ['string.10', 'string.20', 'string.30']
}
]
Actual Result:
approval_groups:
[
{
"group_name": "Math",
"name_dot_numbers": ['string.10', 'string.20', 'string.30']
},
{
"group_name": "Math",
"name_dot_numbers": ['string.10', 'string.20', 'string.30']
}
]
What is happening, and how do I fix it?
Your problem is not the immutability, but the mutability of objects. I'm sure you would have ended up with the same result with the equivalent C++ code.
You construct approval_group_dict before the for loop and keep reusing it. All you have to do is to move the construction inside for so that a new object is created for each loop:
approval_groups_list = list()
for groupMemKey, groupvals in groupsAndMembersDict.items():
approval_group_dict = dict()
...
Through writing this question, it dawned on me to try a few things including this, which fixed my problem - however, I still don't know exactly why this works. Perhaps it is more like a pointer/referencing problem?
approval_groups_list = list()
approval_group_dict = dict()
for groupMemKey, groupvals in groupsAndMembersDict.items():
approval_group_dict["group_name"] = groupMemKey
approval_group_dict["name_dot_numbers"] = groupvals
approval_groups_list.append(approval_group_dict.copy()) # <== note, here is the difference ".copy()"
entity_approval_unit["approval_groups"] = approval_groups_list
EDIT: The problem turns out to be that Python is Pass by [object] reference all the time. If you are new to Python like me, this means: "pass by reference, except when the thing you are passing is immutable, then its pass by value". So in a way it did have to do with [im]mutability. Mostly it had to do with my lack of understanding how Python passes references.

How to call get() on dictionary with indexes?

I have an array of dictionaries but i am running into a scenario where I have to get the value from 1st index of the array of dictionaries, following is the chunk that I am trying to query.
address_data = record.get('Rdata')[0].get('Adata')
This throws the following error:
TypeError: 'NoneType' object is not subscriptable
I tried following:
if record.get('Rdata') and record.get('Rdata')[0].get('Adata'):
address_data = record.get('Rdata')[0].get('Adata')
but I don't know if the above approach is good or not.
So how to handle this in python?
Edit:
"partyrecord": {
"Rdata": [
{
"Adata": [
{
"partyaddressid": 172,
"addressid": 142165
}
]
}
]
}
Your expression assumes that record['Rdata'] will return a list with at least one element, so provide one if that isn't the case.
address_data = record.get('Rdata', [{}])[0].get('Adata')
Now if record['Rdata'] doesn't exist, you'll still have an empty dict on which to invoke get('Adata'). The end result will be address_data being set to None.
(Checking for the key first is preferable if a suitable default is expensive to create, since it will be created whether get needs to return it or not. But [{}] is fairly lightweight, and the compiler can generate it immediately.)
You might want to go for the simple, not exciting route:
role_data = record.get('Rdata')
if role_data:
address_data = role_data[0].get('Adata')
else:
address_data = None

TypeError : Trouble accessing JSON metadata with Python

So I'm trying to access the following JSON data with python and when i give the statement :
print school['students']
The underlying data gets printed but what I really want to be able to do is print the 'id' value.
{ 'students':[
{
'termone':{
'english':'fifty',
'science':'hundred'
},
'id':'RA1081310005'
}
]
}
So when I do the following I get an error :
print school ['students']['id']
TypeError: list indices must be integers, not str
Can anyone suggest how i can access the ID & where I'm going wrong!
school['students'] is a list. You are trying to access the first element of that list and id key belongs to that element. Instead, try this:
school['students'][0]['id']
Out: 'RA1081310005'
The problem here is that in your list, 'id' is not a part of a dictionary, it is part of a list. To fix this, change your dictionary to the following:
school = {'students':{
'termone': {
"english": "fifty:,
"science": "hundred
},
"id":"RA1081310005"
}
}
Basically, you have a list, and there is no reason to have it, so I removed it.

Python parsing json issue

I'm having troubles parsing a complicated json. That is it:
{
"response": {
"players": [
{
"bar": "76561198034360615",
"foo": "49329432943232423"
}
]
}
}
My code:
url = urllib.urlopen("foobar").read()
js = json.load(url)
data = js['response']
print data['players']
The problem is that this would print the dict. What I want is to reach the key's values, like foo and bar. What I tried so far is doing data['players']['foo'] and it gives me an error that list indices should be integers, I tried of course, it didn't work. So my question is how do I reach these values? Thanks in advance.
data['response']['players'] is an array (as defined by the brackets ([, ]), so you need to access items using a specific index (0 in this case):
data['players'][0]['foo']
Or iterate over all players:
for player in data['response']['players']:
print player['foo']
The problem is that players is a list of items ([ ] in json). So you need to select the first and only item in this case using [0].
print data['players'][0]['foo']
But, keep in mind that you may have more than one player, in which case you either need to specify the player, or loop through the players using a for loop
for player in data['players']:
print player['foo']

Categories