We Keep Coding

How to avoid nested loops in python parsing aws

I'm relatively new to python and I don't know all the mysteries of this language yet so I was wondering if there are any ways I can optimize this code.
I'm trying to list the name of my EC2 instances in an AWS lambda using boto3 and python.
Here's the code :
import json
import boto3
import botocore
import logging
# Create a logging message
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(name)s:%(message)s')
# Create EC2 resource
ec2 = boto3.client('ec2')
ec2_list = ec2.describe_instances()
def lambda_handler(event, context):
try:
for reservation in ec2_list['Reservations']:
for instance in reservation['Instances']:
for tag in instance['Tags']:
print(tag['Value'])
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
except botocore.exceptions.ClientError as e:
logger.debug(e)
raise e
I also tried that as seen in another post, but it didnt work, because reservation var is referenced before assignment - seems logic:
for reservation, instance, tag in itertools.product(ec2_list['Reservations'], reservation['Instances'], instance['Tags']):
print(tag['Value'])
And here is the thing I need to parse (I reduced it a lottttt for the post) :
[
{
'Groups': [],
'Instances': [
{
'Tags': [
{
'Key': 'Name',
'Value': 'Second Instance'
}
],
}
],
},
{
'Groups': [],
'Instances': [
{
'Tags': [
{
'Key': 'Name',
'Value': 'First Instance'
}
],
}
],
}
]
So, right now it's working and I got the 'Value' that I want, but I would like to know if there are ways to simplify/optimize it ? I'm not good at list comprehension yet, so maybe this way ?
Thank you !

You can do it in one line using list comprehensions, but at the end it is similar to have nested loops:
tags = [tag['Value'] for res in ec2_list['Reservations'] for instances in res['Instances'] for tag in instances['Tags']]
What you get is a list with all the 'Values' like this one:
print(tags)
# ['Second Instance', 'First Instance']

How would I get my response to the Google Assistant via Dialogflow, Actions SDK, Python, etc

I am trying to get my reply to the Google assistant as a string to search it through a Google spreadsheet and return additional information about it. Specifically, I have a Google spreadsheet with two columns, one being for the name of a thing, and another for my rating of it. I'm using ngrok and Flask to get my script's response online. The two parts I'm struggling with are...
(1) Getting my requested thing into the program.
(2) Switching the url of the webhook and/or changing the webhook so that the system can output the final result.
I am a beginner so I'm probably misusing a few terms here but anyways my code is below. I put a bunch of underscores in the places where I thought sensitive information was.
action.json >>
"actions": [
{
"description": "Welcome Intent",
"name": "MAIN",
"fulfillment": {
"conversationName": "jdii ratings"
},
"intent": {
"name": "actions.intent.MAIN",
"trigger": {
"queryPatterns": [
"What did I rate $Test:text",
"Look up $Test:text on my ratings spreadsheet"
]
},
"parameters": [
{
"name": "text",
"type": "Test"
}
]
}
}
],
"conversations": {
"jdii ratings": {
"name": "jdii ratings",
"url": "<_________>",
"fulfillmentApiVersion": 2
}
}
}
program.py >>
import random
import types
import gspread
from flask import Flask
from google.cloud import dialogflow_v2
from oauth2client.service_account import ServiceAccountCredentials
app = Flask(__name__)
wanted_thingy = "Squirt"
def get_requested_thingy_rating(requested_thingy_name):
scopes = [
'https://www.googleapis.com/auth/spreadsheets',
'https://www.googleapis.com/auth/drive'
]
credentials = ServiceAccountCredentials.from_json_keyfile_name("_______.json", scopes)
file = gspread.authorize(credentials)
sheet = file.open("JDII Ratings API")
requested_thingy_rating_final = 0.0
if not isinstance(requested_thingy_name, types.NoneType):
sheet = sheet.sheet1
thingy_list = sheet.col_values(1)
ratings_list = sheet.col_values(2)
requested_thingy_rating = 0.0
if requested_thingy_name in thingy_list:
thingy_pos = thingy_list.index(requested_thingy_name)
requested_thingy_rating = ratings_list[thingy_pos]
requested_thingy_rating_final = str(requested_thingy_rating)
# split_string = ready_string.split('\'')
# requested_thingy_rating_final = split_string[1]
print(requested_thingy_rating_final)
return requested_thingy_rating_final
#app.route("/start", methods=['POST', 'GET', 'PUT'])
def main():
response = {
"expectUserResponse": True,
"expectedInputs": [
{
"inputPrompt": {
"richInitialPrompt": {
"items": [
{
"simpleResponse": {
"textToSpeech": "What would you like to look up?",
"displayText": "What would you like to look up?"
}
}
]
}
},
"possibleIntents": [
{
"intent": "actions.intent.TEXT"
}
]
}
]
}
response_text = json.dumps(response)
return response_text
#app.route("/webhook", methods=['POST', 'GET', 'PUT'])
def main2():
global wanted_thingy
wanted_thingy_final = str(wanted_thingy)
wanted_thingy_rating = get_requested_thingy_rating(wanted_thingy_final)
print(wanted_thingy)
string = f"You rated {wanted_thingy} a {wanted_thingy_rating} out of ten."
response2 = {
"expectUserResponse": False,
"finalResponse": {
"richResponse": {
"items": [
{
"simpleResponse": {
'ssml': f'<speak>{string}</speak>'
}
}
]
}
}
}
response_text = json.dumps(response2)
return response_text
app.run()

Why am I getting TypeError string indices must be integers in this API call? (seeding Django DB)

I am trying to seed a django DB from an external API using this guide (https://medium.com/#chilinski.a/how-to-seed-a-django-api-with-data-from-an-external-api-b577b6e6ad54).
I have replicated the code accurately for my own project, I think, but get a TypeError when i run python manage.py seed and am not sure why. Here's the error message:
File "...nrel/nrel_app/management/commands/seed.py", line 15, in seed_nrel
utility_name = i["utility_name"],
TypeError: string indices must be integers
Here's my code:
import requests
from django.core.management.base import BaseCommand
from nrel_app.models import Nrel
def get_nrel():
url = 'https://developer.nrel.gov/api/utility_rates/v3.json?api_key=DEMO_KEY&lat=35.45&lon=-82.98'
r = requests.get(url, headers={'Content=Type': 'nrel_app/json'})
Nrels = r.json()
return Nrels
def seed_nrel():
for i in get_nrel():
Nrels = Nrel(
utility_name = i['utility_name'],
company_id = i['company_id'],
utility_info =i['utility_info'],
residential = i['residential'],
commercial = i['commercial'],
industrial = i['industrial'],
)
Nrels.save()
class Command(BaseCommand):
def handle(self, *args, **options):
seed_nrel()
print("Completed.")
Here's the json being requested from the nrel api:
{
"inputs": {
"lat": "35.45",
"lon": "-82.98"
},
"errors": [],
"warnings": [],
"version": "3.1.0",
"metadata": {
"sources": [
"Ventyx Research (2012)"
]
},
"outputs": {
"company_id": "8333|18642",
"utility_name": "Haywood Electric Member Corp|Tennessee Valley Authority",
"utility_info": [
{
"company_id": "8333",
"utility_name": "Haywood Electric Member Corp"
}, {
"company_id": "18642",
"utility_name": "Tennessee Valley Authority"
}
],
"commercial": 0.0977,
"industrial": 0.0862,
"residential": 0.123
}
}

I think you should remove the for cycle and get the result from the object returned by the get_nrel() method directly:
def seed_nrel():
i = get_nrel()['options']
Nrels = Nrel(
utility_name = i['utility_name'],
company_id = i['company_id'],
utility_info =i['utility_info'],
residential = i['residential'],
commercial = i['commercial'],
industrial = i['industrial'],
)
Nrels.save()

How yo APPEND a AWS S3 bucket Notification Configuration

Trying to append a new notification to a bucket. Couldn't find any example in the internet.
I need to have ObjectCreated events sent to SQS. I need to decide which queue the event is sent by the prefix. Thus, each notification would have a diferent queue and prefix over the same bucket.
The problem is that I cannot append new notification. I just override the previous notification configured in the bucket.
This is the code I have so far:
bucket_notifications_configuration = {
'QueueConfigurations': [{
'Events': ['s3:ObjectCreated:*'],
'Id': f"Notif_{queue_name}",
'QueueArn': queue.attributes['QueueArn'] ,
"Filter": {
"Key": {
"FilterRules": [
{
"Name": "suffix",
"Value": f"{prefix}"
}
]
}
}
}]
}
qpolicy = {
"Version": "2012-10-17",
"Id": f"{queue_arn}/SQSDefaultPolicy",
"Statement": [{
"Sid": f"allow bucket {bucket} to notify queue {queue_name}",
"Effect": "Allow",
"Principal": {"AWS": "*"},
"Action": "SQS:SendMessage",
"Resource": queue_arn,
"Condition": {
"ArnLike": {
"aws:SourceArn": f"arn:aws:s3:*:*:{bucket}"
}
}
}]
}
queue_attrs = queue.attributes
queue_attrs = {"Policy": json.dumps(qpolicy), }
queue.set_attributes(Attributes=queue_attrs)
logger.debug(f"queue ready with attributes: {queue.attributes}")
previous_notif = client.get_bucket_notification_configuration(Bucket=bucket)
previous_notif.pop('ResponseMetadata')
try:
print("apendado")
previous_notif['QueueConfigurations'].append(bucket_notifications_configuration['QueueConfigurations'][0])
print(f"apendado {previous_notif} ")
except KeyError:
previous_notif['QueueConfigurations'] = bucket_notifications_configuration['QueueConfigurations'][0]
print("cread")
client.put_bucket_notification_configuration(
Bucket=bucket,
NotificationConfiguration=bucket_notifications_configuration)
I make sure the notification id is diferent from any other, also I make sure the prefix is difefent.
This code overrides previous notification with the new one, instead of appending the new one.

I managed to get it working with the code below.
It takes the existing configurations, adds a new configuration, then saves it back to the bucket. The code assumes that there is an existing configuration.
import boto3
s3_client = boto3.client('s3', region_name = 'ap-southeast-2')
queue_name = 'queue2'
queue_arn = 'arn:aws:sqs:ap-southeast-2:123456789012:queue2'
bucket = 'my-bucket'
prefix = 'folder2/'
# Get the current notification configurations
response = s3_client.get_bucket_notification_configuration(Bucket=bucket)
configurations = response['QueueConfigurations']
# New configuration to add
new_configuration = {
'Id': f"Notif_{queue_name}",
'QueueArn': queue_arn,
'Events': [
's3:ObjectCreated:*',
],
'Filter': {
'Key': {
'FilterRules': [
{
'Name': 'prefix',
'Value': prefix
},
]
}
}
}
configurations.append(new_configuration)
# Save combined configurations
response = s3_client.put_bucket_notification_configuration(
Bucket = bucket,
NotificationConfiguration = {'QueueConfigurations' : configurations}
)

openstack cannot retrieve authentication token from API

I am trying to retrieve the authentication token from the API using requests library from python. Here is the attempt I have made so far:
def get_token():
data = {
"auth" : {
"identity" : {
"methods" : [ "password" ],
"password": {
"user" : {
"name" : OS_USERNAME,
"domain": { "name": "Default" },
"password": OS_PASSWORD
}
}
}
}
}
r = requests.post(
OS_AUTH_URL+'/auth/tokens',
headers = HEADERS,
json = data, # https://stackoverflow.com/questions/9733638
verify = False
)
print(r.content)
j = json.loads(r.content)
return j['token']['user']['id']
I get token in the response :
{
"token": {
"issued_at": "2018-07-03T11:03:59.000000Z",
"audit_ids": [
"Fg1ywtZBQ1CkigCw70If9g"
],
"methods": [
"password"
],
"expires_at": "2018-07-03T12:03:59.000000Z",
"user": {
"password_expires_at": null,
"domain": {
"id": "default",
"name": "Default"
},
"id": "e0dc5beb383a46b98dad824c5d76e719",
"name": "admin"
}
}
}
However, when I am reusing this token to get, for instance, the list of projects :
def get_tenantID():
r = requests.get(
OS_AUTH_URL+'/auth/projects',
headers = HEADERS,
verify = False
)
return r
r = get_token()
HEADERS['X-Auth-Project-Id'] = 'admin'
HEADERS['X-Auth-Token'] = r
r = get_tenantID()
I get this error as if I would not be authenticated:
<Response [401]>
{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
Googling around and using openstack token issue command showed me that usually, token are more like:
gAAAAABaCo1F1CIMVlsTBeuqYH8tm2qR29tbkmUL4vZuhCNPXJI39TQ2YzL6Twoj8fNcAyLe3WhCYW2O1YpbBF0G8mo4bt7Kf0IRsoDOoJ6uWa3RYyJ5SQNoB_5n8EnVXbKPxFYOZ_iFBnaVtL1_XDrGbwsrlDeyy8lZTDdLsqY52pUhFR-7Uow
which is not what I get with get_token.
What am I doing wrong?
Many thanks!

When you use the auth API directly, the token issued comes in the X-Subject-Token header.
Thus, to retrieve in your python example, you could access the response.headers dict like this:
token = r.headers['X-Subject-Token']
More info about authentication in the Keystone v3 docs

1. fetch authentication token as mentioned below:
r = requests.post(
OS_AUTH_URL+'/auth/tokens',
headers = HEADERS,
json = data,
verify = False
)
token = r.headers[X-Subject-Token]
2. Pass this token in header for further request:
{
'X-Auth-Token': token
}