We Keep Coding

How to avoid nested loops in python parsing aws

I'm relatively new to python and I don't know all the mysteries of this language yet so I was wondering if there are any ways I can optimize this code.
I'm trying to list the name of my EC2 instances in an AWS lambda using boto3 and python.
Here's the code :
import json
import boto3
import botocore
import logging
# Create a logging message
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(name)s:%(message)s')
# Create EC2 resource
ec2 = boto3.client('ec2')
ec2_list = ec2.describe_instances()
def lambda_handler(event, context):
try:
for reservation in ec2_list['Reservations']:
for instance in reservation['Instances']:
for tag in instance['Tags']:
print(tag['Value'])
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
except botocore.exceptions.ClientError as e:
logger.debug(e)
raise e
I also tried that as seen in another post, but it didnt work, because reservation var is referenced before assignment - seems logic:
for reservation, instance, tag in itertools.product(ec2_list['Reservations'], reservation['Instances'], instance['Tags']):
print(tag['Value'])
And here is the thing I need to parse (I reduced it a lottttt for the post) :
[
{
'Groups': [],
'Instances': [
{
'Tags': [
{
'Key': 'Name',
'Value': 'Second Instance'
}
],
}
],
},
{
'Groups': [],
'Instances': [
{
'Tags': [
{
'Key': 'Name',
'Value': 'First Instance'
}
],
}
],
}
]
So, right now it's working and I got the 'Value' that I want, but I would like to know if there are ways to simplify/optimize it ? I'm not good at list comprehension yet, so maybe this way ?
Thank you !

You can do it in one line using list comprehensions, but at the end it is similar to have nested loops:
tags = [tag['Value'] for res in ec2_list['Reservations'] for instances in res['Instances'] for tag in instances['Tags']]
What you get is a list with all the 'Values' like this one:
print(tags)
# ['Second Instance', 'First Instance']

python script to create the cloud function

We have written python script to create the cloud function , the trigger is Https. We need to invoke fetch the output of the function , So for that we are using the environment variables but some how that is not getting stored ?
def generate_config(context):
""" Entry point for the deployment resources. """
name = context.properties.get('name', context.env['name'])
project_id = context.properties.get('project', context.env['project'])
region = context.properties['region']
resources = []
resources.append(
{
'name': 'createfunction',
'type': 'gcp-types/cloudfunctions-v1:projects.locations.functions',
'properties':
{
'function': "licensevalidation",
'parent': 'projects//locations/',
'sourceArchiveUrl': 'gs://path',
'entryPoint':'handler',
'httpsTrigger': {"url": "https://.cloudfunctions.net/licensevalidation","securityLevel": "SECURE_ALWAYS"},
'timeout': '60s',
'serviceAccountEmail' : '.iam.gserviceaccount.com',
'availableMemoryMb': 256,
'runtime': 'python37' ,
'environmentvaiable' :
}
}
)
call ={
'type': 'gcp-types/cloudfunctions-v1:cloudfunctions.projects.locations.functions.call',
'name': 'call',
'properties':
{
'name':'/licensevalidation',
'data': '{""}'
},
'metadata': {
'dependsOn': ['createfunction']
}
}
resources.append(call)
return{
'resources': resources,
'outputs':
[
{
'name': 'installationtoken',
'value': 'os.environ.get(environment_variable)'
},
]
}

AWS Lambda function for AWS Metric using Autoscaling groups

I am in the midst of coding a lambda function which will create an alarm based upon some disk metrics. The code so far looks like this:
import collections
from datetime import datetime
import calendar
def lambda_handler(event, context):
client = boto3.client('cloudwatch')
alarm = client.put_metric_alarm(
AlarmName='Disk Monitor',
MetricName='disk_used_percent',
Namespace='CWAgent',
Statistic='Maximum',
ComparisonOperator='GreaterThanOrEqualToThreshold',
Threshold=60.0,
Period=10,
EvaluationPeriods=3,
Dimensions=[
{
'Name': 'InstanceId',
'Value': '{instance_id}'
},
{
'Name': 'AutoScalingGroupName',
'Value': '{instance_id}'
},
{
'Name': 'fstype',
'Value': 'xfs'
},
{
'Name': 'path',
'Value': '/'
}
],
Unit='Percent',
ActionsEnabled=True)
As seen, {instance_id} is a variable because the idea is that this will be used for every instance. However, I am wondering how I would code the same for AutoScalingGroupName because I require this to be a variable also. I know that that the below pulls out the AutoScalingGroupName for me, but how would I add that to the above block in terms of syntax, is my problem:
aws autoscaling describe-auto-scaling-instances --output text --query "AutoScalingInstances[?InstanceId == '<instance_dets>'].{AutoScalingGroupName:AutoScalingGroupName}"
For example, would I add a block beginning as below:
def lambda_handler(event, context):
client = boto3.client('autoscaling')
And if so, how would I then code what is needed in terms of syntax to get the 'Value': '{AutoScalingGroupName}' by which I mean a variable to hold the ASG?

describe_auto_scaling_instances takes InstanceIds as a parameter. So if you know your instance_id you can find its asg as follows:
client = boto3.client('autoscaling')
response = client.describe_auto_scaling_instances(
InstanceIds=[instance_id])
asg_name = ''
if response['AutoScalingInstances']:
asg_name = response['AutoScalingInstances'][0]['AutoScalingGroupName']
print(asg_name)

Python Google Analytics Active users list

I have a lot of potential users for my website (not open to the public).
I do have a Google Analytics account and everything is working well.
I don't want to iterate through all potential users because calling for each individual user will take a very long time (I have about 1200 users).
Instead, I want a list of only active users in the given time period.
Surely this must be possible
(Simple problem, I am happy to answer any questions as I know this is a very brief question I am asking)
EDITED:
I am working in python and need to write code to achieve this

If you're looking for a list of user ids that you can use with the user activity API, the analytics API has a dimension called 'ga:clientId' that you can call and then filter using the standard parameters - there's a list of options of what you can filter on here:
https://developers.google.com/analytics/devguides/reporting/core/v4/rest/v4/reports/batchGet#reportrequest
Depending on how you are describing 'active users', below is an example calling the REST API from python:
import requests
import json
credentials = #{ 'your credentials as a dict' }
r = requests.post("https://www.googleapis.com/oauth2/v4/token", data = {
"client_id": credentials["client_id"],
"client_secret": credentials["client_secret"],
"refresh_token": credentials["refresh_token"],
"grant_type": "refresh_token"
}
)
access_token =json.loads(r.text)
body = {
"reportRequests": [
{
'viewId': # "your ga view ID",
'pageSize': 100000,
"includeEmptyRows": True,
"samplingLevel": "LARGE",
'dateRanges': [
{
'startDate': "7DaysAgo",
'endDate': "yesterday"
}
],
'metrics': [
{
'expression': "ga:sessions"
}
],
'filtersExpression': "ga:sessions>2",
'dimensions': [
{
'name': "ga:clientId"
}
]
}
]
}
resp = requests.post("https://analyticsreporting.googleapis.com/v4/reports:batchGet",
json=body,
headers = {"Authorization" : "Bearer " + access_token["access_token"]}
)
resp = resp.json()
print(json.dumps(resp, indent = 4))
clientIds = [ x["dimensions"][0] for x in resp["reports"][0]["data"]["rows"] ]
print(clientIds)

To build on the answer above, you need to use a combination of the above plus the useractivity.list method.
I have written a full blog post on it https://medium.com/#alrowe/how-to-pull-out-the-user-explorer-report-with-python-useractivity-search-369bc5052093
Once you have used the above to get a list of client ids, you then need to iterate through those.
My 2 api calls look like this:
return analytics.reports().batchGet(
body = {
"reportRequests": [
{
'viewId': VIEW_ID,
'pageSize': 100000,
'includeEmptyRows': True,
'samplingLevel': 'LARGE',
'dateRanges': [
{
'startDate': '30DaysAgo',
'endDate': 'yesterday'
}
],
'metrics': [
{
'expression': 'ga:sessions'
}
],
'filtersExpression': 'ga:sessions>2',
'dimensions': [
{
'name': "ga:clientId"
}
]
}
]
}
).execute()
and then
def get_client_list_report(analytics,client_id):
return analytics.userActivity().search(
body = {
'user': {
'type': 'CLIENT_ID',
'userId': client_id
},
'dateRange':
{
'startDate': '30DaysAgo',
'endDate': 'yesterday'
},
'viewId': VIEW_ID,
'pageSize': 100000,
}
).execute()

What "operations" do I use to get keyword search volume from the googleads api? (TargetingIdeaService)

I am trying to get a search volume metric from the Google Ads API. I am running into trouble when I using the "SearchVolumeSearchParameter" argument. This argument requires an "operation" field and the documentation does not do a great job on explaining what these operations can be. Preferably, I would like the script to return a list of keywords and their respective search volumes for the previous month.
adwords_client = adwords.AdWordsClient.LoadFromStorage()
targeting_idea_service = adwords_client.GetService(
'TargetingIdeaService', version='v201809')
selector = {
'ideaType': 'KEYWORD',
'requestType': 'STATS'
}
selector['requestedAttributeTypes'] = [
'KEYWORD_TEXT',
'SEARCH_VOLUME',
# 'TARGETED_MONTHLY_SEARCHES',
]
offset = 0
PAGE_SIZE = 500
selector['paging'] = {
'startIndex': str(offset),
'numberResults': str(PAGE_SIZE)
}
selector['searchParameters'] = [{
'xsi_type': 'SearchVolumeSearchParameter',
'operation': []
}]
page = targeting_idea_service.get(selector)

You use maximum, minimum like so:
selector['searchParameters'] = [
{
'xsi_type': 'RelatedToQuerySearchParameter',
'queries': search_keywords
},
{
'xsi_type': 'LocationSearchParameter',
'locations': [
{'id': location_id}
]
},
{
'xsi_type': 'SearchVolumeSearchParameter',
'operation': [
{'minimum': 100}
]
}
]