I'm relatively new to python and I don't know all the mysteries of this language yet so I was wondering if there are any ways I can optimize this code.
I'm trying to list the name of my EC2 instances in an AWS lambda using boto3 and python.
Here's the code :
import json
import boto3
import botocore
import logging
# Create a logging message
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(name)s:%(message)s')
# Create EC2 resource
ec2 = boto3.client('ec2')
ec2_list = ec2.describe_instances()
def lambda_handler(event, context):
try:
for reservation in ec2_list['Reservations']:
for instance in reservation['Instances']:
for tag in instance['Tags']:
print(tag['Value'])
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
except botocore.exceptions.ClientError as e:
logger.debug(e)
raise e
I also tried that as seen in another post, but it didnt work, because reservation var is referenced before assignment - seems logic:
for reservation, instance, tag in itertools.product(ec2_list['Reservations'], reservation['Instances'], instance['Tags']):
print(tag['Value'])
And here is the thing I need to parse (I reduced it a lottttt for the post) :
[
{
'Groups': [],
'Instances': [
{
'Tags': [
{
'Key': 'Name',
'Value': 'Second Instance'
}
],
}
],
},
{
'Groups': [],
'Instances': [
{
'Tags': [
{
'Key': 'Name',
'Value': 'First Instance'
}
],
}
],
}
]
So, right now it's working and I got the 'Value' that I want, but I would like to know if there are ways to simplify/optimize it ? I'm not good at list comprehension yet, so maybe this way ?
Thank you !
You can do it in one line using list comprehensions, but at the end it is similar to have nested loops:
tags = [tag['Value'] for res in ec2_list['Reservations'] for instances in res['Instances'] for tag in instances['Tags']]
What you get is a list with all the 'Values' like this one:
print(tags)
# ['Second Instance', 'First Instance']
Related
I am trying to create a work-item using Python and requests library.
def test_create_work_item(work_items):
payload = {
'op': 'add',
'path': '/fields/System.Title',
'value': 'Sample bug'
}
pl = json.dumps(payload)
work_item = work_items.create(body=pl, type='bug')
assert work_item.status_code == 200
I am getting the below error for this :
{"$id":"1","innerException":null,"message":"You must pass a valid patch document in the body of the request.","typeName":"Microsoft.VisualStudio.Services.Common.VssPropertyValidationException,Microsoft.VisualStudio.Services.Common","typeKey":"VssPropertyValidationException","errorCode":0,"eventId":3000}
The same body works okay with Postman. So not sure what more is needed here to get it working.
I`m not familiar with Python.... Check this example: Create work item
The API uses an array of new fields:
[
{
"op": "add",
"path": "/fields/System.Title",
"from": null,
"value": "Sample task"
}
]
In your case, you use just one field in the request:
{
'op': 'add',
'path': '/fields/System.Title',
'value': 'Sample bug'
}
I'm working to implement the search_all_iam_policies() method in google-cloud-asset as follows:
from google.cloud import asset_v1
ASSET_CLIENT = asset_v1.AssetServiceClient()
response = ASSET_CLIENT.search_all_iam_policies(
scope='projects/my_project',
query='my.email#domain.com'
)
policies = []
for policy in response:
policies.append(policy)
return json.dumps({
'policies': policies
})
But cannot find a way to get JSON representation of policies nor policy. In this case 'response' is a google.cloud.asset_v1.services.asset_service.pagers.SearchAllIamPoliciesPager and each 'policy' is an google.cloud.asset_v1.types.assets.IamPolicySearchResult. I can print them to the console but need them in JSON format to send to another system.
Just to expand on Michaels answer.
When using that approach you "lose" some information namely the resource, project, asset_type and organization.
from google.cloud import asset_v1
from google.protobuf.json_format import MessageToJson
ASSET_CLIENT = asset_v1.AssetServiceClient()
response = ASSET_CLIENT.search_all_iam_policies(
scope='projects/my_project',
query='my.email#domain.com' # This field is optional
)
policies = []
for policy in response:
policies.append(
{
"resource": f"{policy.resource}",
"project": f"{policy.project}",
"bindings": json.loads(MessageToJson(policy.policy)).get('bindings'),
"asset_type": f"{policy.asset_type}",
"organization": f"{policy.organization}"
}
)
This will give you a list of dicts that look like the following:
{
'resource': '//some_resource',
'project': 'some_project',
'bindings': [
{
'role': 'some_role',
'members': [
'projectEditor:some_project',
'projectOwner:some_project'
]
},
{
'role': 'some_other_role',
'members': [
'projectViewer:some_project'
]
},
],
'asset_type': 'some_asset_type',
'organization': 'some_organization'
}
Found a way to decode the message like this:
from google.cloud import asset_v1
from google.protobuf.json_format import MessageToDict
ASSET_CLIENT = asset_v1.AssetServiceClient()
response = ASSET_CLIENT.search_all_iam_policies(
scope='projects/my_project',
query='my.email#domain.com'
)
policies = []
for policy in response:
policies.append(MessageToDict(policy.policy))
return json.dumps({
'policies': policies
})
I have a method where I build a table for multiple items for Google's DLP inspect API which can take either a ContentItem, or a table of values
Here is how the request is constructed:
def redact_text(text_list):
dlp = google.cloud.dlp.DlpServiceClient()
project = 'my-project'
parent = dlp.project_path(project)
items = build_item_table(text_list)
info_types = [{'name': 'EMAIL_ADDRESS'}, {'name': 'PHONE_NUMBER'}]
inspect_config = {
'min_likelihood': "LIKELIHOOD_UNSPECIFIED",
'include_quote': True,
'info_types': info_types
}
response = dlp.inspect_content(parent, inspect_config, items)
return response
def build_item_table(text_list):
rows = []
for item in text_list:
row = {"values": [{"stringValue": item}]}
rows.append(row)
table = {"table": {"headers": [{"name": "something"}], "rows": rows}}
return table
When I run this I get back the error ValueError: Protocol message Value has no "stringValue" field. Even though the this example and the docs say otherwise.
Is there something off in how I build the request?
Edit: Here's the output from build_item_table
{
'table':
{
'headers':
[
{'name': 'value'}
],
'rows':
[
{
'values':
[
{
'stringValue': 'My name is Jenny and my number is (555) 867-5309, you can also email me at anemail#gmail.com, another email you can reach me at is email#email.com. '
}
]
},
{
'values':
[
{
'stringValue': 'Jimbob Doe (555) 111-1233, that one place down the road some_email#yahoo.com'
}
]
}
]
}
}
Try string_value .... python uses the field names, not the type name.
I got the following error when trying to set a VPC's tags after creating it (once out of a few dozen tries. It works most of the time, but not always):
botocore.exceptions.ClientError: An error occurred (InvalidVpcID.NotFound) when calling the CreateTags operation: The vpc ID 'vpc-4240de24' does not exist
I checked afterwards, and the VPC vpc-4240de24 did exist, so CreateTags was called too early.
The error occurred in the following method:
def create_vpc(self, region, vpc_name):
"""create VPC in region and attach tags (threaded)"""
ec2_client = aws.ec2_client(region) # Custom method, essentially calls boto3.client('ec2')
vpc_id = ec2_client.create_vpc(
CidrBlock="172.31.0.0/16",
AmazonProvidedIpv6CidrBlock=False
)["Vpc"]["VpcId"]
# TODO: Attach tags on VPC creation when (if) it becomes supported
ec2_client.get_waiter("vpc_exists").wait(VpcIds=[vpc_id])
ec2_client.create_tags(
Resources=[vpc_id],
Tags=[
{
"Key": "Namespace",
"Value": config.NAMESPACE
},
{
"Key": "Name",
"Value": vpc_name
}
]
)
I do not understand how getting that error is even possible. Shouldn't the vpc_exists waiter return only when the VPC exists, and raise the WaiterError exception otherwise? I would set a sleep for 1 second after the waiter, but is there something I'm not doing correctly?
You can now tag the VPC on creation
response = client.create_vpc(
CidrBlock='string',
AmazonProvidedIpv6CidrBlock=True|False,
Ipv6Pool='string',
Ipv6CidrBlock='string',
DryRun=True|False,
InstanceTenancy='default'|'dedicated'|'host',
Ipv6CidrBlockNetworkBorderGroup='string',
TagSpecifications=[
{
'ResourceType': 'vpc',
'Tags': [
{
'Key': 'string',
'Value': 'string'
},
]
},
]
)
I have 3 slots (account, dollar_value, recipient_first) within my intent schema for an Alexa skill and I want to save whatever slots are provided by the speaker in the session Attributes.
I am using the following methods to set session attributes:
def create_dollar_value_attribute(dollar_value):
return {"dollar_value": dollar_value}
def create_account_attribute(account):
return {"account": account}
def create_recipient_first_attribute(recipient_first):
return {"recipient_first": recipient_first}
However, as you may guess, if I want to save more than one slot as data in sessionAttributes, the sessionAttributes is overwritten as in the following case:
session_attributes = {}
if session.get('attributes', {}) and "recipient_first" not in session.get('attributes', {}):
recipient_first = intent['slots']['recipient_first']['value']
session_attributes = create_recipient_first_attribute(recipient_first)
if session.get('attributes', {}) and "dollar_value" not in session.get('attributes', {}):
dollar_value = intent['slots']['dollar_value']['value']
session_attributes = create_dollar_value_attribute(dollar_value)
The JSON response from my lambda function for a speech input in which two slots (dollar_value and recipient_first) were provided is as follows (my guess is that the create_dollar_value_attribute method in the second if statement is overwriting the first):
{
"version": "1.0",
"response": {
"outputSpeech": {
"type": "PlainText",
"text": "Some text output"
},
"card": {
"content": "SessionSpeechlet - Some text output",
"title": "SessionSpeechlet - Send Money",
"type": "Simple"
},
"reprompt": {
"outputSpeech": {
"type": "PlainText"
}
},
"shouldEndSession": false
},
"sessionAttributes": {
"dollar_value": "30"
}
}
The correct response for sessionAttributes should be:
"sessionAttributes": {
"dollar_value": "30",
"recipient_first": "Some Name"
},
How do I create this response? Is there a better way to add values to sessionAttributes in the JSON response?
The easiest way to add sessionAttributes with Python in my opinion seems to be by using a dictionary. For example, if you want to store some of the slots for future in the session attributes:
session['attributes']['slotKey'] = intent['slots']['slotKey']['value']
Next, you can just pass it on to the build response method:
buildResponse(session['attributes'], buildSpeechletResponse(title, output, reprompt, should_end_session))
The implementation in this case:
def buildSpeechletResponse(title, output, reprompt_text, should_end_session):
return {
'outputSpeech': {
'type': 'PlainText',
'text': output
},
'card': {
'type': 'Simple',
'title': "SessionSpeechlet - " + title,
'content': "SessionSpeechlet - " + output
},
'reprompt': {
'outputSpeech': {
'type': 'PlainText',
'text': reprompt_text
}
},
'shouldEndSession': should_end_session
}
def buildResponse(session_attributes, speechlet_response):
return {
'version': '1.0',
'sessionAttributes': session_attributes,
'response': speechlet_response
}
This creates the sessionAttributes in the recommended way in the Lambda response JSON.
Also just adding a new sessionAttribute doesn't overwrite the last one if it doesn't exist. It will just create a new key-value pair.
Do note, that this may work well in the service simulator but may return a key attribute error when testing on an actual Amazon Echo. According to this post,
On Service Simulator, sessions starts with Session:{ ... Attributes:{}, ... }
When sessions start on the Echo, Session does not have an Attributes key at all.
The way I worked around this was to just manually create it in the lambda handler whenever a new session is created:
if event['session']['new']:
event['session']['attributes'] = {}
onSessionStarted( {'requestId': event['request']['requestId'] }, event['session'])
if event['request']['type'] == 'IntentRequest':
return onIntent(event['request'], event['session'])
First, you have to define the session_attributes.
session_attributes = {}
Then instead of using
session_attributes = create_recipient_first_attribute(recipient_first)
You should use
session_attributes.update(create_recipient_first_attribute(recipient_first)).
The problem you are facing is because you are reassigning the session_attributes. Instead of this, you should just update the session_attributes.
So your final code will become:
session_attributes = {}
if session.get('attributes', {}) and "recipient_first" not in session.get('attributes', {}):
recipient_first = intent['slots']['recipient_first']['value']
session_attributes.update(create_recipient_first_attribute(recipient_first))
if session.get('attributes', {}) and "dollar_value" not in session.get('attributes', {}):
dollar_value = intent['slots']['dollar_value']['value']
session_attributes.update(create_dollar_value_attribute(dollar_value))
The ASK SDK for Python provides an attribute manager, to manage request/session/persistence level attributes in the skill. You can look at the color picker sample, to see how to use these attributes in skill development.
Take a look at the below:
account = intent['slots']['account']['value']
dollar_value = intent['slots']['dollar_value']['value']
recipient_first = intent['slots']['recipient_first']['value']
# put your data in a dictionary
attributes = {
'account':account,
'dollar_value':dollar_value,
'recipient_first':recipient_first
}
Put the attributes dictionary in 'sessionAttributes' in your response. You should get it back in 'sessionAttributes' once Alexa replies to you.
Hope this helps.
The following code snippet will also prevent overwriting the session attributes:
session_attributes = session.get('attributes', {})
if "recipient_first" not in session_attributes:
session_attributes['recipient_first'] = intent['slots']['recipient_first']['value']
if "dollar_value" not in session_attributes:
session_attributes['dollar_value'] = = intent['slots']['dollar_value']['value']