I am trying to get specific values for a describe instance call. So for example, if I want to get the 'Hypervisor' value or the Ebs has 'DeleteOnTermintation' value from the output. Below is the current code I am currently using to make the call and iterate thru the dictionary output.
import boto3
import pprint
from datetime import datetime
import json
client = boto3.client('ec2')
filters = [{
'Name': 'tag:Name',
'Values': ['*']
}]
class DatetimeEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime):
return obj.strftime('%Y-%m-%dT%H:%M:%SZ')
elif isinstance(obj, date):
return obj.strftime('%Y-%m-%d')
# Let the base class default method raise the TypeError
return json.JSONEncoder.default(self, obj)
output = json.dumps((client.describe_instances(Filters=filters)), cls=DatetimeEncoder)
pprint.pprint(output)
for v in output:
print v['Hypervisor']
Getting this error:
TypeError: string indices must be integers, not str
Using the pprint to see all the values available from the output.
Here's how you could display the information via the AWS Command-Line Interface (CLI):
aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId, Hypervisor, NetworkInterfaces[0].Attachment.DeleteOnTermination]'
Here's some Python:
import boto3
client = boto3.client('ec2')
response = client.describe_instances()
for r in response['Reservations']:
for i in r['Instances']:
print i['InstanceId'], i['Hypervisor']
for b in i['BlockDeviceMappings']:
print b['Ebs']['DeleteOnTermination']
Here's John's answer but updated for Python3
import boto3
client = boto3.client('ec2')
response = client.describe_instances()
for r in response['Reservations']:
for i in r['Instances']:
print(i['InstanceId'], i['Hypervisor'])
for b in i['BlockDeviceMappings']:
print(b['Ebs']['DeleteOnTermination'])
I know I am kinda late to the party, but my 2 cents for readability is to use generator comprehension (python 3):
import boto3
client = boto3.client('ec2')
response = client.describe_instances()
block_mappings = (block_mapping
for reservation in response["Reservations"]
for instance in reservation["Instances"]
for block_mapping in instance["BlockDeviceMappings"])
for block_mapping in block_mappings:
print(block_mapping["Ebs"]["DeleteOnTermination"])
You can also use jmespath, the same query engine behind awscli --query flag, to get the nested results automatically:
import jmespath
import boto3
client = boto3.client('ec2')
response = client.describe_instances()
print(jmespath.search(
"Reservations[].Instances[].DeviceBlockMappings[].Ebs.DeleteOnTermination",
response
))
Or, in case you need more power, use pyjq. Its syntax is a little different from jmespath which is used in awscli, but it has more advantages over it. Let's say you want not only the DeviceBlockMappings but also to keep to which InstanceId it is related to. In jmespath you can't really do this, because there is no access to outer structures, just a single nested path. In pyjq you can do something like this:
import pyjq
import boto3
client = boto3.client('ec2')
response = client.describe_instances()
print(pyjq.all(
"{id: .Reservations[].Instances[].InstanceId, d:.Reservations[].Instances[].DeviceBlockMappings[]}",
response
))
This will yield a list of device block mappings with their corresponding InstanceId, kind of like a mongo's unwind operation:
{'id': string, d: {'Ebs': {'DeleteOnTermination': boolean}}}[]
print (jmespath.search("Reservations[].Instances[].[InstanceId, SubnetId, ImageId, PrivateIpAddress, Tags[*]]", response))
Related
I'm trying to update AWS security group with one inbound rules using lambda function Python 3.7. For example: I would like to add my public IP with 8443 Port in existing security group. I have followed below link to achieve.
Modifying ec2 security group using lambda function
When I used this code on Lambda function with Python 3.7, it's not working.
import boto3
import hashlib
import json
import copy
import urllib2
# ID of the security group we want to update
SECURITY_GROUP_ID = "sg-XXXX"
# Description of the security rule we want to replace
SECURITY_RULE_DESCR = "My Home IP"
def lambda_handler(event, context):
new_ip_address = list(event.values())[0]
result = update_security_group(new_ip_address)
return result
def update_security_group(new_ip_address):
client = boto3.client('ec2')
response = client.describe_security_groups(GroupIds=[SECURITY_GROUP_ID])
group = response['SecurityGroups'][0]
for permission in group['IpPermissions']:
new_permission = copy.deepcopy(permission)
ip_ranges = new_permission['IpRanges']
for ip_range in ip_ranges:
if ip_range['Description'] == 'My Home IP':
ip_range['CidrIp'] = "%s/32" % new_ip_address
client.revoke_security_group_ingress(GroupId=group['GroupId'], IpPermissions=
[permission])
client.authorize_security_group_ingress(GroupId=group['GroupId'], IpPermissions=
[new_permission])
return ""
I got response as "", when I tested this code on lambda function.
The revoke_security_group_ingress() and authorize_security_group_ingress() calls in boto3 includes a Return field in the response:
Returns true if the request succeeds; otherwise, returns an error.
However, your code does not seem to be storing a response or examining its contents.
Boto3 lib allows retrieving various metadata for AWS' services. Some of the methods use pagination marker to harvest entire data.
I have an issue with retrieving list of all DMS tasks under my account. How to pass retried marker to get the data in one run?
import boto3
import sys
dms = boto3.client('dms')
replication_tasks = dms.describe_replication_tasks(
MaxRecords=100,
Marker='',
WithoutSettings=True)
dictr = replication_tasks['ReplicationTasks'][0]
lst = replication_tasks['ReplicationTasks']
marker = replication_tasks['Marker']
if marker:
print(lst)
next_set = dms.describe_replication_tasks(MaxRecords=100, Marker=marker, WithoutSettings=True)
lst.append(next_set)
print(lst)
I would assume something like this.
import boto3
client = boto3.client('dms')
paginator = client.get_paginator('describe_replication_tasks')
response_iterator = paginator.paginate(
WithoutSettings=True,
)
for response in response_iterator:
print(response)
Based on AWS TimeStream SDK documentation for Python, I have the following code:
import boto3
def list_databases(self):
print("Listing databases")
try:
result = self.client.list_databases(MaxResults=5)
self._print_databases(result['Databases'])
next_token = result.get('NextToken', None)
while next_token:
result = self.client.list_databases(NextToken=next_token, MaxResults=5)
self._print_databases(result['Databases'])
next_token = result.get('NextToken', None)
except Exception as err:
print("List databases failed:", err)
session = boto3.Session(profile_name='superuser', region_name='eu-west-1')
query_client = session.client('timestream-query')
list_databases(query_client)
The authentication for my user superuser seems to work fine, but the boto3 session used for my query_client does not have a client object:
Listing databases
List databases failed: 'TimestreamQuery' object has no attribute 'client'
What am I missing?
This argument name in this method is likely a mistake:
# this name is discouraged in non-OO code:
def list_databases(self):
self is typically used in object oriented python code, which is not the case here.
Rename it as follows:
# this is better:
def list_databases(client):
Then, remove any mention of self in the body of the function, for example:
# this is incorrect:
result = self.client.list_databases(MaxResults=5)
should be:
# this should work:
result = client.list_databases(MaxResults=5)
How does one use put_bucket_policy()? It throws a MalformedPolicy error even when I try to pass in an existing valid policy:
import boto3
client = boto3.client('s3')
dict_policy = client.get_bucket_policy(Bucket = 'my_bucket')
str_policy = str(dict_policy)
response = client.put_bucket_policy(Bucket = 'my_bucket', Policy = str_policy)
* error message: *
botocore.exceptions.ClientError: An error occurred (MalformedPolicy) when calling the PutBucketPolicy operation: This policy contains invalid Json
That's because applying str to a dict doesn't turn it into a valid json, use json.dumps instead:
import boto3
import json
client = boto3.client('s3')
dict_policy = client.get_bucket_policy(Bucket = 'my_bucket')
str_policy = json.dumps(dict_policy)
response = client.put_bucket_policy(Bucket = 'my_bucket', Policy = str_policy)
Current boto3 API doesn't have a function to APPEND the bucket policy, whether add another items/elements/attributes. You need load and manipulate the JSON yourself. E.g. write script load the policy into a dict, append the "Statement" element list, then use the policy.put to replace the whole policy. Without the original statement id, user policy will be appended. HOWEVER, there is no way to tell whether later user policy will override rules of the earlier one.
For example
import boto3
s3_conn = boto3.resource('s3')
bucket_policy = s3_conn.BucketPolicy('bucket_name')
policy = s3_conn.get_bucket_policy(Bucket='bucket_name')
user_policy = { "Effect": "Allow",... }
new_policy = policy['Statement'].append(user_policy)
bucket_policy.put(Policy=new_policy)
The user don't need know the old policy in the process.
Trying to get started with Amazon SNS. API is very simple REST. I seem to be hung up on the signature part of the call. The API example is:
http://sns.us-east-1.amazonaws.com/
?Subject=My%20first%20message
&TopicArn=arn%3Aaws%3Asns%3Aus-east-1%3A698519295917%3AMy-Topic
&Message=Hello%20world%21
&Action=Publish
&SignatureVersion=2
&SignatureMethod=HmacSHA256
&Timestamp=2010-03-31T12%3A00%3A00.000Z
&AWSAccessKeyId=AKIAJHS4T6XPF7XIURNA
&Signature=9GZysQ4Jpnz%2BHklqM7VFTvEcjR2LIUtn6jW47054xxE%3D
I've been following the API docs for signatures, so I'm trying:
from time import strftime,gmtime,time
import urllib2
import hmac
import hashlib
import base64
import string
def publichSNSMsg(Subject,TopicArn,Message,AWSAccessKeyId,privatekey):
#http://docs.amazonwebservices.com/AWSSimpleQueueService/2008-01-01/SQSDeveloperGuide/
amzsnshost = 'sns.us-east-1.amazonaws.com'
values = {'Subject' : Subject,
'TopicArn' : TopicArn,
'Message' :Message,
'Timestamp' : strftime("%Y-%m-%dT%H:%M:%S.000Z", gmtime(time())),
'AWSAccessKeyId' : AWSAccessKeyId,
'Action' : 'Publish',
'SignatureVersion' : '2',
'SignatureMethod' : 'HmacSHA256',
}
amazquote=lambda v: urllib2.quote(v).replace('%7E','~')
cannqs=string.join(["%s=%s"%(amazquote(key),amazquote(values[key])) for key in sorted(values.keys(),key=str.lower)],'&')
string_to_sign=string.join(["GET",amzsnshost,"/",cannqs],'\n')
sig=base64.encodestring(hmac.new(privatekey,string_to_sign,hashlib.sha1).digest())
querystring = "%s&Signature=%s"%(cannqs,amazquote(sig))
url="http://%s/?%s"%(amzsnshost,querystring)
try:
return urllib2.urlopen(url).read()
except urllib2.HTTPError, exception:
return "Error %s (%s):\n%s"%(exception.code,exception.msg,exception.read())
And getting back:
<ErrorResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
<Error>
<Type>Sender</Type>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
</Error>
<RequestId>8d6e5a41-dafb-11df-ac33-f981dc4e6c50</RequestId>
</ErrorResponse>
Any ideas?
Aw, it was simple!
Keys in the query string were supposed to be byte-order sorted, not case-insensitive sorted (that was for version 1 signatures).
Slightly updated (and now correct) code is available on gist.
I found this example so useful that I rewrote it in C#. Since AWS does not have a WP7 library for SNS, perhaps this will be of help to someone: https://gist.github.com/2705156
I would suggest you to use boto3 for SNS. The documentation can be found at http://boto3.readthedocs.io/en/latest/reference/services/sns.html. The following code snippet can be used for fetching signature as JSON.
import boto3
from django.views.decorators.csrf import csrf_exempt
topic_arn = <your topic>
client = boto3.client('sns',region_name="us-west-2")
#Call this to publish
def sns_publish(arg1,arg2):
response = client.publish(
TopicArn=topic_arn,
Message=arg2,
Subject=arg2
)
#Function to subscribe to topic
#csrf_exempt
def sns_parse(request):
print request.body.decode('utf-8')
if request.method == "POST":
response = json.loads(request.body.decode('utf-8'))
type = response.get('Type')
if type == 'Subscription':
print(type)
return("Subscription URL: "+<Url obtained>)
elif type == 'Notification':
return HttpResponse(type.get('Signature'))
else:
return HttpResponse('Not a POST object')
This is just a template code though, but yeah, boto3 is indeed helpful.