I am trying to get the list of all unused AMIs using boto3.
creating variables
import boto3
REGION = 'us-east-1'
OWNER_ID = 'XXXXXXXXXXXX'
ec2_client = boto3.client('ec2', REGION)
ec2_resource = boto3.resource('ec2', REGION)
get AMIs ID by instances
def get_used_amis_by_instances():
reservations = ec2_client.describe_instances(
Filters=[
{
'Name': 'owner-id',
'Values': [
OWNER_ID,
]
},
])['Reservations']
amis_used_list = []
for reservation in reservations:
ec2_instances = reservation['Instances']
for ec2 in ec2_instances:
ImageId = ec2['ImageId']
if ImageId not in amis_used_list:
amis_used_list = amis_used_list + [ImageId]
return amis_used_list
get the all AMIs ID
I just need the list of AMIs ID
def get_all_amis():
amis = ec2_resource.images.filter(Owners = [OWNER_ID])
all_amis = []
for ami in amis.all():
if ami.id not in all_amis:
all_amis = all_amis + [ami.id]
return all_amis
get unused AMIs.
Using the previous methods I have the all_amis and all_used_amis.
def get_unused_amis(all_amis, all_used_amis):
unused_amis_list = []
for ami_id in all_amis:
if ami_id not in all_used_amis:
if ami_id not in unused_amis_list:
unused_amis_list = unused_amis_list + [ami_id]
return unused_amis_list
Output the results
def deregister_amis(event, context):
all_amis = get_all_amis()
print("All AMIs: %d" % len(all_amis) )
all_used = get_used_amis_by_instances()
print("All used AMIs: %d" % len(all_used) )
all_unused = get_unused_amis(all_amis, all_used)
print("All unused AMIs: %d" % len(all_unused) )
My problem
The deregister_amis is returning
All AMIs: 201
All used AMIs: 102
All unused AMIs: 140
I expect 99 for all unused AMIs. I don't see where is my error. But I don't see if there is maybe an error with the used value of 102. The total is correct 201, but with the other two values maybe I am doing something wrong or missing something. Let me know if you are able to see where is my error because I can.
One AMI can be used by many instance. Using list, you might get same AMI added into the list(where multiple instance using there same AMI). Your list is alright if you want number of instances running, but not AMI.
You should use set() in the first place, not list. Just convert all your list to set(). In addition, using set, you can do subtraction, unison,etc. So you really don't need the redundant get_unused_amis() function.
It is so simple using the set
unused_ami = all_amis - all_used_amis
quick hack
def deregister_amis(event, context):
all_amis = set(get_all_amis())
print("All AMIs: %d" % len(all_amis) )
all_used = set(get_used_amis_by_instances())
print("All used AMIs: %d" % len(all_used) )
all_unused = all_amis - all_used
print("All unused AMIs: %d" % len(all_unused) )
After another view, I think using the quick hack is better. Retaining the list give you one extra advantage : You can check out how many AMI is used more than once.
Related
This is the same as this question, but I also want to limit the depth returned.
Currently, all answers return all the objects after the specified prefix. I want to see just what's in the current hierarchy level.
Current code that returns everything:
self._session = boto3.Session(
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
)
self._session.resource("s3")
bucket = self._s3.Bucket(bucket_name)
detections_contents = bucket.objects.filter(Prefix=prefix)
for object_summary in detections_contents:
print(object_summary.key)
How to see only the files and folders directly under prefix? How to go n levels deep?
I can parse everything locally, and this is clearly not what I am looking for here.
There is no definite way to do this using list objects without getting all the objects in the dir.
But there is a way using s3 select which uses sql query like format to get n levels deep to get the file content as well as to get object keys.
If you are fine with writing sql then use this.
reference doc
import boto3
import json
s3 = boto3.client('s3')
bucket_name = 'my-bucket'
prefix = 'my-directory/subdirectory/'
input_serialization = {
'CompressionType': 'NONE',
'JSON': {
'Type': 'LINES'
}
}
output_serialization = {
'JSON': {}
}
# Set the SQL expression to select the key field for all objects in the subdirectory
expression = 'SELECT s.key FROM S3Object s WHERE s.key LIKE \'' + prefix + '%\''
response = s3.select_object_content(
Bucket=bucket_name,
ExpressionType='SQL',
Expression=expression,
InputSerialization=input_serialization,
OutputSerialization=output_serialization
)
# The response will contain a Payload field with the selected data
payload = response['Payload']
for event in payload:
if 'Records' in event:
records = event['Records']['Payload']
data = json.loads(records.decode('utf-8'))
# The data will be a list of objects, each with a "key" field representing the file name
for item in data:
print(item['key'])
There is not built in way with the Boto3 or S3 APIs to do this. You'll need some version of processing each level and asking in turn for a list of objects at that level:
import boto3
s3 = boto3.client('s3')
max_depth = 2
paginator = s3.get_paginator('list_objects_v2')
# Track all prefixes to show with a list
common_prefixes = [(0, "")]
while len(common_prefixes) > 0:
# Pull out the next prefix to show
current_depth, current_prefix = common_prefixes.pop(0)
# Loop through all of the items using a paginator to handle common prefixes with more
# than a thousand items
for page in paginator.paginate(Bucket=bucket_name, Prefix=current_prefix, Delimiter='/'):
for cur in page.get("CommonPrefixes", []):
# Show each common prefix, here just use a format like AWS CLI does
print(" " * 27 + f"PRE {cur['Prefix']}")
if current_depth < max_depth:
# This is below the max depth we want to show, so
# add it to the list to be shown
common_prefixes.append((current_depth + 1, cur['Prefix']))
for cur in page.get("Contents", []):
# Show each item sharing this common prefix using a format like the AWS CLI
print(f"{cur['LastModified'].strftime('%Y-%m-%d %H:%M:%S')}{cur['Size']:11d} {cur['Key']}")
We are trying to use boto3 to query our AWS instances and get all security groups which have rules with no description. I have not been able to figure out how to retrieve the Rule description. I have included an image of the column I want to retrieve below.
I wanted to share the solution I came up with late in the evening, piecing together bits from other solutions - primarily from #Abdul Gill and hist ec2_sg_rules script.
import boto3
# Explicitly declaring variables here grants them global scope
cidr_block = ""
ip_protpcol = ""
from_port = ""
to_port = ""
from_source = ""
description = ""
sg_filter = [{'Name': 'group-name', 'Values': ['*ssh*']}]
print("%s,%s,%s" % ("Group-Name","Group-ID","CidrIp"))
ec2 = boto3.client('ec2' )
sgs = ec2.describe_security_groups(Filters=sg_filter)["SecurityGroups"]
for sg in sgs:
group_name = sg['GroupName']
group_id = sg['GroupId']
# print("%s,%s" % (group_name,group_id))
# InBound permissions ##########################################
inbound = sg['IpPermissions']
for rule in inbound:
#Is source/target an IP v4?
if len(rule['IpRanges']) > 0:
for ip_range in rule['IpRanges']:
cidr_block = ip_range['CidrIp']
if 'Description' not in ip_range:
if '10.' not in cidr_block:
print("%s,%s,%s" % (group_name, group_id, cidr_block))
print('\n')
Hopefully this helps someone else.
Currently, I'm making two calls to AWS ec2 using boto3 to fetch subnetIDs that start with tag name org-production-* and org-non-production-*. How can I combine these two functions in python and still be able to access the SubnetID's all_prod_subnets and all_non_prod_subnets ? I want to perhaps avoid code duplicatin make just one call to aws ec2, get all subnets and iterate them and filter the response based on tag name.
def get_all_production_subnets_from_accounts():
subnet = vpc_client.describe_subnets(
Filters=[{'Name': 'tag:Name', 'Values': ['org-production-*']}])['Subnets']
if len(subnet) > 0:
# print([s['SubnetId'] for s in subnet])
all_prod_subnets = [ s['SubnetId'] for s in subnet ]
print("[DEBUG]Queried Subnet ID's of Production are: {}".format(all_prod_subnets))
return all_prod_subnets
else:
return None
def get_all_nonproduction_subnets_from_acccounts():
subnet = vpc_client.describe_subnets(
Filters=[{'Name': 'tag:Name', 'Values': ['org-non-production-*']}])['Subnets']
if len(subnet) > 0:
# print([s['SubnetId'] for s in subnet])
all_non_prod_subnets = [ s['SubnetId'] for s in subnet ]
print("[DEBUG]Queried Subnet ID's of Non-Production are: {}".format(all_non_prod_subnets))
return all_non_prod_subnets
else:
return None
One way would be as follows:
def get_all_subnets_from_connectivity():
subnets_found = {}
# define subnet types of interest
subnets_found['org-production'] = []
subnets_found['org-non-production'] = []
results = vpc_client.describe_subnets()
for subnet in results['Subnets']:
if 'Tags' not in subnet:
continue
for tag in subnet['Tags']:
if tag['Key'] != 'Name': continue
for subnet_type in subnets_found:
if subnet_type in tag['Value']:
subnets_found[subnet_type].append(subnet['SubnetId'])
return subnets_found
all_subnets = get_all_subnets_from_connectivity()
print(all_subnets)
The example output:
{
'org-production': ['subnet-033bad31433b55e72', 'subnet-019879db91313d56a'],
'org-non-production': ['subnet-06e3bc20a73b55283']
}
I want to create a python script where I can pass arguments/inputs to specify instance type and later attach an extra EBS (if needed).
ec2 = boto3.resource('ec2','us-east-1')
hddSize = input('Enter HDD Size if you want extra space ')
instType = input('Enter the instance type ')
def createInstance():
ec2.create_instances(
ImageId=AMI,
InstanceType = instType,
SubnetId='subnet-31d3ad3',
DisableApiTermination=True,
SecurityGroupIds=['sg-sa4q36fc'],
KeyName='key'
)
return instanceID; ## I know this does nothing
def createEBS():
ebsVol = ec2.Volume(
id = instanceID,
volume_type = 'gp2',
size = hddSize
)
Now, can ec2.create_instances() return ID or do I have to do an iteration of reservations?
or do I do an ec2.create(instance_id) / return instance_id? The documentation isn't specifically clear here.
in boto3, create_instances returns a list so to get instance id that was created in the request, following works:
ec2_client = boto3.resource('ec2','us-east-1')
response = ec2_client.create_instances(ImageId='ami-12345', MinCount=1, MaxCount=1)
instance_id = response[0].instance_id
The docs state that the call to create_instances()
https://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances
Returns list(ec2.Instance). So you should be able to get the instance ID(s) from the 'id' property of the object(s) in the list.
you can the following
def createInstance():
instance = ec2.create_instances(
ImageId=AMI,
InstanceType = instType,
SubnetId='subnet-31d3ad3',
DisableApiTermination=True,
SecurityGroupIds=['sg-sa4q36fc'],
KeyName='key'
)
# return response
return instance.instance_id
actually create_instances returns an ec2.instance instance
I'm trying to create a Recurly account using their python API, and keep hitting the same error:
def upsert_recurly_account(office):
try:
account = recurly.Account.get(office.pk)
except recurly.errors.NotFoundError:
logger.warning("Creating new recurly account for pk %s" % office.pk)
account = recurly.Account(account_code=office.pk)
else:
logger.info("Recurly account %s already exists, we will update it")
account.email = office.manager.email
account.first_name = office.manager.first_name
account.last_name = office.manager.last_name
account.company_name = '%s - %s' % (office.legal_name, office.name)
account.vat_number = office.tax_id
account.tax_exempt = office.tax_exempt
billing_info = recurly.BillingInfo()
billing_info.first_name = office.manager.first_name
billing_info.last_name = office.manager.last_name
billing_info.address1 = office.address1
billing_info.address2 = office.address2
billing_info.city = office.city
billing_info.country = office.country
billing_info.zip = office.postal_code
billing_info.phone = office.phone
billing_info.vat_number = office.tax_id
account.billing_info = billing_info
account.save()
The error I'm getting is:
ValidationError:required: account.number is required
Manually adding
account.number = 1234
Doesn't solve the problem.
Any ideas?
Since you're creating a new Account with nested billing info you need to provide a valid credit card number in the billing_info hash like so:
billing_info.number = 123
There are other mandatory fields (like month and year) when creating a new account with nested billing info. Please read the docs
Account.number isn't a valid Recurly field - it looks like you need to be passing through account_code - see the sample at https://docs.recurly.com/api/accounts#create-account