using AWS Boto3 Resource service i'm getting a list of all my ec2 instance:
ec2_instances = ec2.instances.all()
every time i'm doing something using my `ec2_instance' variable - it seams to reload the whole instance list..
Is there a way to just pull the list once and then work on it (filter and such)
example of what I do:
getting the list and showing it as a menu after filtering some values:
In my Aws() class:
def load_ec2_instance(self, region):
"""
Load the EC2 instances a region
:param region:
:rtype: list
:return: a list of the instances in a region or None if there are no instances
"""
ec2 = self._get_resource("ec2", region)
ec2_instances = ec2.instances.all()
counter = collections.Counter(ec2_instances);
ec2_size = sum(counter.itervalues());
if ec2_size == 0:
return None
return ec2_instances
in my menu module:
instances = Aws.get_instance().load_ec2_instance(chosen_region)
show_environments_menu(instances)
def show_environments_menu(instances):
subprocess.call("clear")
print "Please choose the environment your instance is located in:"
environments = Aws.get_instance().get_environments_from_instances(instances)
for i, environment in enumerate(environments):
print "%d. %s" % (i + 1, environment)
def get_environments_from_instances(self, instances):
"""
Get all the environments available from instances lists
:param list instances: the list of instance
:rtype: list
:return: a list of the environments
"""
environments = []
for instance in instances:
tags = instance.tags
for tag in tags:
key = tag.get("Key")
if key == "Environment":
environment = tag.get("Value").strip()
if environment not in environments:
environments.append(environment)
return environments
It takes time depending on my internet connection, but I see that when i'm disconnecting my internet - it can't filter..
I only have 12 instances so the loop to filter them shouldn't take time at all..
Update:
I changed Aws() class to module and i'm using these two functions:
def load_ec2_instances(region):
"""
Load the EC2 instances a region
:param region:
:rtype: list
:return: a list of the instances in a region or None if there are no instances
"""
ec2 = _get_resource("ec2", region)
ec2_instances = ec2.instances.all()
counter = collections.Counter(ec2_instances);
ec2_size = sum(counter.itervalues());
if ec2_size == 0:
return None
return ec2_instances
def get_environments_from_instances(instances):
"""
Get all the environments available from instances lists
:param list instances: the list of instance
:rtype: list
:return: a list of the environments
"""
environments = []
for instance in instances:
tags = instance.tags
for tag in tags:
key = tag.get("Key")
if key == "Environment":
environment = tag.get("Value").strip()
if environment not in environments:
environments.append(environment)
return environments
For anyone still facing this issue
Try using the ec2 client rather than the resource.
Resources in boto3 are higher level and do more work for you whereas the client just gives you what you ask of it.
When you use the client you get python dictionary objects which you can manipulate in memory to your hearts content. See the example below.
You could edit this to use list comprehension and methods etc.
See EC2.Client.describe_instances for the documentation on the ec2 client method for describing ec2 instances.
import boto3
ec2 = boto3.client("ec2", region_name = 'us-west-2')
reservations = ec2.describe_instances()['Reservations']
instances = []
for reservation in reservations:
for instance in reservation['Instances']:
instances.append(instance)
environments = []
for instance in instances:
for tag in instance['Tags']:
if tag['Key'] == 'Environment':
environments.append(tag['Value'])
Related
I'm working on a project to automate the deployment of VMs in GCP using Python. I recently figured out how to create a custom image using Python and I'm now at the step where I need to create the VM. I have the example template from the Google documentation but I'm stuck on a particular method and don't understand the argument that it wants.
I can successfully get the image from the family, create and attached disk from the image, but when I get to create_instance function I'm unsure of how it wants me to reference the disk. disks: List[compute_v1.AttachedDisk]. I keep getting google.cloud.compute.v1.Instance.disks is not iterable when I try and specify the name or path.
Any help or guidance is appreciated.
import re
import sys
from typing import Any, List
import warnings
from google.api_core.extended_operation import ExtendedOperation
from google.cloud import compute_v1
def get_image_from_family(project: str, family: str) -> compute_v1.Image:
"""
Retrieve the newest image that is part of a given family in a project.
Args:
project: project ID or project number of the Cloud project you want to get image from.
family: name of the image family you want to get image from.
Returns:
An Image object.
"""
image_client = compute_v1.ImagesClient()
# List of public operating system (OS) images: https://cloud.google.com/compute/docs/images/os-details
newest_image = image_client.get_from_family(project=project, family=family)
return newest_image
def disk_from_image(
disk_type: str,
disk_size_gb: int,
boot: bool,
source_image: str,
auto_delete: bool = True,
) -> compute_v1.AttachedDisk:
"""
Create an AttachedDisk object to be used in VM instance creation. Uses an image as the
source for the new disk.
Args:
disk_type: the type of disk you want to create. This value uses the following format:
"zones/{zone}/diskTypes/(pd-standard|pd-ssd|pd-balanced|pd-extreme)".
For example: "zones/us-west3-b/diskTypes/pd-ssd"
disk_size_gb: size of the new disk in gigabytes
boot: boolean flag indicating whether this disk should be used as a boot disk of an instance
source_image: source image to use when creating this disk. You must have read access to this disk. This can be one
of the publicly available images or an image from one of your projects.
This value uses the following format: "projects/{project_name}/global/images/{image_name}"
auto_delete: boolean flag indicating whether this disk should be deleted with the VM that uses it
Returns:
AttachedDisk object configured to be created using the specified image.
"""
boot_disk = compute_v1.AttachedDisk()
initialize_params = compute_v1.AttachedDiskInitializeParams()
initialize_params.source_image = source_image
initialize_params.disk_size_gb = disk_size_gb
initialize_params.disk_type = disk_type
boot_disk.initialize_params = initialize_params
# Remember to set auto_delete to True if you want the disk to be deleted when you delete
# your VM instance.
boot_disk.auto_delete = auto_delete
boot_disk.boot = boot
return boot_disk
def wait_for_extended_operation(
operation: ExtendedOperation, verbose_name: str = "operation", timeout: int = 300
) -> Any:
"""
This method will wait for the extended (long-running) operation to
complete. If the operation is successful, it will return its result.
If the operation ends with an error, an exception will be raised.
If there were any warnings during the execution of the operation
they will be printed to sys.stderr.
Args:
operation: a long-running operation you want to wait on.
verbose_name: (optional) a more verbose name of the operation,
used only during error and warning reporting.
timeout: how long (in seconds) to wait for operation to finish.
If None, wait indefinitely.
Returns:
Whatever the operation.result() returns.
Raises:
This method will raise the exception received from `operation.exception()`
or RuntimeError if there is no exception set, but there is an `error_code`
set for the `operation`.
In case of an operation taking longer than `timeout` seconds to complete,
a `concurrent.futures.TimeoutError` will be raised.
"""
result = operation.result(timeout=timeout)
if operation.error_code:
print(
f"Error during {verbose_name}: [Code: {operation.error_code}]: {operation.error_message}",
file=sys.stderr,
flush=True,
)
print(f"Operation ID: {operation.name}", file=sys.stderr, flush=True)
raise operation.exception() or RuntimeError(operation.error_message)
if operation.warnings:
print(f"Warnings during {verbose_name}:\n", file=sys.stderr, flush=True)
for warning in operation.warnings:
print(f" - {warning.code}: {warning.message}", file=sys.stderr, flush=True)
return result
def create_instance(
project_id: str,
zone: str,
instance_name: str,
disks: List[compute_v1.AttachedDisk],
machine_type: str = "n1-standard-1",
network_link: str = "global/networks/default",
subnetwork_link: str = None,
internal_ip: str = None,
external_access: bool = False,
external_ipv4: str = None,
accelerators: List[compute_v1.AcceleratorConfig] = None,
preemptible: bool = False,
spot: bool = False,
instance_termination_action: str = "STOP",
custom_hostname: str = None,
delete_protection: bool = False,
) -> compute_v1.Instance:
"""
Send an instance creation request to the Compute Engine API and wait for it to complete.
Args:
project_id: project ID or project number of the Cloud project you want to use.
zone: name of the zone to create the instance in. For example: "us-west3-b"
instance_name: name of the new virtual machine (VM) instance.
disks: a list of compute_v1.AttachedDisk objects describing the disks
you want to attach to your new instance.
machine_type: machine type of the VM being created. This value uses the
following format: "zones/{zone}/machineTypes/{type_name}".
For example: "zones/europe-west3-c/machineTypes/f1-micro"
network_link: name of the network you want the new instance to use.
For example: "global/networks/default" represents the network
named "default", which is created automatically for each project.
subnetwork_link: name of the subnetwork you want the new instance to use.
This value uses the following format:
"regions/{region}/subnetworks/{subnetwork_name}"
internal_ip: internal IP address you want to assign to the new instance.
By default, a free address from the pool of available internal IP addresses of
used subnet will be used.
external_access: boolean flag indicating if the instance should have an external IPv4
address assigned.
external_ipv4: external IPv4 address to be assigned to this instance. If you specify
an external IP address, it must live in the same region as the zone of the instance.
This setting requires `external_access` to be set to True to work.
accelerators: a list of AcceleratorConfig objects describing the accelerators that will
be attached to the new instance.
preemptible: boolean value indicating if the new instance should be preemptible
or not. Preemptible VMs have been deprecated and you should now use Spot VMs.
spot: boolean value indicating if the new instance should be a Spot VM or not.
instance_termination_action: What action should be taken once a Spot VM is terminated.
Possible values: "STOP", "DELETE"
custom_hostname: Custom hostname of the new VM instance.
Custom hostnames must conform to RFC 1035 requirements for valid hostnames.
delete_protection: boolean value indicating if the new virtual machine should be
protected against deletion or not.
Returns:
Instance object.
"""
instance_client = compute_v1.InstancesClient()
# Use the network interface provided in the network_link argument.
network_interface = compute_v1.NetworkInterface()
network_interface.name = network_link
if subnetwork_link:
network_interface.subnetwork = subnetwork_link
if internal_ip:
network_interface.network_i_p = internal_ip
if external_access:
access = compute_v1.AccessConfig()
access.type_ = compute_v1.AccessConfig.Type.ONE_TO_ONE_NAT.name
access.name = "External NAT"
access.network_tier = access.NetworkTier.PREMIUM.name
if external_ipv4:
access.nat_i_p = external_ipv4
network_interface.access_configs = [access]
# Collect information into the Instance object.
instance = compute_v1.Instance()
instance.network_interfaces = [network_interface]
instance.name = instance_name
instance.disks = disks
if re.match(r"^zones/[a-z\d\-]+/machineTypes/[a-z\d\-]+$", machine_type):
instance.machine_type = machine_type
else:
instance.machine_type = f"zones/{zone}/machineTypes/{machine_type}"
if accelerators:
instance.guest_accelerators = accelerators
if preemptible:
# Set the preemptible setting
warnings.warn(
"Preemptible VMs are being replaced by Spot VMs.", DeprecationWarning
)
instance.scheduling = compute_v1.Scheduling()
instance.scheduling.preemptible = True
if spot:
# Set the Spot VM setting
instance.scheduling = compute_v1.Scheduling()
instance.scheduling.provisioning_model = (
compute_v1.Scheduling.ProvisioningModel.SPOT.name
)
instance.scheduling.instance_termination_action = instance_termination_action
if custom_hostname is not None:
# Set the custom hostname for the instance
instance.hostname = custom_hostname
if delete_protection:
# Set the delete protection bit
instance.deletion_protection = True
# Prepare the request to insert an instance.
request = compute_v1.InsertInstanceRequest()
request.zone = zone
request.project = project_id
request.instance_resource = instance
# Wait for the create operation to complete.
print(f"Creating the {instance_name} instance in {zone}...")
operation = instance_client.insert(request=request)
wait_for_extended_operation(operation, "instance creation")
print(f"Instance {instance_name} created.")
return instance_client.get(project=project_id, zone=zone, instance=instance_name)
def create_from_custom_image(
project_id: str, zone: str, instance_name: str, custom_image_link: str
) -> compute_v1.Instance:
"""
Create a new VM instance with custom image used as its boot disk.
Args:
project_id: project ID or project number of the Cloud project you want to use.
zone: name of the zone to create the instance in. For example: "us-west3-b"
instance_name: name of the new virtual machine (VM) instance.
custom_image_link: link to the custom image you want to use in the form of:
"projects/{project_name}/global/images/{image_name}"
Returns:
Instance object.
"""
disk_type = f"zones/{zone}/diskTypes/pd-standard"
disks = [disk_from_image(disk_type, 10, True, custom_image_link, True)]
instance = create_instance(project_id, zone, instance_name, disks)
return instance
create_instance('my_project', 'us-central1-a', 'testvm-01','?')
I'm relatively new to pytest and unit tests in general. Specifically I'm struggling to implement mocking, but I do understand the concept. I have a Class, let's call it MyClass. It has a constructor which takes a number of arguments used by other functions within MyClass.
I have a get_tables() function that I have to test and it relies on some arguments defined in the constructor. I need to mock the BigQuery connection and return a mocked list of tables. A small snippet of the script is below
from google.cloud import bigquery
from google.cloud import storage
import logging
class MyClass:
def __init__(self, project_id: str, pro_dataset_id: str, balancing_dataset_id: str, files: list,
key_file: str = None, run_local=True):
"""Creates a BigQuery client.
Args:
project_id: (string), name of project
pro_dataset_id: (string), name of production dataset
balancing_dataset_id: (string), name of balancing dataset
files: (list), list of tables
key_file: (string), path to the key file
"""
if key_file is None:
self.bq_client = bigquery.Client(project=project_id)
self.storage_client = storage.Client(project=project_id)
self.bucket = self.storage_client.get_bucket("{0}-my-bucket".format(project_id))
else:
self.bq_client = bigquery.Client.from_service_account_json(key_file)
self.project_id = project_id
self.run_local = run_local
self.pro_dataset_id = pro_dataset_id
self.balancing_dataset_id = balancing_dataset_id
def get_tables(self):
"""Gets full list of all tables in a BigQuery dataset.
Args:
Returns:
List of tables from a specified dataset
"""
full_table_list = []
dataset_ref = '{0}.{1}'.format(self.project_id, self.pro_dataset_id)
tables = self.bq_client.list_tables(dataset_ref) # Make an API request.
logging.info(f"Tables contained in '{dataset_ref}':")
for table in tables:
full_table_list.append(table.table_id)
logging.info(f"tables: {full_table_list}")
return full_table_list
This was my attempt at mocking the connection and response based on an amalgamation of articles I've read and stackoverflow answers on other questions including this one How can I mock requests and the response?
import pytest
from unittest import mock
from MyPackage import MyClass
class TestMyClass:
def mocked_list_tables():
tables = ['mock_table1', 'mock_table2', 'mock_table3', 'mock_table4']
return tables
#mock.patch('MyPackage.MyClass', side_effect=mocked_list_tables)
def test_get_tables(self):
m = MyClass()
assert m.get_tables() == ['mock_table1', 'mock_table2', 'mock_table3', 'mock_table4']
This is the error I get with the above unit test
TypeError: test_get_tables() takes 1 positional argument but 2 were given
How do I get this test to work correctly? Incase you're wondering, the arguments in the constructor are declared using argparse.ArgumentParser().add_argument()
Im trying to copy the specified ec2 tags to their respective volumes. the function is invoked when the instance state changed to 'running'. However I don't want the code to run on every instance--for the first version of the code, when it as invoked my a single instance, it ran on all instances. even those that were already tagged. I want to have it run only for the specific instances that are booting up.
with some changes, Im getting error: ec2.Instance' object is not iterable
im really new to python and not sure how to proceed. Any inputs from you bright minds?
----HERE IS MY LAMBDA CODE----
import boto3
import json
def lambda_handler(event, context):
# is_test = context.function_name == 'test' # this value is injected by SAM local
instance = boto3.resource('ec2').Instance(id=event["detail"]["instance-id"])
tags_to_use = ['Stack-Name', 'StackID']
for instance in instance:
tags = instance.tags
to_tag = [t for t in tags if t['Key'] in tags_to_use]
for vol in instance.volumes.all():
print(f"Tagging volume {vol.id} from instance {instance.id}")
vol.create_tags(Tags=to_tag)
I am trying to create 2 instances of linux on amazon ec2 there is one instance which I am starting in script
import sys
import boto3
instance_id = "i-03e7f6391a0f523ee"
action = 'ON'
ec2 = boto3.client('ec2')
if action == 'ON':
response = ec2.start_instances(InstanceIds=[instance_id], DryRun=False)
x2=boto3.resource('ec2')
instance=x2.Instance(instance_id)
foo=instance.wait_until_running('self',Filters=[{'Name':'instance-state-name','Values':['running']}])
print ("instance is now running")
else:
response = ec2.stop_instances(InstanceIds=[instance_id], DryRun=False)
print(response)
ec2 = boto3.resource('ec2', region_name="ap-south-1")
ami-ImageId="ami-00b6a8a2bd28daf19"
#creating instances
ec2.create_instances(ImageId=ami-ImageId,MinCount=1,MaxCount=2,KeyName="datastructutre key",InstanceType="t2.micro")
#using filter to retrive instance status
instances = ec2.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
for instance in instances:
print(instance.id, instance.instance_type)
#Checking health status of instances
for status in ec2.meta.client.describe_instance_status()['InstanceStatuses']:
print(status)
However this line
ami-ImageId="ami-00b6a8a2bd28daf19"
is giving me errors instead of double quotes I have also tried single quotes
ami-ImageId='ami-00b6a8a2bd28daf19'
but in both the cases error is same, the error is
Syntax error: can't assign to operator
What is the mistake in assigning ami-id in this way? It is normal string being assigned to a variable.
The problem is not in the assignment. It's in the variable name. The - in ami-ImageId is an operator. Use a different variable name e.g. ami_ImageId
The expression "ami-ImageId" is a substraction of ami and ImageId variables but not a correct single variable name. Use ami_ImageId correct name for example.
I'm trying to obtain tags from instances in my AWS account using Python's boto library.
While this snippet works correctly bringing all tags:
tags = e.get_all_tags()
for tag in tags:
print tag.name, tag.value
(e is an EC2 connection)
When I request tags from individual instances,
print vm.__dict__['tags']
or
print vm.tags
I'm getting an empty list (vm is actually an instance class).
The following code:
vm.__dict__['tags']['Name']
of course results in:
KeyError: 'Name'
My code was working until yesterday and suddenly I'm not able to get the tags from an instance.
Does anybody know whether there is a problem with AWS API?
You have to be sure that the 'Name' tag exists before accessing it. Try this:
import boto.ec2
conn=boto.ec2.connect_to_region("eu-west-1")
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
if 'Name' in inst.tags:
print "%s (%s) [%s]" % (inst.tags['Name'], inst.id, inst.state)
else:
print "%s [%s]" % (inst.id, inst.state)
will print:
i-4e444444 [stopped]
Amazon Linux (i-4e333333) [running]
Try something like this:
import boto.ec2
conn = boto.ec2.connect_to_region('us-west-2')
# Find a specific instance, returns a list of Reservation objects
reservations = conn.get_all_instances(instance_ids=['i-xxxxxxxx'])
# Find the Instance object inside the reservation
instance = reservations[0].instances[0]
print(instance.tags)
You should see all tags associated with instance i-xxxxxxxx printed out.
For boto3 you will need to do this.
import boto3
ec2 = boto3.resource('ec2')
vpc = ec2.Vpc('<your vpc id goes here>')
instance_iterator = vpc.instances.all()
for instance in instance_iterator:
for tag in instance.tags:
print('Found instance id: ' + instance.id + '\ntag: ' + tag)
It turned out to be an error in my code. I did not consider the case of having one instance without the tag 'Name'.
There was one instance without the tag "Name" and my code was trying to get this tag from every instance.
When I ran this piece of code in an instance without the tag 'Name',
vm.__dict__['tags']['Name']
I got: KeyError: 'Name'. vm is a AWS instance.
With the instances that actually had this tag set, I didn't have any problem.
Thank you for your help and sorry for asking when it was only my own mistake.