We Keep Coding

Need help creating a VM in GCP using Python

I'm working on a project to automate the deployment of VMs in GCP using Python. I recently figured out how to create a custom image using Python and I'm now at the step where I need to create the VM. I have the example template from the Google documentation but I'm stuck on a particular method and don't understand the argument that it wants.
I can successfully get the image from the family, create and attached disk from the image, but when I get to create_instance function I'm unsure of how it wants me to reference the disk. disks: List[compute_v1.AttachedDisk]. I keep getting google.cloud.compute.v1.Instance.disks is not iterable when I try and specify the name or path.
Any help or guidance is appreciated.
import re
import sys
from typing import Any, List
import warnings
from google.api_core.extended_operation import ExtendedOperation
from google.cloud import compute_v1
def get_image_from_family(project: str, family: str) -> compute_v1.Image:
"""
Retrieve the newest image that is part of a given family in a project.
Args:
project: project ID or project number of the Cloud project you want to get image from.
family: name of the image family you want to get image from.
Returns:
An Image object.
"""
image_client = compute_v1.ImagesClient()
# List of public operating system (OS) images: https://cloud.google.com/compute/docs/images/os-details
newest_image = image_client.get_from_family(project=project, family=family)
return newest_image
def disk_from_image(
disk_type: str,
disk_size_gb: int,
boot: bool,
source_image: str,
auto_delete: bool = True,
) -> compute_v1.AttachedDisk:
"""
Create an AttachedDisk object to be used in VM instance creation. Uses an image as the
source for the new disk.
Args:
disk_type: the type of disk you want to create. This value uses the following format:
"zones/{zone}/diskTypes/(pd-standard|pd-ssd|pd-balanced|pd-extreme)".
For example: "zones/us-west3-b/diskTypes/pd-ssd"
disk_size_gb: size of the new disk in gigabytes
boot: boolean flag indicating whether this disk should be used as a boot disk of an instance
source_image: source image to use when creating this disk. You must have read access to this disk. This can be one
of the publicly available images or an image from one of your projects.
This value uses the following format: "projects/{project_name}/global/images/{image_name}"
auto_delete: boolean flag indicating whether this disk should be deleted with the VM that uses it
Returns:
AttachedDisk object configured to be created using the specified image.
"""
boot_disk = compute_v1.AttachedDisk()
initialize_params = compute_v1.AttachedDiskInitializeParams()
initialize_params.source_image = source_image
initialize_params.disk_size_gb = disk_size_gb
initialize_params.disk_type = disk_type
boot_disk.initialize_params = initialize_params
# Remember to set auto_delete to True if you want the disk to be deleted when you delete
# your VM instance.
boot_disk.auto_delete = auto_delete
boot_disk.boot = boot
return boot_disk
def wait_for_extended_operation(
operation: ExtendedOperation, verbose_name: str = "operation", timeout: int = 300
) -> Any:
"""
This method will wait for the extended (long-running) operation to
complete. If the operation is successful, it will return its result.
If the operation ends with an error, an exception will be raised.
If there were any warnings during the execution of the operation
they will be printed to sys.stderr.
Args:
operation: a long-running operation you want to wait on.
verbose_name: (optional) a more verbose name of the operation,
used only during error and warning reporting.
timeout: how long (in seconds) to wait for operation to finish.
If None, wait indefinitely.
Returns:
Whatever the operation.result() returns.
Raises:
This method will raise the exception received from `operation.exception()`
or RuntimeError if there is no exception set, but there is an `error_code`
set for the `operation`.
In case of an operation taking longer than `timeout` seconds to complete,
a `concurrent.futures.TimeoutError` will be raised.
"""
result = operation.result(timeout=timeout)
if operation.error_code:
print(
f"Error during {verbose_name}: [Code: {operation.error_code}]: {operation.error_message}",
file=sys.stderr,
flush=True,
)
print(f"Operation ID: {operation.name}", file=sys.stderr, flush=True)
raise operation.exception() or RuntimeError(operation.error_message)
if operation.warnings:
print(f"Warnings during {verbose_name}:\n", file=sys.stderr, flush=True)
for warning in operation.warnings:
print(f" - {warning.code}: {warning.message}", file=sys.stderr, flush=True)
return result
def create_instance(
project_id: str,
zone: str,
instance_name: str,
disks: List[compute_v1.AttachedDisk],
machine_type: str = "n1-standard-1",
network_link: str = "global/networks/default",
subnetwork_link: str = None,
internal_ip: str = None,
external_access: bool = False,
external_ipv4: str = None,
accelerators: List[compute_v1.AcceleratorConfig] = None,
preemptible: bool = False,
spot: bool = False,
instance_termination_action: str = "STOP",
custom_hostname: str = None,
delete_protection: bool = False,
) -> compute_v1.Instance:
"""
Send an instance creation request to the Compute Engine API and wait for it to complete.
Args:
project_id: project ID or project number of the Cloud project you want to use.
zone: name of the zone to create the instance in. For example: "us-west3-b"
instance_name: name of the new virtual machine (VM) instance.
disks: a list of compute_v1.AttachedDisk objects describing the disks
you want to attach to your new instance.
machine_type: machine type of the VM being created. This value uses the
following format: "zones/{zone}/machineTypes/{type_name}".
For example: "zones/europe-west3-c/machineTypes/f1-micro"
network_link: name of the network you want the new instance to use.
For example: "global/networks/default" represents the network
named "default", which is created automatically for each project.
subnetwork_link: name of the subnetwork you want the new instance to use.
This value uses the following format:
"regions/{region}/subnetworks/{subnetwork_name}"
internal_ip: internal IP address you want to assign to the new instance.
By default, a free address from the pool of available internal IP addresses of
used subnet will be used.
external_access: boolean flag indicating if the instance should have an external IPv4
address assigned.
external_ipv4: external IPv4 address to be assigned to this instance. If you specify
an external IP address, it must live in the same region as the zone of the instance.
This setting requires `external_access` to be set to True to work.
accelerators: a list of AcceleratorConfig objects describing the accelerators that will
be attached to the new instance.
preemptible: boolean value indicating if the new instance should be preemptible
or not. Preemptible VMs have been deprecated and you should now use Spot VMs.
spot: boolean value indicating if the new instance should be a Spot VM or not.
instance_termination_action: What action should be taken once a Spot VM is terminated.
Possible values: "STOP", "DELETE"
custom_hostname: Custom hostname of the new VM instance.
Custom hostnames must conform to RFC 1035 requirements for valid hostnames.
delete_protection: boolean value indicating if the new virtual machine should be
protected against deletion or not.
Returns:
Instance object.
"""
instance_client = compute_v1.InstancesClient()
# Use the network interface provided in the network_link argument.
network_interface = compute_v1.NetworkInterface()
network_interface.name = network_link
if subnetwork_link:
network_interface.subnetwork = subnetwork_link
if internal_ip:
network_interface.network_i_p = internal_ip
if external_access:
access = compute_v1.AccessConfig()
access.type_ = compute_v1.AccessConfig.Type.ONE_TO_ONE_NAT.name
access.name = "External NAT"
access.network_tier = access.NetworkTier.PREMIUM.name
if external_ipv4:
access.nat_i_p = external_ipv4
network_interface.access_configs = [access]
# Collect information into the Instance object.
instance = compute_v1.Instance()
instance.network_interfaces = [network_interface]
instance.name = instance_name
instance.disks = disks
if re.match(r"^zones/[a-z\d\-]+/machineTypes/[a-z\d\-]+$", machine_type):
instance.machine_type = machine_type
else:
instance.machine_type = f"zones/{zone}/machineTypes/{machine_type}"
if accelerators:
instance.guest_accelerators = accelerators
if preemptible:
# Set the preemptible setting
warnings.warn(
"Preemptible VMs are being replaced by Spot VMs.", DeprecationWarning
)
instance.scheduling = compute_v1.Scheduling()
instance.scheduling.preemptible = True
if spot:
# Set the Spot VM setting
instance.scheduling = compute_v1.Scheduling()
instance.scheduling.provisioning_model = (
compute_v1.Scheduling.ProvisioningModel.SPOT.name
)
instance.scheduling.instance_termination_action = instance_termination_action
if custom_hostname is not None:
# Set the custom hostname for the instance
instance.hostname = custom_hostname
if delete_protection:
# Set the delete protection bit
instance.deletion_protection = True
# Prepare the request to insert an instance.
request = compute_v1.InsertInstanceRequest()
request.zone = zone
request.project = project_id
request.instance_resource = instance
# Wait for the create operation to complete.
print(f"Creating the {instance_name} instance in {zone}...")
operation = instance_client.insert(request=request)
wait_for_extended_operation(operation, "instance creation")
print(f"Instance {instance_name} created.")
return instance_client.get(project=project_id, zone=zone, instance=instance_name)
def create_from_custom_image(
project_id: str, zone: str, instance_name: str, custom_image_link: str
) -> compute_v1.Instance:
"""
Create a new VM instance with custom image used as its boot disk.
Args:
project_id: project ID or project number of the Cloud project you want to use.
zone: name of the zone to create the instance in. For example: "us-west3-b"
instance_name: name of the new virtual machine (VM) instance.
custom_image_link: link to the custom image you want to use in the form of:
"projects/{project_name}/global/images/{image_name}"
Returns:
Instance object.
"""
disk_type = f"zones/{zone}/diskTypes/pd-standard"
disks = [disk_from_image(disk_type, 10, True, custom_image_link, True)]
instance = create_instance(project_id, zone, instance_name, disks)
return instance
create_instance('my_project', 'us-central1-a', 'testvm-01','?')

AWS CDK reference existing image on ECR

New to AWS CDK and I'm trying to create a load balanced fargate service with the construct ApplicationLoadBalancedFargateService.
I have an existing image on ECR that I would like to reference and use. I've found the ecs.ContainerImage.from_ecr_repository function, which I believe is what I should use in this case. However, this function takes an IRepository as a parameter and I cannot find anything under aws_ecr.IRepository or aws_ecr.Repository to reference a pre-existing image. These constructs all seem to be for making a new repository.
Anyone know what I should be using to get the IRepository object for an existing repo? Is this just not typically done this way?
Code is below. Thanks in Advance.
from aws_cdk import (
# Duration,
Stack,
# aws_sqs as sqs,
)
from constructs import Construct
from aws_cdk import (aws_ec2 as ec2, aws_ecs as ecs,
aws_ecs_patterns as ecs_patterns,
aws_route53,aws_certificatemanager,
aws_ecr)
class NewStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
_repo = aws_ecr.Repository(self, 'id1', repository_uri = repo_uri)
vpc = ec2.Vpc(self, "applications", max_azs=3) # default is all AZs in region
cluster = ecs.Cluster(self, "id2", vpc=vpc)
hosted_zone = aws_route53.HostedZone.from_lookup(self,
'id3',
domain_name = 'domain'
)
certificate = aws_certificatemanager.Certificate.from_certificate_arn(self,
id4,
'cert_arn'
)
image = ecs.ContainerImage.from_ecr_repository(self, _repo)
ecs_patterns.ApplicationLoadBalancedFargateService(self, "id5",
cluster=cluster, # Required
cpu=512, # Default is 256
desired_count=2, # Default is 1
task_image_options=ecs_patterns.ApplicationLoadBalancedTaskImageOptions(
image = image,
container_port=8000),
memory_limit_mib=2048, # Default is 512
public_load_balancer=True,
domain_name = 'domain_name',
domain_zone = hosted_zone,
certificate = certificate,
redirect_http = True)

You are looking for from_repository_attributes() to create an instance of IRepository from an existing ECR repository.

Create instanceprofile with CDK and Python

I am new to cdk and trying to create an instance profile with CDK+Python with the following code. I have already created the Role (gitLabRunner-glue) successfully thru CDK and wanting to use it with the intance profile. However, when i run the following code, i get an error gitLabRunner-glue already exists
Can somebody please explain what am i missing ?
from aws_cdk import core as cdk
from aws_cdk import aws_glue as glue
from aws_cdk import aws_ec2 as _ec2
from aws_cdk import aws_iam as _iam
class Ec2InstanceProfile(cdk.Stack):
def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
# role = _iam.Role(self, "instanceprofilerole", role_name="gitLabRunner-glue",
# assumed_by=_iam.ServicePrincipal(service='ec2.amazonaws.com'))
ec2gitLabRunnerinstanceprofile = _iam.CfnInstanceProfile(
self,
"ec2gitLabRunnerinstanceprofile",
instance_profile_name="ec2-gitLabRunner-glue",
roles=["gitLabRunner-glue"] # also tried with this[role.role_name]
)

Does your AWS account already have a role with that name in it?
the Cfn Functions in cdk represent constructs and services that have not been fully hooked into all that is CDK. As such, they often don't do things that others would - where as a CloudFormation Template for the instance profile may just hook into the existing role, the coding in the back of this cfn function may go ahead and create a role item in the template output.
if you do a cdk synth, look in your cdk.out directory, find your cloudformation template, then do a search for gitLabRunner-glue - you may find there is a AWS::IAM::ROLE being created, indicating when CloudFormation attempts to run based of the template created by cdk it tries to create a new resource and it cant.
You have a couple options to try:
As you tried, uncomment the role again and use role.role_name but name the role something else or, as CDK recommends, don't include a name and let it name it for you
search your aws account for the role and delete it
If you absolutely cannot delete the existing role or cannot create a new one with a new name, then import the role, using (based off your imports)
role = _iam.Role.from_role_arn(self, "ImportedGlueRole", role_arn="arn:aws:of:the:role", add_grants_to_resources=True)
be wary a bit of the add_grants_to_resources - if its not your role to mess with cdk can make changes if you make that true and that could cause issues elsewhere - but if its not true, then you have to update the Role itself in the aws console (or cli) to accept your resources as able to assume it.

I made it work like this, not the desired model though, but given the limitations of cdk, i couldn't find any other way.
from aws_cdk import core as cdk
from aws_cdk import aws_glue as glue
from aws_cdk import aws_ec2 as _ec2
from aws_cdk import aws_iam as _iam
class Ec2InstanceProfile(cdk.Stack):
def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
boundary = _iam.ManagedPolicy.from_managed_policy_arn(self, "Boundary",
"arn:aws:iam::${ACCOUNT_ID}:policy/app-perm-boundary")
# role = _iam.Role(self, "instanceprofilerole", role_name="gitLabRunner-glue",
# assumed_by=_iam.ServicePrincipal(service='ec2.amazonaws.com'))
ec2_gitlabrunner_glue = _iam.Role(
self, 'ec2-gitlabrunner-glue',
role_name='gitLabRunner-glue',
description='glue service role to be attached to the runner',
# inline_policies=[write_to_s3_policy],
assumed_by=_iam.ServicePrincipal('ec2.amazonaws.com'),
permissions_boundary=boundary
)
ec2gitLabRunnerinstanceprofile = _iam.CfnInstanceProfile(
self,
"gitLabRunnerinstanceprofile",
instance_profile_name="gitLabRunner-glue",
roles=["gitLabRunner-glue"]
)

connection not getting created to the new dbname in mongoengine

Using python Mongoengine I am trying to create databases and add documents to different databases. Here's how I am trying to do it :
from mongoengine import *
class MongoDataAccessObject():
# method to connect to the database and initialize the tables etc.
def __init__(self, my_env, helperObj):
print "initializing db for the environment ", my_env
self.con = None
self.dbName = my_env
self.helper_obj = helperObj
try:
self.con = connect(db=self.dbName)
except Exception as e:
print e
def addRecord(self, document_object):
document_object.save()
Now, I pass the names of different databases that I want created while creating the object of the above class, and add the documents like this :
for my_env in list_of_envs:
dao = MongoDataAccessObject(my_env, helper_object)
dao.addRecord(myDocument)
Now there are 2 questions here:
For some reason all my documents keep getting added to the same DB (the first one being passed while MongoDataAccessObject object creation. I would assume that when I am creating a new object every time, while passing a different db name each time, a new connection should get created to the new db passed and documents should get added to the db which is currently connected to.
To verify if I am actually connected to the DB in question or not, I could not find a method like get_database_name() on the connection object. Is there a way to verify if I am getting connected to the DB name being passed or not ?
Ok did some more research and found this:
https://github.com/MongoEngine/mongoengine/issues/605
Tried it out like this in iptyhon:
from mongoengine import *
import datetime
class Page(Document):
title = StringField(max_length=200, required=True)
date_modified = DateTimeField(default=datetime.datetime.now)
def switch(model, db):
model._meta['db_alias'] = db
# must set _collection to none so it is re-evaluated
model._collection = None
return model
register_connection('default', name='testing')
register_connection('mycon', name='db1')
page = Page(title="Test Page")
page = switch(page, 'mycon')
page.save()
This works and creates a db named db1 and stores the document there.
Now I do this again:
register_connection('mycon2', name='db2')
page = Page(title="Test Page")
page = switch(page, 'mycon2')
page.save()
Contrary to my expectation this time db2 was not created (checked from both mongo client and from Robomongo), however the document was saved successfully. Wonder where exactly did the document get saved then ??
So to figure that out repeated the above exercise with a small change as below:
register_connection('mycon2', name='db2')
page = Page(title="Test Page")
page = switch(page, 'mycon2')
x = page.save()
# did a dir(x) and found that there is _get_db, so tried it out as below
x._get_db()
and the output was :
Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, read_preference=Primary()), u'db2')
which I guess means that the document got saved in a database named db2. But where on earth is this db2 ???? Why can't I see it through either mongo client or even Robomongo etc. ?

Finally the only way I could find to achieve the above was through the context manager with statement provided by MongoEngine, which also is well documented here : http://docs.mongoengine.org/guide/connecting.html#switch-database
The way I did it in my code above is something like this :
In the first call to the db, a default db needs to be created, which should have the alias as default. Only then can another alias be created, else Mongoengine throws an error saying that no default db found.
So to do this, when the very first db object is created, a False flag is sent to the __init__of MongoDataAccessObject
So to do this, the MongoDataAccessObject was also changed to something like this:
class MongoDataAccessObject():
# method to connect to the database and initialize the tables etc.
def __init__(self, my_env, helperObj, is_default_db_set):
print "initializing db for the environment ", my_env
self.con = None
self.dbName = my_env
self.helper_obj = helperObj
self.db_alias = my_env
self.is_default_db_set = is_default_db_set
try:
# all of this part with the is_default_db_set and register_connection() is needed because MongoEngine does not
# provide a simple way of changing dbs or switching db connections. The only way to do it is through the switch_db()
# context manager they provide (which has been used in the addRecord() below)
if not self.is_default_db_set:
self.con = connect(db=self.dbName, alias='default')
else:
# register_connection(alias_name, db_name)
register_connection(self.db_alias, self.dbName)
except Exception as e:
print e
And the addRecord() was also modified as :
def addRecord(self, document_object):
with switch_db(model_object, self.db_alias) as model_object:
document_object = model_object()
document_object.save()
And this part above:
for my_env in list_of_envs:
dao = MongoDataAccessObject(my_env, helper_object)
dao.addRecord(myDocument)
was also modified as:
for my_env in list_of_envs:
dao = MongoDataAccessObject(my_env,helper_object,mongo_default_db_flag)
dao.addRecord(myDocument)
And this seemed to do the job for me.

HIVE via Python hive_utils/hive_service: logging in as a particular user

Currently I can successfully connect and run queries from Python on HIVE using hive_utils doing the following:
import hive_utils
query = """ select * from table
where partition = x
"""
conn = hive_utils.HiveClient(server=x, port=10000,db='default')
a = conn.execute(query)
a = list(a)
Queries that include conditional statements however (and that work over HUE) such as:
query = """ select * from table
where partition = x
and app_id = y
"""
have returned this error:
HiveServerException: errorCode=1, message='Query returned non-zero
code:1' cause: FAILED: Execution Error return code 1 from
org.apache.hadoop.hive.ql.exec.MapRedTask SQLState=’08S01’
Since i am not sending any kind of user information when i establish a connection, I suspect the error is due to the type of permissions available to whichever user is being set.
How do I identify myself as a particular user?

Please refer to the following:
Re: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.MapRedTask
Most relevant section is:
Go to this link :
http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_Hive.html
and add
hadoop-0.20-core.jar
hive/lib/hive-exec-0.7.1.jar
hive/lib/hive-jdbc-0.7.1.jar
hive/lib/hive-metastore-0.7.1.jar
hive/lib/hive-service-0.7.1.jar
hive/lib/libfb303.jar
lib/commons-logging-1.0.4.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
to the class path of your project , add this jars from the lib of hadoop
and hive, and try the code. and also add the path of hadoop, hive, and
hbase(if your are using) lib folder path to the project class path, like
you have added the jars.