We Keep Coding

Pulumi inputs outpus and resource dependency Python GCP

I'm having some trouble in trying to understand how to pass an output of a resource as an input to another resource, so they have a dependency and the order at the creation time works properly.
Scenario:
Resource B has a dependency from Resource A.
I was trying to pass to resource B something like these
opts = ResourceOptions(depends_on=[ResourceA])
But for some reason, it acts as that parameter wasn't there and keeps creating Resource B before creating Resource A, therefore throwing an error.
If I execute pulumi up a second time, as Resource A exists, Resource B gets created.
I noticed that you could also pass an output as an input of another resource, and because of this, Pulumi understands that there is a relationship and makes it so automatically
https://www.pulumi.com/docs/intro/concepts/inputs-outputs/
But I can't get my head around it in how to pass that, so, any help regarding this would be appreciate it.
I also used the following explanation regarding how to use ResourceOptions, which I think that I'm using it correctly as the code above, but still no case
How to control resource creation order in Pulumi
Thanks in advance.

#mrthopson,
Let me try to explain using one of the public examples. I took it from this Pulumi example:
https://github.com/pulumi/examples/blob/master/aws-ts-eks/index.ts
// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("vpc", { numberOfAvailabilityZones: 2 });
// Create the EKS cluster itself and a deployment of the Kubernetes dashboard.
const cluster = new eks.Cluster("cluster", {
vpcId: vpc.id,
subnetIds: vpc.publicSubnetIds,
instanceType: "t2.medium",
desiredCapacity: 2,
minSize: 1,
maxSize: 2,
});
The example first creates a VPC in AWS. The VPC contains a number of different networks and the identifiers of these networks are exposed as outputs. When we create the EKS cluster, we pass the ids of the public subnets (output vpc.publicSubnetIds) as an input to the cluster (input: subnetIds).
That is the only thing you need to do to have a dependency from the EKS cluster on the VPC. When running Pulumi, the engine will find out it first needs to create the VPC and only after that it can create the EKS cluster.
Ringo

Fetch Azure Managed Identity from within Function

I am using Azure Managed Identity feature for my python Azure Functions App
and would like to be able to fetch currently assigned Client ID from within the Function App itself.
Search through documentation and azure-identity python sources did not give result I would expect.
Maybe I could:
Query Azure Instance Metadata Service myself to get this ID. (not really happy with this option)
Provision it as env variable during ARM deployment stage/ or by hands later on. (seems good and efficient, but not sure what is the best practice here)
UPDATE
Managed o get it working with ARM template and env variable
Deploys FunctionApp with System Identity
Provisions System Identity as env variable of this same FunctionApp
Idea is to use Microsoft.Resources/deployments subtemplate to update Function App configuration with:
{
"name": "AZURE_CLIENT_ID",
"value": "[reference(resourceId('Microsoft.Web/sites', variables('appName')), '2019-08-01', 'full').identity.principalId]"
},

The simplest option is to go to the identity tab for your Functions app, and turn on "System assigned managed identity".
You can then get the access token without having to provide the client_id, since the token request simply picks the system assigned identity if there is one for the Function app.
If you are using "user assigned managed identity", then you need to provide the client_id: either through env or directly in your code.
You may already be aware, but just an additional note: that you also need to make sure you have given access to your managed identity for the resource you are accessing, for example: going to the Azure resource your Function app needs to access and assigning an appropriate role for your managed identity.
your option 1 (query Azure Instance Metadata Service), is only available on VMs.
UPDATE
Since you need the client_id for other purposes, you may also consider reading it from the response to your request for the access token: client_id is one of the parameters in the JSON token returned to you along with the access token, and its value is the client_id of the managed identity you used (in your case, the system-assigned managed identity)
Here is a sample token response to illustrate this:
{
access_token: <...>,
resource: <...>,
token_type: 'Bearer',
client_id: <client_id of the managed identity used to get this token>
}

How to verify the status of an EC2 instance launched from a newly created AMI?

How to check if an EC2 instance is up and running/one particular service is up or down using boto3. The instance is created from a newly created AMI.

You can uses the following Instance:
ec2 = boto3.resource('ec2')
instance = ec2.Instance('instance_id')
print(instance.image_id)
Compare instance.image_id to the id of image you expect.
To check if instance is already running, use EC2.Waiter.InstanceRunning waiter.

To check if an EC2 instance is up or down:
import sys
import boto3
ec2 = boto3.client('ec2')
if sys.argv[1] == 'ON':
response = ec2.monitor_instances(InstanceIds=['INSTANCE_ID'])
else:
response = ec2.unmonitor_instances(InstanceIds=['INSTANCE_ID'])
print(response)
To check the health of an instance:
# Boto 2.x
for status in ec2_connection.get_all_instance_statuses():
print(status)
# Boto 3
for status in ec2.meta.client.describe_instance_status()['InstanceStatuses']:
print(status)
For more details refer to:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/migrationec2.html
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/ec2-example-managing-instances.html
Update: (To check if particular service in EC2 is running or not)
You can publish a custom metric to CloudWatch in the form of a "heart beat".
Have a small script running via cron on your server checking the process list to see whether my_service is running and if it is, make a put-metric-data call to CloudWatch.
The metric could be as simple as pushing the number "1" to your custom metric in CloudWatch.
Set up a CloudWatch alarm that triggers if the average for the metric falls below 1
Make the period of the alarm be >= the period that the cron runs e.g. cron runs every 5 minutes, make the alarm trigger if it sees the average is below 1 for two 5 minute periods.
Make sure you also handle the situation in which the metric is not published (e. g. cron fails to run or whole machine dies). you would want to setup an alert in case the metric is missing. (see here:AWS Cloudwatch heartbeat alarm)
Be aware that the custom metric will add an additional cost of 50c to your AWS bill (not a big deal for one metric - but the equation changes drastically if you want to push hundred/thousands of metrics - i.e. good to know it's not free as one would expect)
You have to create a topic in SNS (Simple Notification Service) service in AWS. There you have to define your email under SNS on which you will receive the notification.
References:
Use cloudwatch to determine if linux service is running
AWS Cloudwatch Heartbeat Alarm

While you ask specifically about boto3, there is a way to ensure some service is in ready state more AWS natively.
Move creation of your instance to CloudFormation template and define CreationPolicy, like described in AWS Blog post:
https://aws.amazon.com/blogs/devops/use-a-creationpolicy-to-wait-for-on-instance-configurations/
TL;DR: when the instance is launched, it will run a script which will check if whatever service you need is working, and then signal CloudFormation "all systems go". If it does not signal in time, instance is marked as CREATE_FAILED and deleted.

Communicating with azure container using server-less function

I have created a python serverless function in azure that gets executed when a new file is uploaded to azure blob (BlobTrigger). The function extracts certain properties of the file and saves it in the DB. As the next step, I want this function copy and process the same file inside a container instance running in ACS. The result of processing should be returned back to the same azure function.
This is a hypothetical architecture that I am currently brainstorming on. I wanted to know if this is feasible. Can you provide me some pointers on how I can achieve this.
I dont see any ContainerTrigger kind of functionality that can allow me to trigger the container and process my next steps.
I have tried utilizing the code examples mentioned here but they have are not really performing the tasks that I need: https://github.com/Azure-Samples/aci-docs-sample-python/blob/master/src/aci_docs_sample.py

Based on the comments above you can consider.
Azure Container Instance
Deploy your container in ACI (Azure Container Instance) and expose HTTP end point from container , just like any web url. Trigger Azure Function using blob storage trigger and then pass your blob file URL to the exposed http end point to your container. Process the file there and return the response back to azure function just like normal http request/response.
You can completely bypass azure function and can trigger your ACI (container instance) using logic apps , process the file and directly save in database.
When you are using Azure function make sure this is short lived process since Azure function will exit after certain time (default 5 mins). For long processing you may have to consider azure durable functions.
Following url can help you understand better.
https://github.com/Azure-Samples/aci-event-driven-worker-queue

AWS Lambda hangs on completing lifecycle action

I use aws lambda to perform custom actions as Auto Scaling terminates instances. It looks like this
def scaledown_handler(event, context):
# customs actions
client = boto3.client('autoscaling')
response = client.complete_lifecycle_action(LifecycleHookName=event['detail']['LifecycleHookName'],
LifecycleActionToken=event['detail']['LifecycleActionToken'],
AutoScalingGroupName=event['detail']['AutoScalingGroupName'],
LifecycleActionResult='CONTINUE',
InstanceId=event['detail']['EC2InstanceId'])
The problem is that the function just hangs on client.complete_lifecycle_action() and finishes by timeout without any response and my ec2 instances are always "Waiting for Terminate Lifecycle Action".
aws autoscaling complete-lifecycle-action in aws CLI works fine, but i need to be done this from AWS lambda. How can I find out why does complete_lifecycle_action() hang without a response?

If you don't have a NAT gateway in your VPC then the Lambda function won't have access to anything outside the VPC. The AWS API exists outside your VPC, so the Lambda function is getting a network timeout trying to access it.
You have to add a NAT Gateway to your VPC in order for Lambda functions (and other things in your VPC that don't have a public IP) to access anything outside the VPC.

You need to use put_lifecycle_hook() API.
http://www.callumpember.com/auto-scaling-lifecycle-hooks/
On this link, you can get the complete python script for executing the custom actions before terminating the instance.