Azure Function - Python - ServiceBus Output Binding - Setting Custom Properties - python

I have an Azure Function written in Python that has an Service Bus (Topic) output binding. The function is triggered by another queue, we process some files from a blobl storage and then put another message in a queue.
My function.json file looks like that:
{
"bindings": [
{
"type": "serviceBus",
"connection": "Omnibus_Input_Send_Servicebus",
"name": "outputMessage",
"queueName": "validation-output-queue",
"accessRights": "send",
"direction": "out"
}
],
"disabled": false
}
In my function, I can send a message to another queue like that:
with open(os.environ['outputMessage'], 'w') as output_message:
output_message.write('This is my output test message !')
It is working fine. Now I'd like to send a message to a topic. I've created a subscription with an SQLFilter and I need to set some custom properties to the BrokeredMessage.
From the azure sdk for python, I've found that I can add custom properties like that (I've installed the azure module using pip):
from azure.servicebus import Message
sent_msg = Message(b'This is the third message',
broker_properties={'Label': 'M3'},
custom_properties={'Priority': 'Medium',
'Customer': 'ABC'}
)
My new function.json file looks like that:
{
"bindings": [
{
"type": "serviceBus",
"connection": "Omnibus_Input_Send_Servicebus",
"name": "outputMessage",
"topicName": "validation-output-topic",
"accessRights": "send",
"direction": "out"
}
],
"disabled": false
}
And I've modify my function like that:
from azure.servicebus import Message
sent_msg = Message(b'This is the third message',
broker_properties={'Label': 'M3'},
custom_properties={'Priority': 'Medium',
'Customer': 'ABC'}
)
with open(os.environ['outputMessage'], 'w') as output_message:
output_message.write(sent_msg)
When I run the function, I get this exception:
TypeError: expected a string or other character buffer object
I tried to use the buffer and the memoryview function but still get another exception:
TypeError: cannot make memory view because object does not have the buffer interface
I am wondering if the actual binding supports BrokeredMessage and how to deal with it ?

The ServiceBus output binding for Python (and other script languages) only supports a simple string mapping, where the string you specify becomes the content of the BrokeredMessage created behind the scenes. To set any extended properties or do anything more sophisticated, you'll have to drop down to using the Azure Python SDK yourself in your function.

In the same situation, where I need to add user properties in the output service bus queue/topic, I used azure.servicebus.ServiceBusClient directly.
sb.Message class has a user_properties setter:
def main(
httpreq: func.HttpRequest,
context: func.Context ):
sbClient : sb.ServiceBusClient = sb.ServiceBusClient.from_connection_string( os.getenv("AzureWebJobsServiceBus") )
topicClient : sb.TopicClient = sbClient.get_topic('scoring-testtopic')
message = sb.Message( httpreq.get_body().decode( 'UTF-8' ))
message.user_properties = {
'#AzureWebJobsParentId' : context.invocation_id,
'Prom' : '31000001'
}
topicClient.send( message )

Related

how to restart instance group via python google cloud library

I am not able to find any code sample or relevant documentation on python library for google cloud
Want to restart managed instance groups all vms via cloud function.
To list instances I am using something like this
import googleapiclient.discovery
def list_instances(compute, project, zone):
result = compute.instances().list(project=project, zone=zone).execute()
return result['items'] if 'items' in result else None
in requirement file I have
google-api-python-client==2.31.0
google-auth==2.3.3
google-auth-httplib2==0.1.0
From command line this is possible via SDK ->
https://cloud.google.com/sdk/gcloud/reference/compute/instance-groups/managed/rolling-action/restart
gcloud compute instance-groups managed rolling-action restart NAME [--max-unavailable=MAX_UNAVAILABLE] [--region=REGION | --zone=ZONE] [GCLOUD_WIDE_FLAG …]
But in python I am not able to write any code.
This is an incomplete answer since the python docs are pretty unreadable to me.
Looking at the gcloud cli code (which I couldn't find an official repo for so I looked here),
the restart command is triggered by something called a "minimal action".
minimal_action = (client.messages.InstanceGroupManagerUpdatePolicy.
MinimalActionValueValuesEnum.RESTART)
In the Python docs, there's references to these fields in the applyUpdatesToInstances method.
So I think the relevant code is something similar to:
compute.instanceGroupManagers().applyUpdatesToInstances(
project=project,
zone=zone,
instanceGroupManager='NAME',
body={"allInstances": True, "minimalAction": "RESTART"},
)
There may or may not be a proper Python object for the body, the docs aren't clear.
And the result seems to be an Operation object of some kind, but I don't know if there's execute() method or not.
This is confusing, because gcloud compute instance-groups managed rolling-action is syntactic sugar that does two things:
It turns on Proactive updater, by setting appropriate UpdatePolicy on the InstanceGroupManager resource
And it changes version name on the same resource to trigger an update.
It is covered in the docs in https://cloud.google.com/compute/docs/instance-groups/rolling-out-updates-to-managed-instance-groups#performing_a_rolling_replace_or_restart
Compare the gcloud and API tabs to get the idea.
Unfortunately I am illiterate in Python, so I am not able to translate it into Python code :(.
Using the documentation that #Grzenio provided, use patch() method to restart the instance group. See patch documentation to check its parameters.
This could be written in python using the code below. I provided the required parameters project,zone,instanceGroupManager and body. The value of body is from the example in the documentation.
import googleapiclient.discovery
import json
project = 'your-project-id'
zone = 'us-central1-a' # the zone of your instance group
instanceGroupManager = 'instance-group-1' # instance group name
body = {
"updatePolicy": {
"minimalAction": "RESTART",
"type": "PROACTIVE"
},
"versions": [{
"instanceTemplate": "global/instanceTemplates/instance-template-1",
"name": "v2"
}]
}
compute = googleapiclient.discovery.build('compute', 'v1')
rolling_restart = compute.instanceGroupManagers().patch(
project=project,
zone=zone,
instanceGroupManager=instanceGroupManager,
body=body
)
restart_operation = rolling_restart.execute() # execute the request
print(json.dumps(restart_operation,indent=2))
This will return an operation object and the instance group should restart in the rolling fashion:
{
"id": "3206367254887659944",
"name": "operation-1638418246759-5d221f9977443-33811aed-eed3ee88",
"zone": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a",
"operationType": "patch",
"targetLink": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a/instanceGroupManagers/instance-group-1",
"targetId": "810482163278776898",
"status": "RUNNING",
"user": "serviceaccountused#your-project-id.iam.gserviceaccount.com",
"progress": 0,
"insertTime": "2021-12-01T20:10:47.654-08:00",
"startTime": "2021-12-01T20:10:47.670-08:00",
"selfLink": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a/operations/operation-1638418246759-5d221f9977443-33811aed-eed3ee88",
"kind": "compute#operation"
}

SQS to AWS Lambda Function with AWS Chalice and BOTO3

I am using AWS SQS to store information coming in from an external server and then sending it to a Lambda function to process it and dequeue the information.
The information that I am sending in is in the form of a JSON and is being used as a python dictionary.
def lambda_handler(event, context):
for record in event['Records']:
messageHandler(record)
return {
'statusCode': 200,
'body': json.dumps('Batch Processed')
}
Assuming that the code for the messageHandler is working and properly implemented, how do I catch the messages from the queue in their batches. This is all being deployed by AWS Chalice without the use of CLI.
I am well out of my depth right now and have no idea why this is not working when I deploy it but is working when I trigger a normal Lambda Function in the AWS Console through the SQS Send/Recieve Message feature. As far as I know the triggers are set up correctly and they should have no issue.
If you have any questions please let me know.
The event that you are processing will look something like this:
{
"Records": [
{
"messageId": "11d6ee51-4cc7-4302-9e22-7cd8afdaadf5",
"receiptHandle": "AQEBBX8nesZEXmkhsmZeyIE8iQAMig7qw...",
"body": "Test message.",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1573251510774",
"SequenceNumber": "18849496460467696128",
"MessageGroupId": "1",
"SenderId": "AIDAIO23YVJENQZJOL4VO",
"MessageDeduplicationId": "1",
"ApproximateFirstReceiveTimestamp": "1573251510774"
},
"messageAttributes": {},
"md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:fifo.fifo",
"awsRegion": "us-east-2"
}
]
}
where the "body" is your json encoded message. You'll want your message handler function to do something like this:
def message_handler(event):
message = json.loads(event["body"])
# do stuff...
The return value from the lambda is pretty pointless if it is being used as the event target from sqs.

How to retrieve a URL query string parameter from inside an AWS Lambda Python function?

How do you access URL querystring parameters from inside an AWS Lambda function served though an API Gateway?
I have both the API gateway + Lambda function setup so I can call it from a public URL. My Python function is simply:
def lambda_handler(event, context):
print('event:', event)
print('context:', context)
I've configured the API's GET "Method Request" handler to pass through the "abc" querystring parameter.
I've also configured the API's GET "Integration Request" handler to map "abc" from "method.request.querystring.abc".
However, when I access my URL, e.g. https://myapp.execute-api.us-east-1.amazonaws.com/prod/myfunc?abc=123, the only thing logged is:
event: {}
context: <bootstrap.LambdaContext object at 0x7fc7a6cb0850>
What am I doing wrong? Why isn't "abc" being passed through in the event dictionary?
Check Use Lambda Proxy integration in the Integration Request to have it pass all request details in the event.
I have a similar problem and I know how frustrating it is. Use this mapping template:
{
"method": "$context.httpMethod",
"body" : $input.json('$'),
"headers": {
#foreach($param in $input.params().header.keySet())
"$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
#end
},
"queryStringParameters": {
#foreach($param in $input.params().querystring.keySet())
"$param": "$util.escapeJavaScript($input.params().querystring.get($param))" #if($foreach.hasNext),#end
#end
},
"pathParameters": {
#foreach($param in $input.params().path.keySet())
"$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
#end
}
}
Then you should find your event looks like this:
{
"method":"GET",
"body":{
},
"headers":{
},
"queryParams":{
"id":"459463732",
"command":"join_session"
},
"pathParams":{
}
}
The context is used for other information, such as IP addresses and timeout settings.

Routing in Azure Functions using python

I know that I can use query parameters in Azure functions to get the values
"myfunction?p=one&p2=two"
I am referring to this question
How can I do Routing in Azure Functions?
However it only addresses C# and node.js, I want to get the values following flask style
/function/<name>/<id>
which I can directly access, how do I do it in python in Azure functions
I also referred this doc, which only talks about node.js and C#
https://github.com/Azure/azure-functions-host/wiki/Http-Functions
You can add "route" to the function.json file to change the path, for example:
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"post"
],
"route": "contact/{version}/certificate"
}
....
]
And in file __init__.py,
you can get version by version = req.route_params.get('version')
To be more dynamic like the path type of Python Flask route in azure functions you can customize the route to be like below.
Main Concept is in the route's *{restOfPath}
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "function",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
],
"route": "Function1/{*restOfPath}"
},
{
"type": "http",
"direction": "out",
"name": "$return"
}
]
}
and edit the host.json to remove the word api from the URL like below.
{
"http": {
"routePrefix": ""
}
}
so with above configuration you can have the Function to serve URLs like below
http://localhost:7071/Function1
http://localhost:7071/Function1/name
http://localhost:7071/Function1/name/id
http://localhost:7071/Function1/name/id/sub-name
http://localhost:7071/Function1/name/id/sub-name/sub-id
etc ...
Again to access it in init.py use it like below. You would get it as string
req.route_params.get("restOfPath")
It depends on if you are using Functions V1 or Functions V2. Note that neither is currently in General Availability: the Python language in V1 is considered experimental and will likely never go GA. The Python language worker for Functions V2 just went into private preview, and is currently in active development (see here). Overall, I highly recommend using Functions 2.0 for the best python support, and because it supports Python 3, while I believe that Functions V1 only supports Python 2.
With that said, the two versions both use the route parameter on the HTTP trigger function.json to configure the routes. To actually read the values, the two versions have very different approaches.
V1
You can set the custom route the same way you do for Node and C# in the function.json, by setting the "route" field of your HTTP Trigger binding. The route parameters are available as strings on the environment variables REQ_PARAMS_{route_param_name_as_upper_case}. So for your example, the route parameter values for name and id would be REQ_PARAMS_NAME and REQ_PARAMS_ID respectively.
V2
Again, you set the custom route in the function.json file the same way you do in all other languages. The route parameters are attached on the http request object as the property route_params, with the type of Mapping[str, str]
def main(req):
name = req.route_params.get('name')
return f'Hello, {name}!'

Execute Lambda from AWS SSM Automation with Parameters

I currently have a lambda function that updates a DynamoDB table with a value passed as a parameter. I am able to run the following within the Lambda console with a test parameter set to "TEST":
import boto3
import json
def lambda_handler(event, context):
# TODO implement
update_ami(event)
def update_ami(ami_id):
#DO STUFF
I am attempting to call this from an SSM Automation built from the following JSON document:
{
"description":"Test Execute Lambda Function.",
"schemaVersion":"0.3",
"assumeRole":"MYARN",
"parameters":{},
"mainSteps":[
{
"name": "invokeMyLambdaFunction",
"action": "aws:invokeLambdaFunction",
"maxAttempts": 3,
"timeoutSeconds": 120,
"onFailure": "Abort",
"inputs": {
"FunctionName": "MyLambdaFunction",
"Payload": "TESTER"
}
}
]
}
Executing this automation results in the following error:
Automation Step Execution fails when it is invoking the lambda function. Get Exception from Invoke API of lambda Service. Exception Message from Invoke API: [Could not parse request body into json: Unrecognized token 'TESTER': was expecting ('true', 'false' or 'null')
I have also tried passing the Payload input as a JSON object instead of a string, and adjusted my lambda method accordingly:
JSON Automation:
...
"inputs": {
"FunctionName": "MyLambdaFunction",
"Payload": {
"ami_id": "AMI-TESTER"
}
}
...
Lambda Python:
def lambda_handler(event, context):
# TODO implement
update_ami(event['ami-id'])
This results in the following error coming from the Automation Document editor within the SSM console:
Input {ami_id=TESTER} is of type class java.util.LinkedHashMap, but expected type is String.
So in a nutshell... How do I pass a single string from an Automation document to a Lambda Function?
The error which is shown below looks to be issue with payload not passed as string:
Input {ami_id=TESTER} is of type class java.util.LinkedHashMap, but expected type is String
Please try to use escape character before double quotes.
"inputs": {
"FunctionName": "MyLambdaFunction",
"Payload": "{
\"ami_id\": \"AMI-TESTER\"
}"
}
AWS has provided proper syntax, if you see this Url
{
"name":"updateSsmParam",
"action":"aws:invokeLambdaFunction",
"timeoutSeconds":1200,
"maxAttempts":1,
"onFailure":"Abort",
"inputs":{
"FunctionName":"Automation-UpdateSsmParam",
"Payload":"{\"parameterName\":\"latestAmi\", \"parameterValue\":\"{{createImage.ImageId}}\"}"
}
}

Categories