I have simple lambda function that is located under following endpoint:
https://******.execute-api.eu-west-2.amazonaws.com/lambda/add?x=1&y=2
AWS Chalice was used for adding simple endpoints here.
#app.route('/{exp}', methods=['GET'])
def add(exp):
app.log.debug("Received GET request...")
request = app.current_request
app.log.debug(app.current_request.json_body)
x = request.query_params['x']
y = request.query_params['y']
if exp == 'add':
app.log.debug("Received ADD command...")
result = int(x) + int(y)
return {'add': result}
Basically, it checks if the path is equal to add and sums two values from query_params.
Now, I am trying to invoke this lambda in another lambda.
My question:
How I can pass the path and query_params to my original lambda function using boto3 lambda client?
What I have tried so far:
I added two lines to policy.json file that allow me to invoke original function.
I saw a lot of similar question on StackOverflow, but most of them pass payload as a json.
#app.route('/')
def index():
lambda_client = boto3.client('lambda')
invoke_response = lambda_client.invoke(
FunctionName="function-name",
InvocationType="RequestResponse"
)
app.log.debug(invoke_response['Payload'].read())
Thank you in advance!
Maybe you can add the next code to your add function, so it accepts payloads too:
#app.route('/{exp}', methods=['GET'])
def add(exp, *args, **kwargs):
if isinstance(exp, dict):
# exp is event here
request = exp.get('request', {'x': 0, 'y': 0})
exp = exp.get('exp', 'add')
I'm going to write a general example, and you can easily modify it to match your needs. In your case the data dictionary would have request and exp keys, and you need to find your lambda function's arn.
AWS Documentation Lambda.invoke
Let's assume from now on we have 2 Lambdas named "master" and "slave". master will call slave.
At the moment there are 3 types of invocations:
RequestResponse (Default): master calls and waits for slave response
Event: Async, master calls and forgets
DryRun: Do some verification before running
I keep with #1 RequestResponse:
Slave:
def lambda_handler(event, context):
result = {}
result['event'] = event
result['result'] = "It's ok"
return result
And its arn is something like arn:aws:lambda:us-east-1:xxxxxxxxxxxxxx:function:slave
In the example, slave is just an echo function
Now, the master needs the necessary role's permission to call it, and the arn or name. Then you can write something like this:
import boto3
from datetime import datetime
import json
client = boto3.client('lambda')
def lambda_handler(event, context):
arn = 'arn:aws:lambda:us-east-1:xxxxxxxxxxxxxx:function:slave'
data = {'my_dict': {'one': 1, 'two': 2}, 'my_list': [1,2,3], 'my_date': datetime.now().isoformat()}
response = client.invoke(FunctionName=arn,
InvocationType='RequestResponse',
Payload=json.dumps(data))
result = json.loads(response.get('Payload').read())
return result
Usually you would get arn with something like os.environ.get('slave_arn')
All data from/to lambdas must be JSON serializable.
Related
I am trying to add routes from a file and I don't know the actual arguments beforehand so I need to have a general function that handles arguments via **kwargs.
To add routes I am using add_api_route as below:
from fastapi import APIRouter
my_router = APIRouter()
def foo(xyz):
return {"Result": xyz}
my_router.add_api_route('/foo/{xyz}', endpoint=foo)
Above works fine.
However enrty path parameters are not fixed and I need to read them from a file, to achieve this, I am trying something like this:
from fastapi import APIRouter
my_router = APIRouter()
def foo(**kwargs):
return {"Result": kwargs['xyz']}
read_from_file = '/foo/{xyz}' # Assume this is read from a file
my_router.add_api_route(read_from_file, endpoint=foo)
But it throws this error:
{"detail":[{"loc":["query","kwargs"],"msg":"field required","type":"value_error.missing"}]}
FastAPI tries to find actual argument xyz in foo signature which is not there.
Is there any way in FastAPI to achieve this? Or even any solution to accept a path like /foo/... whatever .../?
This will generate a function with a new signature (I assume every parameter is a string):
from fastapi import APIRouter
import re
import inspect
my_router = APIRouter()
def generate_function_signature(route_path: str):
args = {arg: str for arg in re.findall(r'\{(.*?)\}', route_path)}
def new_fn(**kwargs):
return {"Result": kwargs['xyz']}
params = [
inspect.Parameter(
param,
inspect.Parameter.POSITIONAL_OR_KEYWORD,
annotation=type_
) for param, type_ in args.items()
]
new_fn.__signature__ = inspect.Signature(params)
new_fn.__annotations__ = args
return new_fn
read_from_file = '/foo/{xyz}' # Assume this is read from a file
my_router.add_api_route(
read_from_file,
endpoint=generate_function_signature(read_from_file)
)
However I am sure there is a better way of doing whatever you are trying to do, but I would need to understand your problem first
As described here, you can use the path convertor, provided by Starlette, to capture arbitrary paths. Example:
from fastapi import APIRouter, FastAPI
app = FastAPI()
my_router = APIRouter()
def foo(rest_of_path: str):
return {"rest_of_path": rest_of_path}
route_path = '/foo/{rest_of_path:path}'
my_router.add_api_route(route_path, endpoint=foo)
app.include_router(my_router)
Input test:
http://127.0.0.1:8000/foo/https://placebear.com/cache/395-205.jpg
Output:
{"rest_of_path":"https://placebear.com/cache/395-205.jpg"}
I have the code below that powers on or off a DB cluster based on its tags. I use this code with a Lambda function. The code is to run on a schedule I have defined in cloudformation. However, although the Lambda function is invoked, it does not power off or power on the DB cluster.
It fails each time
tag_name = "AutoPower"
tag_value = "true"
#Get configuration variable
identifier = os.environ.get("db_identifier")
#Clients
rds = boto3.client("rds")
#Function to start the cluster.
def start(identifier):
rds.start_db_cluster(
DBClusterIdentifier=identifier
)
#Function to stop the cluster.
def stop(identifier):
rds.stop_db_cluster(
DBClusterIdentifier=identifier
)
def handler(event, context):
#Call AWS' "describe_db_clusters"; retrieve specific cluster info.
resp = rds.describe_db_clusters(
DBClusterIdentifier=identifier
)
# Isolate the one entry in the 'array' (with one result) we want.
raw = resp["DBClusters"][0]
# Pull tag info out of the dict above.
tag_info = raw["TagList"]
for tag in tag_info:
# If tag is 'AutoPower'
if tag["Key"] == tag_name:
# and Value is 'true'
if tag["Value"] == tag_value:
status = raw["Status"]
# and the DB is running
if (status == 'available'):
# Stop the DB
stop(identifier)
# and the DB is off
elif (status == 'stopped'):
# Start the DB
start(identifier)
I run the code using a Lambda function. However each time I run it the Lambda function fails with the message I have posted below. Can anyone see what the issue is with the code?
{
"errorMessage": "Parameter validation failed:\nInvalid type for parameter DBClusterIdentifier, value: None, type: <class 'NoneType'>, valid types: <class 'str'>",
"errorType": "ParamValidationError",
"requestId": "62767449-e633-4594-b5df-0086e385ebeb",
"stackTrace": [
" File \"/var/task/index.py\", line 34, in handler\n resp = rds.describe_db_clusters(\n",
Based on the comments above, you need identifier specified inside describe_db. Right now it's a global, but likely as it's being used as a callback (handler) the global context isn't carried in.
Perhaps try either determining the value inside handler. This may, or may not, work depending on the environment that handler is being run in:
def handler(event, context):
#Call AWS' "describe_db_clusters"; retrieve specific cluster info.
identifier = os.environ.get("db_identifier")
resp = rds.describe_db_clusters(
DBClusterIdentifier=identifier
)
...
or you could try specifying identifier as part of the args for handler:
def handler(event, context, identifier):
....
please observe the below AWS Lambda function ( in python code) to invoke a single AWS Step Function:
import boto3
def lambda_handler(event, context):
print("stating to invoke the AWS Step function.")
response = {"code":400, "message":""}
try :
stepFunction = boto3.client('stepfunctions')
response = stepFunction.start_execution(
stateMachineARN = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
)
except Exception as e:
response["message"] = f"failed to invoke the step function,{e}"
finally:
print(response)
return response
Now, I want to invoke different multiple AWS Step functions at a time ( in series and in parallel ) using the above single AWS Lambda function. Could anyone help me on how to modify the above code ?
Here's the basics, but be careful you don't accidentally call this function back in another lambda function or you'll get into a perpetual loop
def trigger_my_other_lambda(event, context):
invoked_function_name = os.environ['INVOKED_FUNCTION_NAME']
# this lambda function invokes another lambda function
# invoke other lambda as many times as in tasks_list or other range
for myarn in arnslist:
'''pass data in form of dictionary; this shows up and must be parsed by the event argument of receiving lambda'''
data = {"stateMachineARN" : myarn}
# print(data)
response = client.invoke(
FunctionName=invoked_function_name,
# InvocationType='RequestResponse', # synchornous - waits for response
InvocationType='Event', # asynch - much faster - doesn't wait for response
Payload=json.dumps(data)
)
Then your existing lambda would use the event like this:
stepFunction = boto3.client('stepfunctions')
response = stepFunction.start_execution(
stateMachineARN = event['stateMachineARN']
)
hello i guess have problem with client and member config which config should i use as you can see i am inserting json as data when i call get_data it returns with no problem but when i try to use predicate-sql it gives me error "hazelcast.errors.HazelcastSerializationError: Exception from server: com.hazelcast.nio.serialization.HazelcastSerializationException: There is no suitable de-serializer for type -120. This exception is likely caused by differences in t
he serialization configuration between members or between clients and members."
#app.route('/insert_data/<database_name>/<collection_name>', methods=['POST'])
def insert_data(database_name, collection_name):
client = hazelcast.HazelcastClient(cluster_members=[
url
])
dbname_map = client.get_map(f"{database_name}-{collection_name}").blocking()
if request.json:
received_json_data = request.json
received_id = received_json_data["_id"]
del received_json_data["_id"]
dbname_map.put(received_id, received_json_data)
client.shutdown()
return jsonify()
else:
client.shutdown()
abort(400)
#app.route('/get_data/<database_name>/<collection_name>', methods=['GET'])
def get_all_data(database_name, collection_name):
client = hazelcast.HazelcastClient(cluster_members=[
url
])
dbname_map = client.get_map(f"{database_name}-{collection_name}").blocking()
entry_set = dbname_map.entry_set()
output = dict()
datas = []
for key, value in entry_set:
value['_id'] = key
output = value
datas.append(output)
client.shutdown()
return jsonify({"Result":datas})
#bp.route('/get_query/<database_name>/<collection_name>/<name>', methods=['GET'])
def get_query_result(database_name, collection_name,name):
client = hazelcast.HazelcastClient(cluster_members=[
url
])
predicate_map = client.get_map(f"{database_name}-{collection_name}").blocking()
predicate = and_(sql(f"name like {name}%"))
entry_set = predicate_map.values(predicate)
#entry_set = predicate_map.entry_set(predicate)
send_all_data = ""
for x in entry_set:
send_all_data += x.to_string()
send_all_data += "\n"
print(send_all_data)
# print("Retrieved %s values whose age is less than 30." % len(result))
# print("Entry is", result[0].to_string())
# value=predicate_map.get(70)
# print(value)
return jsonify()
i try to change hazelcast.xml according to hazelcast-full-example.xml but i can't start hazelcast after
the changes and do i really have to use serialization ? hazelcast version:4.1 python:3.9
This is most likely happening because you are putting entries of the type dictionary to the map, which is serialized by the pickle because you didn't specify a serializer for that and the client does not know how to handle that correctly, so it fallbacks to the default serializer. However, since pickle serialization is Python-specific, servers cannot deserialize it and throw such an exception.
There are possible solutions to that, see the https://hazelcast.readthedocs.io/en/stable/serialization.html chapter for details.
I think the most appropriate solution for your use case would be Portable serialization which does not require a configuration change or code on the server-side. See the https://hazelcast.readthedocs.io/en/stable/serialization.html#portable-serialization
BTW, client objects are quite heavyweight, so you shouldn't be creating them on demand like this. You can construct it once in your application and share and use it in your endpoints or business-logic code freely since it is thread-safe. The same applies to the map proxy you get from the client. It can also be re-used.
I have created a function in AWS lambda which looks like this:
import boto3
import numpy as np
import pandas as pd
import s3fs
from io import StringIO
def test(event=None, context=None):
# creating a pandas dataframe from an api
# placing 2 csv files in S3 bucket
This function queries an external API and places 2 csv files in S3 bucket. I want to trigger this function in Airflow, I have found this code:
import boto3, json, typing
def invokeLambdaFunction(*, functionName:str=None, payload:typing.Mapping[str, str]=None):
if functionName == None:
raise Exception('ERROR: functionName parameter cannot be NULL')
payloadStr = json.dumps(payload)
payloadBytesArr = bytes(payloadStr, encoding='utf8')
client = boto3.client('lambda')
response = client.invoke(
FunctionName=test,
InvocationType="RequestResponse",
Payload=payloadBytesArr
)
return response
if __name__ == '__main__':
payloadObj = {"something" : "1111111-222222-333333-bba8-1111111"}
response = invokeLambdaFunction(functionName='test', payload=payloadObj)
print(f'response:{response}')
But as I understand this code snippet does not connect to the S3. Is this the right approach to trigger AWS Lambda function from Airflow or there is a better way?
I would advice to use the AwsLambdaHook:
https://airflow.apache.org/docs/stable/_api/airflow/contrib/hooks/aws_lambda_hook/index.html#module-airflow.contrib.hooks.aws_lambda_hook
And you can check a test showing its usage to trigger a lambda function:
https://github.com/apache/airflow/blob/master/tests/providers/amazon/aws/hooks/test_lambda_function.py