SO I'm using boto to invoke my lambda functions and test my backend. I want to invoke them asynchronously. I have noted that "invoke_async" is deprecated and should not be used. Instead you should use "invoke" with an InvocationType of "Event" to do the function asynchronously.
I can't seem to figure out how to get the responses from the functions when they return though. I have tried the following:
payload3=b"""{
"latitude": 39.5732160891,
"longitude": -119.672918997,
"radius": 100
}"""
client = boto3.client('lambda')
for x in range (0, 5):
response = client.invoke(
FunctionName="loadSpotsAroundPoint",
InvocationType='Event',
Payload=payload3
)
time.sleep(15)
print(json.loads(response['Payload'].read()))
print("\n")
Even though I tell the code to sleep for 15 seconds, the response variable is still empty when I try and print it. If I change the invokation InvokationType to "RequestResponse" it all works fine and response variable prints, but this is synchronous. Am I missing something easy? How do i execute some code, for example print out the result, when the async invokation returns??
Thanks.
There is a difference between an 'async AWS lambda invocation' and 'async python code'. When you set the InvocationType to 'Event', by definition, it does not ever send back a response.
In your example, invoke() immediately returns None, and does not implicitly start up anything in the background to change that value at a later time (thank goodness!). So, when you look at the value of response 15 seconds later, it's still None.
It seems what you really want is the RequestResponse invocation type, with asynchronous Python code. You have a bunch of options to choose from, but my favorite is concurrent.futures. Another is threading.
Here's an example using concurrent.futures:
(If you're using Python2 you'll need to pip install futures)
from concurrent.futures import ThreadPoolExecutor
import json
payload = {...}
with ThreadPoolExecutor(max_workers=5) as executor:
futs = []
for x in xrange(0, 5):
futs.append(
executor.submit(client.invoke,
FunctionName = "loadSpotsAroundPoint",
InvocationType = "RequestResponse",
Payload = bytes(json.dumps(payload))
)
)
results = [ fut.result() for fut in futs ]
print results
Another pattern you might want to look into is to use the Event invocation type, and have your Lambda function push messages to SNS, which are then consumed by another Lambda function. You can check out a tutorial for SNS-triggered lambda functions here.
An asynchronously executed AWS Lambda function doesn't return the result of execution. If an asynchronous invocation request is successful (i.e. there were no errors due to permissions, etc), AWS Lambda immediately returns the HTTP status code 202 ACCEPTED and bears no further responsibility for communicating any information about the outcome of this asynchronous invocation.
From the documentation of AWS Lambda Invoke action:
Response Syntax
HTTP/1.1 StatusCode
X-Amz-Function-Error: FunctionError
X-Amz-Log-Result: LogResult
Payload
Response Elements
If the action is successful, the service sends back the following HTTP
response.
StatusCode
The HTTP status code will be in the 200 range for successful request.
For the RequestResponse invocation type this status code will be 200.
For the Event invocation type this status code will be 202. For the DryRun invocation type the status code will be 204.
[...]
The response returns the following as the HTTP body.
Payload
It is the JSON representation of the object returned by the Lambda
function. This is present only if the invocation type is
RequestResponse.
The following are a python function that accepts lambda-function-Name to invoke and payload to send to that function.
It invokes the lambda function by boto3 client.
import boto3, json, typing
def invokeLambdaFunction(*, functionName:str=None, payload:typing.Mapping[str, str]=None):
if functionName == None:
raise Exception('ERROR: functionName parameter cannot be NULL')
payloadStr = json.dumps(payload)
payloadBytesArr = bytes(payloadStr, encoding='utf8')
client = boto3.client('lambda')
return client.invoke(
FunctionName=functionName,
InvocationType="RequestResponse",
Payload=payloadBytesArr
)
And usage:
if __name__ == '__main__':
payloadObj = {"something" : "1111111-222222-333333-bba8-1111111"}
response = invokeLambdaFunction(functionName='myLambdaFuncName', payload=payloadObj)
print(f'response:{response}')
Related
I want to send a acknowledgement to the user that request has been received and FUNCTION_NAME has been called. The FUNCTION_NAME internally calls other functions and takes about 10-15 mins to complete. So the user is acknowledged that his request is captured for processing. Any hints/leads towards handing this approaching
import os
import json
import boto3
import json as js
lambda_client = boto3.client('lambda')
def lambda_handler(event, context):
res=lambda_client.invoke(FunctionName='FUNCTION_NAME',InvocationType='RequestResponse',Payload=js.dumps(event))
res=res['Payload'].read().decode()
data= js.loads(res)['body']
status=js.loads(res)['statusCode']
return {
'isBase64Encoded':"false",
'statusCode': status,
'body': data
}
As mentioned in the boto3 documentation, you can invoke the other lambda synchronously or asynchronously.
To invoke the lambda asynchronously, change the InvocationType to Event.
See the documentation for reference.
Please do not use invoke_async, this is deprecated.
I'm new to Locust, and I am attempting to log statistics for a POST request, and I'm using the following code along with a generic call to locust.
import json
from locust import HttpUser, task, between
import cfg
class BasicUser(HttpUser):
wait_time = between(1, 3)
v1_data = json.load(open("v1_sample_data.json", "r"))
#task
def get_v1_prediction(self):
route = "/" + cfg.lookup("model.v1.route")
response = self.client.post(
route,
json=self.v1_data,
catch_response=True,
name="API Call"
)
print(response.text)
When I start an experiment, the host is called successfully, and response.text has the expected value and is printed to the console repeatedly. However, the statistics aren't logged.
When I use a GET request in place of the POST without passing data, statistics are logged (though it's only failures because the web app only allows POST requests). Any idea what's going on here?
The catch_response=True is the culprit.
From the documentation:
catch_response – (optional) Boolean argument that, if set, can be used to make a request return a context manager to work as argument to a with statement. This will allow the request to be marked as a fail based on the content of the response, even if the response code is ok (2xx). The opposite also works, one can use catch_response to catch a request and then mark it as successful even if the response code was not (i.e 500 or 404).
I have one function where I am calling API to post get the resource. Since my function does not return anything then its tough to write unit test for failure scenario. Here I want to force request.get() to return different HTTP status code.
Is there anyway to mock my function to return desired status code?
foo.py
def getData():
response = requests.get(run_task_status_url, headers=iics_job_header)
logging.debug(f"Activity Monitor API response: {response.json()}")
if 200 == response.status_code:
print("success")
else 401 == response.status_code:
print("401")
According to official doc, you can make batch request through Google API Client Libraries for Python. Here, there is an example
from apiclient.http import BatchHttpRequest
def list_animals(request_id, response, exception):
if exception is not None:
# Do something with the exception
pass
else:
# Do something with the response
pass
def list_farmers(request_id, response):
"""Do something with the farmers list response."""
pass
service = build('farm', 'v2')
batch = service.new_batch_http_request()
batch.add(service.animals().list(), callback=list_animals)
batch.add(service.farmers().list(), callback=list_farmers)
batch.execute(http=http)
Is there a way to access to the request in the callback. E.g., print the request (not the request_id) if there is an exception?
I ran into this same issue, as I needed the original request for using compute.instances().aggregatedList_next() function.
I ended up passing a unique request_id to batch.add() and created a dictionary to keep track of the original requests that were made. Then within the callback, I referenced that dictionary and pulled out the request by using the unique request_id.
So I did something along the following:
requests = {}
project = 'example-project'
batch = service.new_batch_http_request(callback=callback_callable)
new_request = compute.instances().aggregatedList(project=project)
requests[project] = new_request
batch.add(new_request, request_id=project)
Within the callable function:
def callback_callable(self, request_id, response, exception):
original_request = requests[request_id]
Note that the value passed to the request_id needs to be a string. If an int is passed it will raise a TypeError quote_from_bytes() expected bytes.
Hope this helps someone!
In case it helps anyone - I'm using google-api-python-client==2.65.0
and the call back is passed differently - it is specified like this:
def list_animals(request_id, response, exception=None):
if exception is None:
original_request = requests[request_id]
# process here...
batch = service.new_batch_http_request(callback=list_animals)
batch.add(service.animals().list())
batch.add(service.farmers().list())
batch.execute()
I'm using multiprocessing library (not new in python but new in multiprocessing). It seems that I lack of understanding how it works.
What I try to do: I send a lot of http requests to server and if I receive connection error it means that remote service is down and I restart it using paramiko and then resend a request. I use multiprocessing to load all available processors because there are about 70000 requests and it takes about 24 hours to process them all using one processor.
My code:
# Send request here
def send_request(server, url, data, timeout):
try:
return requests.post(server + url, json=data, timeout=(timeout or 60))
except Exception:
return None
# Try to get json from response
def do_gw_requests(request, data):
timeout = 0
response = send_request(server, request, data, timeout)
if response is not None:
response_json = json.loads(response.text)
else:
response_json = None
return response_json
# Function that recall itself if service is down
def safe_build(data):
exception_message = ""
response = {}
try:
response = do_gw_requests("/rgw_find_route", data)
if response is None:
# Function that uses paramiko to start service
# It will not end until service is up
start_service()
while response is None:
safe_build(data)
--some other work here--
return response, exception_message
# Multiprocessing lines in main function
pool = Pool(2)
# build_single_route prepares data, calls safe_build once and write logs
result = pool.map_async(build_single_route, args)
pool.close()
pool.join()
My problem is if service already down at the start of script (and potentially if service got down in the middle of script's work) I can't get non-empty response for two first requests. Script starts, send two first requests (I send them in loop by two), finds out that service is down (response become None), restarts service, resends requests and seems gets None again and again and again (in endless loop). If I remove loop while response is None: then first two requests will process as if they was None and other requests will process as expected. But I need every request result that's why I resend bad requests.
So it recall function with same data again and again but without success. It's very strange as for me. Can anyone please explain what am I doing wrong here?
It seems that problem not with behavior of Pool workers as I expected. response is a local variable of function and thus it become not None after reviving of service at the second call of safe_build, it's still None in the first call. response, _ = safe_build(data) seems work.