I want to send a acknowledgement to the user that request has been received and FUNCTION_NAME has been called. The FUNCTION_NAME internally calls other functions and takes about 10-15 mins to complete. So the user is acknowledged that his request is captured for processing. Any hints/leads towards handing this approaching
import os
import json
import boto3
import json as js
lambda_client = boto3.client('lambda')
def lambda_handler(event, context):
res=lambda_client.invoke(FunctionName='FUNCTION_NAME',InvocationType='RequestResponse',Payload=js.dumps(event))
res=res['Payload'].read().decode()
data= js.loads(res)['body']
status=js.loads(res)['statusCode']
return {
'isBase64Encoded':"false",
'statusCode': status,
'body': data
}
As mentioned in the boto3 documentation, you can invoke the other lambda synchronously or asynchronously.
To invoke the lambda asynchronously, change the InvocationType to Event.
See the documentation for reference.
Please do not use invoke_async, this is deprecated.
I'm trying to understand, how to create a REST API using AWS Lambda and Python. For this, I'm creating a simple Lambda function that will accept 2 numbers from the user and return the sum of these numbers.
Code of my Lambda function:
import json
def lambda_handler(event, context):
# TODO implement
try:
#User Input:n1 & n2
_query = event['queryStringParameters']
_n1 = int(_query['n1'])
print(_n1)
_n2 = int(_query['n2'])
print(_n2)
del(_query)
#Response Body
_body = {
'n1': _n1,
'n2': _n2,
'sum': _n1+_n2
}
print(_body)
#Response Header
_header = {
'Content-Type': 'application/json'
}
print(_header)
#Response
_response = {
'statusCode': 200,
'headers': json.dumps(_header),
'body': json.dumps(_body)
}
print(_response)
return _response
except Exception as e:
print(e)
This Lambda function is connected to AWS API Gateway to allow public access. I have not configured any authentication mechanism for API Gateway.
I'm trying to call this API from another python program.
Source code of calling program:
from requests import get
try:
_url = 'URL_of_the_lambda_function'
_params = {
'n1':1,
'n2':2
}
r = get(
url=_url,
params=_params
)
data = r.json()
print(data)
except Exception as e:
print(e)
This program is getting executed on the terminal with the help of VS Code. When I'm trying to run the program, I'm getting
{'message': 'Internal server error'}
Can anyone please help me to understand, what is the mistake with my code?
[N.B.: As suggested fellow community members:
I have converted header to dictionary.
I have added try/catch block to both(lambda and calling function)
I have tried to print every header, response and body objects
]
I created a lambda function in AWS and apparently it is throwing an error. Here it is:
import json
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
import boto3
#Create a SSM Client to access parameter store
ssm = boto3.client('ssm')
def lambda_handler(event, context):
# TODO implement
#return {
# 'statusCode': 200,
# 'body': json.dumps('Hello from Lambda!')
#}
slack_message = {
'text' = f'Hello World'
}
#retrieve webhook url from parameter store
webhook_url = ssm.get_parameter(Name='slackwebhookurl', WithDecryption=True)
#make request to the API
req = Request(webhook_url['Parameter']['Value'],
json.dumps(slack_message).encode('utf-8'))
try:
response = urlopen(req)
response.read()
print("Messge posted to Slack")
except HTTPError as e:
print(f'Request failed: {e.code} {e.reason})
except URLError as e:
print(f'Server Connection failed: {e.reason})
It's triggered by an AWS SNS notification. It's supposed to grab a webhook url for a slack channel and then send the notification to Slack.
Can anyone see what the problem is?
If it's not obvious, can someone direct me to a tutorial on how to test AWS Lambda functions?
Thanks.
AWS Lambda has an in-built Test functionality. You can click the Test button and configure an input in case the function uses values from event.
Logs will be displayed within the Lambda console.
Also, ensure that the IAM Role associated with the AWS Lambda function has the AWSLambdaBasicExecutionRole permission policy so that it can write to CloudWatch Logs. Then, you can go to the Monitoring tab of the function and click View logs in CloudWatch to see past logs.
You can add print() statements in your code, which will appear in the Logs.
I am trying to make a POST request to Slack using webhooks. I can send a curl to my Slack instance locally but when trying to do so in lambda I run into trouble trying to send the payload.
Everything I've seen says I must use and zip custom libraries but for the purposes of what I'm doing I need to use native python code. Is there a way to send this POST request?
import json
import urllib.request
#import botocore.requests as requests
def lambda_handler(event, context):
message=event['message']
response = urllib.request.urlopen(message)
print(response)
This code gives me a 400 error which is how I know I'm hitting the URL I want (URL is in the message variable) but every attempt at sending a payload by adding headers and a text body seems to fail.
You may try as below:
SLACK_URL = 'https://hooks.slack.com/services/....'
req = urllib.request.Request(SLACK_URL, json.dumps(message).encode('utf-8'))
response = urllib.request.urlopen(req)
Please find attached lambda_handler code, hope this helps you.
All the messages to be posted on slack are put on a SNS topic which in turn is read by a lambda and posted to the slack channel using the slack webhook url.
import os
import json
from urllib2 import Request, urlopen, URLError, HTTPError
# Get the environment variables
SLACK_WEBHOOK_URL = os.environ['SLACK_WEBHOOK_URL']
SLACK_CHANNEL = os.environ['SLACK_CHANNEL']
SLACK_USER = os.environ['SLACK_USER']
def lambda_handler(event, context):
# Read message posted on SNS Topic
message = json.loads(event['Records'][0]['Sns']['Message'])
# New slack message is created
slack_message = {
'channel': SLACK_CHANNEL,
'username': SLACK_USER,
'text': "%s" % (message)
}
# Post message on SLACK_WEBHOOK_URL
req = Request(SLACK_WEBHOOK_URL, json.dumps(slack_message))
try:
response = urlopen(req)
response.read()
print(slack_message['channel'])
except HTTPError as e:
print(e)
except URLError as e:
print(e)
I'm trying to use HTTPHandler class of standard python logging library to send logs. I need to make a https post request with basic credentials(username and password). This is how i'm setting up the HTTPHandler-
host = 'example.com'
url = '/path'
handler = logging.handlers.HTTPHandler(host, url, method='POST', secure=True, credentials=('username','password'), context=None)
logger.addHandler(handler)
But the problem is, I'm not getting anylogs in my remote server.I'm not even seeing any exception from the handler. Am I setting up the handler arguments incorrectly? I can send similar logs using simple pythong http request-
url = 'https://username:password#example.com/path'
headers = {'content-type': 'application/json'}
jsonLog = { 'id': '4444','level': 'info', 'message': 'python log' };
r = requests.post(url, data = json.dumps(jsonLog), headers=headers)
Do i need to setup header somehow because of json content-type? If yes than how do i set that up in the httphandler?
Update
I thought I should update what I ended up doing. After numerous search i found i can create a custom handler by overriding emit() of logging.Handler.
class CustomHandler(logging.Handler):
def emit(self, record):
log_entry = self.format(record)
# some code....
url = 'url'
# some code....
return requests.post(url, log_entry, headers={"Content-type": "application/json"}).content
Feel free to post if any has any better suggestions.
Expanding on the solution saz gave, here's how add a custom HTTP handler that
will forward the logs emitted to the specified URL using a bearer token.
It uses a requests session instead of having to establish a new session every log
event.
Furthermore, if the request fails it attempts to resend the logs for a given number of retries.
Note: make sure your logging handler is as simple as possible to prevent the application from halting because of a log event.
I tested it with a simple localhost echo server and it works.
Feel free to suggest any changes.
import json
import logging
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
class CustomHttpHandler(logging.Handler):
def __init__(self, url: str, token: str, silent: bool = True):
'''
Initializes the custom http handler
Parameters:
url (str): The URL that the logs will be sent to
token (str): The Authorization token being used
silent (bool): If False the http response and logs will be sent
to STDOUT for debug
'''
self.url = url
self.token = token
self.silent = silent
# sets up a session with the server
self.MAX_POOLSIZE = 100
self.session = session = requests.Session()
session.headers.update({
'Content-Type': 'application/json',
'Authorization': 'Bearer %s' % (self.token)
})
self.session.mount('https://', HTTPAdapter(
max_retries=Retry(
total=5,
backoff_factor=0.5,
status_forcelist=[403, 500]
),
pool_connections=self.MAX_POOLSIZE,
pool_maxsize=self.MAX_POOLSIZE
))
super().__init__()
def emit(self, record):
'''
This function gets called when a log event gets emitted. It recieves a
record, formats it and sends it to the url
Parameters:
record: a log record
'''
logEntry = self.format(record)
response = self.session.post(self.url, data=logEntry)
if not self.silent:
print(logEntry)
print(response.content)
# create logger
log = logging.getLogger('')
log.setLevel(logging.INFO)
# create formatter - this formats the log messages accordingly
formatter = logging.Formatter(json.dumps({
'time': '%(asctime)s',
'pathname': '%(pathname)s',
'line': '%(lineno)d',
'logLevel': '%(levelname)s',
'message': '%(message)s'
}))
# create a custom http logger handler
httpHandler = CustomHttpHandler(
url='<YOUR_URL>',
token='<YOUR_TOKEN>',
silent=False
)
httpHandler.setLevel(logging.INFO)
# add formatter to custom http handler
httpHandler.setFormatter(formatter)
# add handler to logger
log.addHandler(httpHandler)
log.info('Hello world!')
You will need to subclass HTTPHandler and override the emit() method to do what you need. You can use the current implementation of HTTPHandler.emit() as a guide.
following up to istvan, you can use threads to prevent slowing down the program
import asyncio
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(max_workers=10)
import time
import json
import logging
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
class CustomHttpHandler(logging.Handler):
def __init__(self, url: str, token: str, silent: bool = True):
'''
Initializes the custom http handler
Parameters:
url (str): The URL that the logs will be sent to
token (str): The Authorization token being used
silent (bool): If False the http response and logs will be sent
to STDOUT for debug
'''
self.url = url
self.token = token
self.silent = silent
# sets up a session with the server
self.MAX_POOLSIZE = 100
self.session = session = requests.Session()
session.headers.update({
'Content-Type': 'application/json',
'Authorization': 'Bearer %s' % (self.token)
})
self.session.mount('https://', HTTPAdapter(
max_retries=Retry(
total=5,
backoff_factor=0.5,
status_forcelist=[403, 500]
),
pool_connections=self.MAX_POOLSIZE,
pool_maxsize=self.MAX_POOLSIZE
))
super().__init__()
def emit(self, record):
'''
This function gets called when a log event gets emitted. It recieves a
record, formats it and sends it to the url
Parameters:
record: a log record
'''
executor.submit(actual_emit, self, record)
def actual_emit(self, record):
logEntry = self.format(record)
response = self.session.post(self.url, data=logEntry)
print(response)
if not self.silent:
print(logEntry)
print(response.content)
# create logger
log = logging.getLogger('test')
log.setLevel(logging.INFO)
# create formatter - this formats the log messages accordingly
formatter = logging.Formatter(json.dumps({
'time': '%(asctime)s',
'pathname': '%(pathname)s',
'line': '%(lineno)d',
'logLevel': '%(levelname)s',
'message': '%(message)s'
}))
# create a custom http logger handler
httpHandler = CustomHttpHandler(
url='<URL>',
token='<YOUR_TOKEN>',
silent=False
)
httpHandler.setLevel(logging.INFO)
log.addHandler(httpHandler)
def main():
print("start")
log.error("\nstop")
print("now")
if __name__ == "__main__":
main()
what this program does is send the logs to the threadpoolexecutor, with 10 max threads, if there are more logs then the threads can handle, it should queue up, this prevents slowdowns of the program.
What you can also do, atleast what I am doing on my project of making a local host logging central database and viewer, I make a seperate thread on the serverside, and then instantly return a HTTP response to make it so all the database stuff happens after the HTTP resonse has been send back. This removes the need for threads on client, seen it is on localhost and then latancy is almost 0