I created SQS service with terraform
resource "aws_sqs_queue" "ses_queue" {
name = "ses_queue"
message_retention_seconds = 86400
receive_wait_time_seconds = 1
visibility_timeout_seconds = 15
}
resource "aws_lambda_event_source_mapping" "send_email_message" {
event_source_arn = aws_sqs_queue.ses_queue.arn
function_name = aws_lambda_function.send_email_message.function_name
batch_size = 5
}
I am sending emails using lambda function
for record in event.get("Records"):
receipt_handle = record.get("receiptHandle", "")
request_body = record.get("body")
response = send_email(request_body)
if response:
sqs_client.delete_message(QueueUrl=constants.SES_QUEUE_URL, ReceiptHandle=receipt_handle)
I am wondering why number of deleted messages is twice as many as received messages
The Lambda Event Source Mapping already deletes the messages from the Queue if your Lambda Function terminates without errors:
If your function successfully processes the batch, Lambda deletes the messages from the queue.
— Source
In your Function Code you explicitly delete the messages you processed as well, this means the delete happens twice.
Fixed it with reporting batch item failures
terraform
resource "aws_lambda_event_source_mapping" "send_email_message" {
event_source_arn = aws_sqs_queue.ses_queue.arn
function_name = aws_lambda_function.send_email_message.function_name
batch_size = 5
function_response_types = ["ReportBatchItemFailures"]
}
lambda
reprocess_messages = []
for record in event.get("Records"):
receipt_handle = record.get("receiptHandle", "")
message_id = record.get("messageId", "")
request_body = record.get("body")
response = send_email(request_body)
if not response:
reprocess_messages.append({"itemIdentifier": message_id})
return {"batchItemFailures": reprocess_messages}
Related
I have pulled data from a private slack channel, using conversation history, and it pulls the userid instead of username, how can I change the code to pull the user name so I can identify who each user is? Code below
CHANNEL = ""
MESSAGES_PER_PAGE = 200
MAX_MESSAGES = 1000
SLACK_TOKEN = ""
client = slack_sdk.WebClient(token=SLACK_TOKEN)
# get first page
page = 1
print("Retrieving page {}".format(page))
response = client.conversations_history(
channel=CHANNEL,
limit=MESSAGES_PER_PAGE,
)
assert response["ok"]
messages_all = response['messages']
# get additional pages if below max message and if they are any
while len(messages_all) + MESSAGES_PER_PAGE <= MAX_MESSAGES and response['has_more']:
page += 1
print("Retrieving page {}".format(page))
sleep(1) # need to wait 1 sec before next call due to rate limits
response = client.conversations_history(
channel=CHANNEL,
limit=MESSAGES_PER_PAGE,
cursor=response['response_metadata']['next_cursor']
)
assert response["ok"]
messages = response['messages']
messages_all = messages_all + messages
It isn't possible to change what is returned from the conversations.history method. If you'd like to convert user IDs to usernames, you'll need to either:
Call the users.info method and retrieve the username from the response.
or
Call the users.list method and iterate through the list and create a local copy (or store in a database) and then have your code look it up.
I am creating a SAM web app, with the backend being an API in front of a Python Lambda function with a DynamoDB table that maintains a count of the number of HTTP calls to the API. The API must also return this number. The yaml code itself loads normally. My problem is writing the Lambda function to iterate and return the count. Here is my code:
def lambda_handler(event, context):
dynamodb = boto3.resource("dynamodb")
ddbTableName = os.environ["databaseName"]
table = dynamodb.Table(ddbTableName)
# Update item in table or add if doesn't exist
ddbResponse = table.update_item(
Key={"id": "VisitorCount"},
UpdateExpression="SET count = count + :value",
ExpressionAttributeValues={":value": Decimal(context)},
ReturnValues="UPDATED_NEW",
)
# Format dynamodb response into variable
responseBody = json.dumps({"VisitorCount": ddbResponse["Attributes"]["count"]})
# Create api response object
apiResponse = {"isBase64Encoded": False, "statusCode": 200, "body": responseBody}
# Return api response object
return apiResponse
I can get VisitorCount to be a string, but not a number. I get this error: [ERROR] TypeError: lambda_handler() missing 1 required positional argument: 'cou response = request_handler(event, lambda_context)le_event_request
What is going on?
[UPDATE] I found the original error, which was that the function was not properly received by the SAM app. Changing the name fixed this, and it is now being read. Now I have to troubleshoot the actual Python. New Code:
import json
import boto3
import os
dynamodb = boto3.resource("dynamodb")
ddbTableName = os.environ["databaseName"]
table = dynamodb.Table(ddbTableName)
Key = {"VisitorCount": { "N" : "0" }}
def handler(event, context):
# Update item in table or add if doesn't exist
ddbResponse = table.update_item(
UpdateExpression= "set VisitorCount = VisitorCount + :val",
ExpressionAttributeValues={":val": {"N":"1"}},
ReturnValues="UPDATED_NEW",
)
# Format dynamodb response into variable
responseBody = json.dumps({"VisitorCount": ddbResponse["Attributes"]["count"]})
# Create api response object
apiResponse = {"isBase64Encoded": False, "statusCode": 200,"body": responseBody}
# Return api response object
return apiResponse
I am getting a syntax error on Line 13, which is
UpdateExpression= "set VisitorCount = VisitorCount + :val",
But I can't tell where I am going wrong on this. It should update the DynamoDB table to increase the count by 1. Looking at the AWS guide it appears to be the correct syntax.
Not sure what the exact error is but ddbResponse will be like this:
ddbResponse = table.update_item(
Key={
'key1': aaa,
'key2': bbb
},
UpdateExpression= "set VisitorCount = VisitorCount + :val",
ExpressionAttributeValues={":val": Decimal(1)},
ReturnValues="UPDATED_NEW",
)
Specify item to be updated with Key (one item for one Lambda call)
Set Decimal(1) for ExpressionAttributeValues
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStarted.Python.03.html#GettingStarted.Python.03.04
I am writing a program for doing Bloomberg data-feed check using the subscription method of Python API. I am close to finishing it and I am now trying to cover edge cases such as a failed subscription.
I want to check if a subscription has failed. If it fails, I will write it into a file named BadSubscription.txt.
One of he example programs that come with Bloomberg API package, SimpleSubcriptionExample.py, has just 1 line of code for Subscription Status so it doesn't give me a clear idea.
try:
# Process received events
eventCount = 0
while(True):
# We provide timeout to give the chance to Ctrl+C handling:
event = session.nextEvent(15000)
for msg in event:
if event.eventType() == blpapi.Event.SUBSCRIPTION_STATUS or \
event.eventType() == blpapi.Event.SUBSCRIPTION_DATA:
print("%s - %s" % (msg.correlationIds()[0].value(), msg))
else:
print(msg)
The above code prints the following when a subscription fails for subscribing to a security/equity that doesn't exist:
SubscriptionFailure = {
reason = {
errorCode = 2
description = "Invalid security, rcode = -11"
category = "BAD_SEC"
source = " [nid:3924]:bbdbm10"
}
}
And when a subscription is successful it prints:
SubscriptionStarted = {
exceptions[] = {
}
streamIds[] = {
"1"
}
receivedFrom = {
address = "localhost:8194"
}
reason = "Subscriber made a subscription"
}
What I want to do is write an if statement for my program to catch the SubscriptionFailure and write the message to the file:
for msg in event:
if (event.eventType() == blpapi.Event.SUBSCRIPTION_STATUS
and (**the condition to catch the error**)):
f = open("BadSubscription.txt", "a+")
f.write(msg)
I am looking for a condition to use in my if statement.
I tried reading the following repository but it doesn't explain much, too.
https://bloomberg.github.io/blpapi-docs/python/3.13/_autosummary/blpapi.Session.html?highlight=subscription%20status
I first tried
msg.correlationIds()[0].value().find("SubscriptionFailure")!=-1
as the condition but that didn't work.
Thanks to #assylias I found the solution.
for msg in event:
if (event.eventType() == blpapi.Event.SUBSCRIPTION_STATUS
and msg.messageType() == "SubscriptionFailure"):
f = open("BadSubscription.txt", "a+")
s = ""
if msg.getElement("reason").getElement("errorCode").getValueAsInteger() !=12:
s = msg.toString()
f.write(s)
The above code writes the following to my file:
SubscriptionFailure = {
reason = {
errorCode = 2
description = "Invalid security, rcode = -11"
category = "BAD_SEC"
source = " [nid:235]:bbdbm10"
}
}
I'm trying to develop a web chat with Flask and Firestore. I set a flow to receive new messages from firestore (when something changes at the database) and send through websockets to UI. Something like that:
Python:
#sockets.route('/messages')
def chat_socket(ws):
message = None
def callback_snapshot(col_snapshot, changes, read_time):
with app.app_context():
Messages = []
for change in changes:
if change.type.name == 'ADDED':
Messages.append(change.document)
conversation = render_template(
'conversation.html',
Messages = Messages,
)
numberID = None
if len(col_snapshot) > 0:
for i in col_snapshot:
a = i
numberID = a.reference.parent.parent.id
response = json.dumps({
'conversation': conversation,
'numberID': numberID
})
ws.send(response)
while not ws.closed:
response = json.loads(ws.receive())
newNumberID = response['newNumberID'].strip()
query_snapshot = fn.GetMessages(newNumberID)
doc_watch = query_snapshot.on_snapshot(callback_snapshot)
if message is None:
continue
Javascript:
function messages(numberID) {
var scheme = window.location.protocol == "https:" ? 'wss://' : 'ws://';
var webSocketUri = scheme
+ window.location.hostname
+ (location.port ? ':'+location.port: '')
+ '/messages';
/* Get elements from the page */
var form = $('#chat-form');
var textarea = $('#chat-text');
var output = $('.messages');
var status = $('.messages');
var websocket = new WebSocket(webSocketUri);
websocket.onopen = function() {};
websocket.onclose = function() {};
websocket.onmessage = function(e) {
numberID = JSON.parse(e.data).numberID
conversation = JSON.parse(e.data).conversation
output.append(conversation);
if (numberID == null){
output.empty();
}};
websocket.onerror = function(e) {console.log(e);};
websocket.onopen = () => websocket.send(numberID);
};
The problem is: When I use col_snapshot as Messages, everything is ok besides I get the whole firestore Collection sent every time to the user when a message is sent. So it's totally not efficient. When I set callback only for changes, as described above, if I trigger the function more than one time, somehow I set multiple listeners for the same collection, so I get multiple "changes updates" in UI. How I can keep track of those listeners so I only set one listener per Collection?
As you can see from the documentation, you should only call GetMessages and on_snapshot once per document.
#sockets.route('/messages')
def chat_socket(ws):
message = None
def callback_snapshot(col_snapshot, changes, read_time):
with app.app_context():
# Rest of the function ...
ws.send(response)
response = json.loads(ws.receive())
numberID = response['newNumberID'].strip()
query_snapshot = fn.GetMessages(numberID)
doc_watch = query_snapshot.on_snapshot(callback_snapshot)
while not ws.closed:
newNumberID = response['newNumberID'].strip()
response = json.loads(ws.receive())
if newNumberID != numberID:
numberID = newNumberID
query_snapshot = fn.GetMessages(numberID)
doc_watch = query_snapshot.on_snapshot(callback_snapshot)
I am trying to create a client used to mainly test out the responses of a server asynchronously. I have created a function that basically waits for the next response from the server, if a requestId is provided when this function is called it will look for the next response with the requestId provided. Here is the function:
def getNextResponse(self, requestId = None):
logger = logging.getLogger(__name__)
self.acknowledge += 1
logger.info("requestId ack for this response: {}".format(requestId))
while(not self.response):
pass
self.acknowledge -= 1
logger.info("requestId unset for this response: {}".format(requestId))
message = json.loads(self.messagesList[len(self.messagesList)-1])
if(requestId != None):
while(requestId != message['body']['requestId']):
self.acknowledge += 1
while(not self.response):
pass
self.acknowledge -= 1
message = self.messagesList[len(self.messagesList)-1]
self.startMonitor -= 1
return message['body']
I also have helper functions for each command which can be sent to the engine below is one of said helper function for a ping command:
def ping(self, sessionId = None, requestId = None, version="1.0"):
result = {
"method": "Ping"
}
if(None != version):
result['version'] = version
if(None != sessionId):
result['sessionId'] = sessionId
if(None != requestId):
result['requestId'] = requestId
logger = logging.getLogger(__name__)
logger.info("Message Sent: " + json.dumps(result, indent=4))
self.startMonitor += 1
self.ws.send(json.dumps(result))
message = self.getNextResponse(requestId = requestId)
return message
It basically sets up a json object which contains all the parameters that the server expects and then sends the entire json message to the server. After it has been sent i call getNextResponse to await a response from the server. The requestId is set to None by default, so if no requestId is provided, it will just look for the very next response returned by the server. Since this can be quite inconsistent because of other asynchronous commands, one can provided a unique requestId for the command so that the response from the server will also contain this requestId thus making each response unique.
In my test case I am generating a random requestId by using:
def genRequestId(self):
x = random.randint(10000000, 99999999)
print x
reqId = str(x)+"-97a2-11e6-9346-fde5d2234523"
return reqId
The problem that I encountered is that sometimes (seems to be random), when I call ping in one of my test cases, i get this error:
message = self.getNextResponse(requestId = requestId)
TypeError: string indices must be integers, not str
I am quite confused by this error, requestId that I am generating inside ping is supposed to be a string and I am not referencing inside it in any way. I have tried removing the reference to the parameter like so:
message = self.getNextResponse(requestId)
But I am still getting this error. The error doesn't go any deeper inside the getNextResponse function which leads me to believe that it is coming from inside the ping function when I try to call it. Any help would be greatly appreciated!
EDIT: Here is the error
Traceback (most recent call last):
File "ctetest./RegressionTest/WebsocketTest\test_18_RecTimer.py", line 385, in test009_recTimer_Start_withAudio
response = client.endSession(sessionId = sessionId, requestId = requestId_2)
File "ctetest./RegressionTest/WebsocketTest../.././CTESetupClass\WebsocketCl
ient.py", line 528, in endSession
message = self.getNextResponse(requestId)
File "ctetest./RegressionTest/WebsocketTest../.././CTESetupClass\WebsocketCl
ient.py", line 49, in wrapper
raise ret
TypeError: string indices must be integers, not str
you have two statements in your code that look very similar:
message = json.loads(self.messagesList[len(self.messagesList)-1])
and then further down:
message = self.messagesList[len(self.messagesList)-1]
The first will set message to a json object (probably dict) where the second one assigns message to a string, I'm assuming this is not intended and the cause for your error.