Python - Azure function service bus trigger batch processing - python

I am using Azure function service bus trigger in Python to receive messages in batch from a service bus queue. Even though this process is not well documented in Python, but I managed to enable the batch processing by following the below Github PR.
https://github.com/Azure/azure-functions-python-library/pull/73
Here is the sample code I am using -
function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"name": "msg",
"type": "serviceBusTrigger",
"direction": "in",
"cardinality": "many",
"queueName": "<some queue name>",
"dataType": "binary",
"connection": "SERVICE_BUS_CONNECTION"
}
]
}
__init__.py
import logging
import azure.functions as func
from typing import List
def main(msg: List[func.ServiceBusMessage]):
message_length = len(msg)
if message_length > 1:
logging.warn('Handling multiple requests')
for m in msg:
#some call to external web api
host.json
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[3.3.0, 4.0.0)"
},
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
},
"batchOptions": {
"maxMessageCount": 100,
"operationTimeout": "00:01:00",
"autoComplete": true
}
}
}
}
After using this code , I can see that service bus trigger is picking up messages in a batch of 100 (or sometimes < 100) based on the maxMessageCount but I have also observed that most of the messages are ending up in the dead letter queue with the MaxDeliveryCountExceeded reason code. I have tried with different values of MaxDeliveryCount from 10-20 but I had the same result. So my question is do we need to adjust/optimize the MaxDeliveryCount in case of batch processing of service bus messages ? How both of them are related ? What kind of change can be done in the configuration to avoid this dead letter issue ?

From what we discussed in the comments, this is what you encounter:
Your function app is fetching 100 messages from ServiceBus (prefetchCount) and locking them for a maximum of maxAutoRenewDuration
Your function code is processing messages one at a time at a slow rate because of the API you call.
By the time you finish a batch of messages (maxMessageCount), the lock already expired which is why you have exceptions and the message gets redelivered again. This eventually causes MaxDeliveryCountExceeded errors.
What can you do to improve this?
Reduce maxMessageCount and prefetchCount
Increase maxAutoRenewDuration
Increase the performance of your API (how to do that would be a different question)
Your current code would be much better off by using a "normal" single message trigger instead of the batch trigger
PS: Beware that your function app may scale horizontally if you are running in a consumption plan, further increasing the load on your struggling API.

Related

Amazon Lex v2 Welcome message

hello everyone I almost finished a bot with lex v2 that uses lambda, I did the integration using the kommunicate client to do before, but now I need the chatbot to start with an intent automatically, this is my first problem.
Also I would like to know if on lambda in the answers there is a way to send a response that makes an elicit slot, and then after 5 seconds send another to move to confirmation intent on lex v2.. I tried with time and asyncio, but it seems that the code does not go on, I can only get the first answer with the slot elicit, I wish that after about 5 seconds I refer to confirmation intent on lex v2, this is my code:
if carFound is not None:
# if resp['total'] < 30:
# print('Conferma intento')
# carDialog = {
# "type": "ConfirmIntent"
# }
# else:
# print('Filtri args')
# carDialog ={
# "type": "ElicitSlot",
# "slotToElicit": "Args"
# }
response = {
"sessionState": {
"dialogAction": {
"type": "ElicitSlot",
"slotToElicit": "Args"
},
"intent": {
'name': intent,
'slots': slots,
"state": "Fulfilled",
}
},
"messages": carFound,
}
practically after every call to my api that sends me carFound as a payload with all the machines, I should verify that when resp['total'] and less than 30 I refer in addition to the answers also to confirmation intent after some time.
as i said yet i tried with sleep() function of python, and i still have only the response that has the elicitslot,maybe sleep and asyncio are not well for lexv2...
I verified through the test of lambda, with my inputs that the condition of <30 is true, so the problem is on the response i think.
for the welcome i don't know how i can do, i want that my bot start with the intent Welcome for example, without write nothing on my kommunicate client that is on a website.

Azure function app not triggering from an Event Hub

I have a event hub which is in Subscription A and a function app in Subscription B, i am trying to trigger the function app from the event hub in Subscription A, as per my research this should be possible and the correct connection string must be provided in the configuration of function app. I have done this but for some reason i am not able to trigger the function app.
Below is my function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"type": "eventHubTrigger",
"name": "event",
"direction": "in",
"eventHubName": "%eventHubName%",
"connection": "TestBench",
"cardinality": "one",
"consumerGroup": "$Default"
},
{
"type": "eventHub",
"name": "outputHub",
"direction": "out",
"connection": "outputConnection"
}
I have double checked the "TestBench" (eventhubs) connection string and also eventhub's name, they are correct.
Below is my function app code in __init__.py :
def main(event: func.EventHubEvent, outputHub: func.Out[List[str]]):
data=json.loads(event.get_body().decode('utf-8'))
logging.info(data)
Please verify if you have configured eventHubName property in the application setting of your function app as you have defined the binding as
"eventHubName": "%eventHubName%"
In case if this is correct then please validate whether the connection string is configured correctly or not.
I will suggest you to review the Diagnostic and solve problem blade on your function app that will help you diagnose the issue and the recommend solution to resolve your issue.
Please review the python example here and test the same at your end.

How to access Azure Service Bus using Function App identity

I am following the steps listed here, but for python code:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-identity-based-connections-tutorial-2
Objective is to create a simple (hello world) function app which is triggered by Azure Service Bus message queue using identity-based connection. Function app works fine when ASB is reference via connection string, but gives this error when trying to connect via managed service identity of function app (used the specific configuration pattern __fullyQualifiedNamespace). MSI has been granted Role (Azure Service Bus Data Receiver) on ASB.
Microsoft.Azure.WebJobs.ServiceBus: Microsoft Azure WebJobs SDK ServiceBus connection string 'ServiceBusConnection__fullyQualifiedNamespace' is missing or empty.
Function code (autogenerated)
import logging
import azure.functions as func
def main(msg: func.ServiceBusMessage):
logging.info('Python ServiceBus queue trigger processed message: %s',
msg.get_body().decode('utf-8'))
function.json (connection value modified based on ms docs)
{
"scriptFile": "__init__.py",
"bindings": [
{
"name": "msg",
"type": "serviceBusTrigger",
"direction": "in",
"queueName": "erpdemoqueue",
"connection": "ServiceBusConnection"
}
]
}
host.json (version modified based on ms docs)
{
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[3.3.0, 4.0.0)"
}
}
To use a managed identity, you'll need to add a setting that identifies the fully qualified namespace of your Service Bus instance.
For example, in your local.settings.json file for local development:
{
"Values": {
"<connection_name>__fullyQualifiedNamespace": "<service_bus_namespace>.servicebus.windows.net"
}
}
Or in the application settings for your function when deployed to Azure:
<connection_name>__fullyQualifiedNamespace=<service_bus_namespace>.servicebus.windows.net
This is mentioned only briefly in the tutorial that you linked. The Microsoft.Azure.WebJobs.Extensions.ServiceBus documentation does covers this a bit better in the Managed identity authentication section.

How to read chrome console using python without selenium?

I would like to read chrome's js console using Python3 without any webdriver such as selenium (bot detection and stuff).
I've tried Chrome DevTools Protocol python libraries such as chromewhip, pychrome and PyChromeDevTools, but I'm unable to read any data from the console.
I want to read Runtime.consoleAPICalled or Log.entryAdded, but I don't know how to implement these callbacks as the documentation for these libraries doesn't specify any of that. Also there are no examples to be found either.
Does anyone know how to properly access these events or some other library which provides it?
#kundapanda could you at least post a snippet of the code that worked for you.
I want to read Runtime.consoleAPICalled or Log.entryAdded, but I don't
know how to implement these callbacks
The following assumes (per your question phrasing) that you're able to send and receive debug protocol messages on the stream that's open to the web debugger endpoint.
After you send debug protocol messages Runtime.enable and Log.enable messages, the Runtime.consoleAPICalled and Log.entryAdded "events" you are looking for are represented by messages you receive on the same debugging channel.
You may need to match the console event messages with the execution context (seen in the Runtime.enable response) by examining the executionContextId field in the received event messages. The log events are not associated with any single execution context. All of these "event" messages will have Id=0, which helps to recognize they're "event" messages and not response messages.
Here are a couple of sample messages received from Chrome (formatted as JSON with arbitrary field order):
Console API event message:
{
"method": "Runtime.consoleAPICalled",
"params":
{
"type": "warning",
"args": [
{ "type": "string",
"value": "Google Maps JavaScript API warning: NoApiKeys https://developers.google.com/maps/documentation/javascript/error-messages#no-api-keys"
}],
"executionContextId": 1,
"timestamp": 1618949706735.553,
"stackTrace":
{
"callFrames": [
{
"functionName": "TA.j",
"scriptId": "206",
"url": "https://maps.googleapis.com/maps-api-v3/api/js/44/10/util.js",
"lineNumber": 228,
"columnNumber": 26
} ]
}
}
},
"id": 0
}
Log event message:
{
"method":"Log.entryAdded",
"params":
{
"entry":
{
"source":"javascript",
"level":"warning",
"text":"The deviceorientation events are blocked by permissions policy. See https://github.com/w3c/webappsec-permissions-policy/blob/master/features.md#sensor-features",
"timestamp":1.6189509536801208e+12
}
},
"id": 0
}

Resource has been exhausted Google Cloud Speech

I am trying to transcribe a 45 min long audio file with google cloud speech but I keep getting
Resource has been exhausted (e.g. check quota)
I have the free credit that the api offers you for a year. Tried it in the api explorer and in python where I will be using it but the result is the same. This is the request I send:
{
"audio": {
"uri": "gs://speech_summarization/mq.3gp"
},
"config": {
"encoding": "AMR",
"sampleRate": 8000
}
}
and response:
429
- Show headers -
{
"error": {
"code": 429,
"message": "Resource has been exhausted (e.g. check quota).",
"status": "RESOURCE_EXHAUSTED"
}
}
I saw similar problems solved by cutting the video in shorter versions but with 10 min it didn't work for me. Any ideas?
Files under 1 minute are working because, in addition to an absolute quota on audio sent to the API there are specific limits on streams of audio:
https://cloud.google.com/speech/limits
I haven't had much luck finding a free version (even on trial) of transcription products like those offered by Nuance.

Categories