upload hyperlink data to azure binding - python

I am trying to stream data from hyperlink destination to azure storage. I have to do this via binding since I want to run this from azure function App.
file -- function.json:
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "$return"
},
{
"type": "blob",
"direction": "out",
"name": "outputBlob",
"path": "samples-workitems/{rand-guid}",
"connection": ""
}
]
}
file -- init.py:
import logging
import cdsapi
import azure.functions as func
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
def main(req: func.HttpRequest, outputBlob:func.Out[func.InputStream]) -> func.HttpResponse:
logging.info('Python HTTP trigger function is about to process request.')
try:
source_blob="http://www.africau.edu/images/default/sample.pdf"
with open(source_blob, "rb") as data:
print(data)
outputBlob.set(data)
except Exception as ex:
logging.info(" error!", ex, "occurred.")
return func.HttpResponse(
"This HTTP triggered function executed successfully.",
status_code=200
)
I have tested binding and it works. When I simply do outputBlob.set("sample string") data is streamed as it should be.
I am stuck with converting data from hyperlink to bytes(or blob). While running code above, i get error Exception: TypeError: not all arguments converted during string formatting. Any help in converting this and uploading to azure storage is appreciated.

Problem is you were trying to read the File from URL with open(source_blob, "rb") as data: which of course won't work since open is for local files only. I have changed your code as below using requests module to get the remote URL response and set the content to blob.
import requests
source_url="http://www.africau.edu/images/default/sample.pdf"
with requests.get(source_url, stream=True) as r:
r.raise_for_status()
outputBlob.set(r.content)

Related

How to overwrite a file in azure cosmos DB using azure functions(python) HTTP trigger

I am able to write a file to cosmos DB with the help of output binding, but what I need is to know how to overwrite the existing file that is already in cosmos DB
My code looks like this
import azure.functions as func
def main(req: func.HttpRequest, doc: func.Out[func.Document]) -> func.HttpResponse:
request_body = req.get_body()
doc.set(func.Document.from_json(request_body))
return 'OK'
and my output binding looks like this
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "function",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "cosmosDB",
"direction": "out",
"name": "doc",
"databaseName": "demodb",
"collectionName": "data",
"createIfNotExists": "true",
"connectionStringSetting": "AzureCosmosDBConnectionString"
},
{
"type": "http",
"direction": "out",
"name": "$return"
}
]
}
All I want is to know how could I overwrite the existing file that is cosmos DB
Please help me with some sample code....
thanks.
I am able to write a file to cosmos DB with the help of output binding, but what I need is to know how to overwrite the existing file that is already in cosmos DB
As far as I know, currently there is no possible way to overwrite the existing file in a cosmos DB.
If you want to update existing document in cosmos DB, you need to query (or read), modify, and then replace the document.
you can refer the SDK for azure cosmos db.
I have followed this blog created by Evan Wong and I am able to append the file details.
def main(req: func.HttpRequest, cosmos:func.DocumentList) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
User_details = []
for user in cosmos:
User_details = {
"id": user['id'],
"name": user['name']
}
users_json.append(User_details)
return func.HttpResponse(
json.dumps(User_details),
status_code=200,
mimetype="application/json"
)

How do i receive json string from Azure Function to save like json file in blob storage

I have a flow on Power Automate. It will post a json-string to my azure function. How can i write this json to a json file in blobs storage using Azure Data Factory or only Azure Function ?
Below is a sample where you can save JSON through the azure function.
For example, Considering this to be my workflow
I am using HTTP trigger in my case and sending the JSON sample to my function app which has output binding as blob storage and My Http function App looks like this:-
init.py
from http.client import HTTPResponse
import logging
import azure.functions as func
def main(req: func.HttpRequest,outputblob: func.Out[str]) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
print(str(req.get_json()))
outputblob.set(str(req.get_json()))
function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"name": "outputblob",
"type": "blob",
"path": "outputcontainer/sample.json",
"connection": "AzureWebJobsStorage",
"direction": "out"
}
]
}
RESULT :
In power Automate
In function app
In Storage Account

How to upload a large string in an Azure Blob?

Right now I'm trying to figure out how to work with Azure, and now I'm stuck in a problem while storing my data in the storage account.
I have three strings and want to store each of them in a separate blob. With the first two, my code works fine, but the third one causes some retries and ends with a timeout.
My code is running within an Azure function.
Here is a minimal example:
from azure.storage.blob import BlobClient
blob_client = BlobClient.from_connection_string(
conn_str. = '<STORAGE_ACCOUNT_CONNECTION_STRING>',
container_name = '<CONTAINER_NAME>',
blob_name. = '<NAME_OF_BLOB>',
)
dic_blob_props = blob_client.upload_blob(
data = '<INFORMATION_THAT_SHOULD_GO_TO_THE_BLOB>',
blob_type = "BlockBlob",
overwrite = True,
)
The for the first two strings everything works fine but the third fails. The strings have the following length:
len(s_1) = 1246209
len(s_2) = 8794086
len(s_3) = 24518001
Most likely it is because the third string is too long, but there must be a way to save it, right?
I have already tried to set the timeout time within the .upload_blob method by timeout=600, but this has not changed the result at all, nor the time until a new attempt to write is made.
The error is:
Exception: ServiceResponseError: ('Connection aborted.', timeout('The write operation timed out'))
If you have any ideas on the problem pleast let me know :-)
In my case, the problem disappeared after I deployed the function in the cloud. It seems that there was a problem debugging with Visual Studio code.
On my side, I don't have the problem. You can have a look of my code:
__init__.py
import logging
import azure.functions as func
def main(req: func.HttpRequest,outputblob: func.Out[func.InputStream],) -> func.HttpResponse:
logging.info('This code is to upload a string to a blob.')
s_3 = "x"*24518001
outputblob.set(s_3)
return func.HttpResponse(
"The string already been uploaded to a blob.",
status_code=200
)
function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"route": "{test}",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "$return"
},
{
"name": "outputblob",
"type": "blob",
"path": "test1/{test}.txt",
"connection": "str",
"direction": "out"
}
]
}
local.settings.json
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "",
"FUNCTIONS_WORKER_RUNTIME": "python",
"str":"DefaultEndpointsProtocol=https;AccountName=0730bowmanwindow;AccountKey=xxxxxx==;EndpointSuffix=core.windows.net"
}
}
Then I hit the endpoint http://localhost:7071/api/bowman, it uploads the string to blob and don't have time out error:
So, I think the problem is related with the method you use.

I'm getting Value 'func.Out' is unsubscriptable trying to write in an azure blob storage using a python function

I want to write in an azure blob storage using Azure Functions in Python.
I'm using the output blob storage bindings for Azure Functions.
My function.json:
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"name": "inputblob",
"type": "blob",
"path": "{containerName}/{blobName}.json",
"connection": "MyStorageConnectionAppSetting",
"direction": "in"
},
{
"name": "outputblob",
"type": "blob",
"path": "{containerName}/{blobName}.json",
"connection": "MyStorageConnectionAppSetting",
"direction": "out"
},
{
"type": "http",
"direction": "out",
"name": "$return"
}
]
}
And my Python code looks like this:
import logging
import azure.functions as func
import azure.storage.blob
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
import json, os
def main(req: func.HttpRequest, inputblob: func.InputStream, outputblob: func.Out[func.InputStream]) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
# Initialize variable for tracking any changes
anyChanges= False
# Read JSON file
jsonData= json.loads(inputblob.read())
# Make changes to jsonData (omitted for simplicity) and update anyChanges
# Upload new JSON file
if anyChanges:
outputblob.set(jsonData)
return func.HttpResponse(f"Input data: {jsonData}. Any changes: {anyChanges}.")
However, this isn't working at all, with the following error being thrown:
Value 'func.Out' is unsubscriptable
Another guy in December already asked for a resolution to the same issue, but the answer does not solve the issue
Same issue using 3.7.0 switched to 3.6.8 to solve it
I've had the same issue, and I actually found out that's the problem is resolved by removing pylint library:
pip uninstall pylint
I am not sure if that is a problem in pylint or if Azure Functions cannot coexist with pylint in Python 3.7, but removing it did the trick for me.
Instead of uninstalling pylint, you can simply disable that specific warning:
# pylint: disable=unsubscriptable-object
def main(req: func.HttpRequest, inputblob: func.InputStream, outputblob: func.Out[func.InputStream]) -> func.HttpResponse:
# your code

How can I set Queue Storage message TTL in the context of an Azure Function output binding in Python?

I have a python azure function with a queue output binding. I am successfully using this binding to queue messages from within the function. Is it possible to set the message TTL on the underlying queue or on the message itself? I don't need to set it on a per message basis, but will do it that way if that is the only option.
host.json
{
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[1.*, 2.0.0)"
}
}
function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "$return"
},
{
"type": "queue",
"direction": "out",
"name": "msg",
"queueName": "predictions",
"connection": "AzureWebJobsStorage"
}
]
}
function code
import json
import logging
import azure.functions as func
from graphene import Schema
from .helpers import responses
from .schema.Query import Query
def main(req: func.HttpRequest, msg: func.Out[func.QueueMessage]) -> func.HttpResponse:
logging.info('Executing GraphQL function.')
try:
query = req.get_body().decode()
except ValueError:
pass
if query:
schema = Schema(Query)
results = schema.execute(query)
response = responses.graphql(results)
# Write response to azure queue storage
message = responses.storage(query, response)
if message:
msg.set(message)
return response
else:
return responses.bad_request(
'Please pass a GraphQL query in the request body.')
For now this only supports c# language, you could bind it to CloudQueue type. If it's other laguage you have to use the SDK method to implement. If you insist this feature, you could got to this github issue to comment you requirements.
And below is my test code to set TTL in a HTTP trigger function with azure-storage-queue 2.1.0.
import logging
import azure.functions as func
from azure.storage.queue import QueueService
import os
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
queue_service = QueueService(connection_string=os.environ['AzureWebJobsStorage'])
message = req.params.get('message')
if not message:
try:
req_body = req.get_json()
except ValueError:
pass
else:
message = req_body.get('message')
if message:
queue_service.put_message('myqueue',message,None,300,None)
return func.HttpResponse(f" {message}!")
else:
return func.HttpResponse(
"Please pass message in the request body",
status_code=400
)

Categories