I'm willing to implement a lazy loading approach to load images stored in an "images" folder inside an azure storage account.
I have a container in my flutter app where whenever the user scrolls down the bottom new 10 images will be loaded from the storage based on the most recent(timestamp).
I looked into this sample retrieved from: https://azuresdkdocs.blob.core.windows.net/$web/python/azure-storage-blob/12.0.0b5/index.html#id20
from azure.storage.blob.aio import ContainerClient
container = ContainerClient.from_connection_string(conn_str="my_connection_string", container_name="my_container")
blob_list = []
async for blob in container.list_blobs():
blob_list.append(blob)
print(blob_list)
But it's not what I need.I am looking for a way to make a get request that will retrieve me new set of images whenever the function is invoked..
Thankful for suggestions!
I was able to implement a lazy loading approach by using the marker continuation object
Example:
mark=req.params.get('NextMarker')
entit = table_service.query_entities('UserRequests','PartitionKey eq \'' + emailAddress + '\'',num_results=21,select= '..', marker=mark)
Dict = {"NextMarker": entit.next_marker}
return json.dumps(Dict)
This way I am able to send the marker in the http get request every time to get the second batch.
I hope this helps someone one day!
If you want to list blobs by blob creation time, unfortunately, it is not supported by Azure list blobs API(SDKs are based on APIs). Blob creation time belongs to blob properties, and as the official doc indicates, blob properties can't be set as a request param.
So if you want to fetch all new images for each request, maybe you should get a blob list first and sort them yourself and cut out the items that you need. There will be some extra codes that you need to write. But if you use Azure PowerShell to do that, you can implement the whole process easier. You can refer to this similar requirement.
Related
I'm new in Azure so I have some gaps in my knowledge. I've been looking for answer but couldn't find one.
Within my app function which is HTTP TRIGGERED I do make some api call to retrieve json. If everything goes fine, then the function returns this json.
Pseudo code:
return func.HttpResponse(
json.dumps(apiReturn),
status_code=200
)
I'd like then to use this json (return) to store it in nsql db or csv for further processing. Is this possible somehow or I just need to create separate json file on function stage and upload it to blob storage?
Yes! you can store it into blob storage using Blob bindings found here: https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob-input?tabs=python
You can also take a look at this to find more examples:
https://github.com/yokawasa/azure-functions-python-samples/tree/master/v2functions/queue-trigger-blob-in-out-binding
Long story short, what you have to do is:
Change your function.json to take in blobs
Change your code to take in an input blob
Add your blob keys to both your local.settings.json as well as on your function app in the portal configuration.
and that should get you where you need to go!
I've written some python code to extract data from a rest api and load them in an Azure SQL database. But this process is taking almost half a hour for 20,000 lines. Is there a more efficient way of doing this? I'm thinking maybe extract the data as json file and put it in blob storate then use azure data factory to load the data into SQL, but have no idea how to code this way.
def manualJournalLineItems(tenantid):
endpoint = "api.xro/2.0/manualjournals/?page=1"
result = (getAPI(endpoint,token,tenantid))
page = 1
while result['ManualJournals']:
endpoint = "api.xro/2.0/manualjournals/?page="+str(page)
result = (getAPI(endpoint,token,tenantid))
for inv in result['ManualJournals']:
for li in inv['JournalLines']:
cursor.execute("INSERT INTO [server].dbo.[Xero_ManualJournalLines](ManualJournalID,AccountID,Description,LineAmount,TaxAmount,AccountCode,Region) VALUES(?,?,?,?,?,?,?)",inv['ManualJournalID'],li['AccountID'],li.get('Description',''),li.get('LineAmount',0),li.get('TaxAmount',0),li.get('AccountCode',0),tenantid)
conn.commit()
page = int(page)+1
If Python is not a mandatory requirement, yes, you can use Data Factory.
You will need to create a pipeline with the following components:
'Copy Data' Activity
Source Dataset (REST API)
Sink Dataset (Azure SQL)
** Also may I know where is your REST API hosted? Is it within Azure through App Service? If not, you will also need to setup a [Self-Hosted Integration Runtime]1
You can refer to the steps here which copies data from Blob storage to Azure SQL
You can also follow my screenshots below which is to create REST API as a Source.
Create a new pipeline.
Type 'copy' in the 'Activity' search box. Drag the 'Copy Data' activity to the pipeline
Click on 'Source' tab, and click on 'New' to create a new Source Dataset.
Type 'REST' in the 'data source' search box.
In the 'REST' dataset window, click on 'Connection' tab. Click on 'New' to create a linked service to point to the REST API.
Here fill up the credentials to the REST API.
Continue setting up the Sink dataset to point to the Azure SQL and test out your pipeline to make sure it works. Hope it helps!
Found the answer. append() the values to a list and insert the list into SQL with executemany()
Is it possible in Django to store image files in different buckets/containers in external object storage OpenStack Swift?
I have an issue related with creating proper way to upload image files through REST service request. The user is able to POST 'Task' instance via first endpoint, containing name, description, owner_id etc. He is also able to POST images via another endpoint, which has a relation many to one with Task.
Images should be stored in OpenStack Swift in my case (server already exists, was set up etc.) in unique containers/buckets as follows: owner_id_task_id. It is related, that users can upload files with the same names and extensions, but different content.
Another part will be sending also to the same containers in OpenStack Swift a files from Celery worker tasks (some processing based on uploaded files).
My goal is to achieve dynamically-created/overrided at runtime container structure, for storing raw images, and also post-processed images.
Any ideas how this problem can be solved?
Thanks for the help!
Yes. This is possible. You can use FileField.storage to configure which storage to use in individual model fields.
I am trying to make a form for adding users that will store user information to my database.
I want to upload an image file using Cloud Endpoints (python). I do not have any idea how to do it.
What will be the input class (request class) and output class (response class)?
#endpoints.method(inputclass, outputclass,
path='anypath', http_method='GET',
name='apiname')"
What url will I provide in the action= form field for uploading the image? How can I then show the file?
You have two ways to store data files (include image file).
The first one is to convert your image in base 64 format, and store it on datastore (Not the best).
The second one, is to store your image file in Google Cloud Storage (That is the best) or the blobstore (By Google themself, there is no good reason to use Blobstore).
So you have to store, your image file to Google Cloud Storage with your endpoint: https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/?hl=fr
Personnaly, I use a servlet (App Engine) to store image in GCS. My endpoints call my servlet and pass image in parameter, and my servlet store image in GCS. It works very well =)
Hope I help you..
I am building a web application based on Google App Engine using python; however I have seem to have hit a roadblock.
Currently, my site fetches remote image URLs from an external website. The URLs are placed in a list and sent back to my application. I want to be able to store the respective images (not the URLs) in my datastore; in order to avoid having to fetch the remote images each time as well as having to deal with broken links.
The solutions I have found online all deal with a user having to upload their own images. Which I tried implementing, but I am not sure what happens to an uploaded image (or how it gets converted into a blob) once the user hits the submit button.
To my understanding, a blob is a collection of binary data stored as a single entity (from Wikipedia). Therefore, I tried using the following:
class rentalPropertyDB(ndb.Model):
streetNAME = ndb.StringProperty(required=True)
image = ndb.BlobProperty(default=None)
class MainPage(BaseHandler):
def get(self):
self.render("index.html")
def post(self):
rental = rentalPropertyDB()
for image in img_urls:
rental.image = urlfetch.Fetch(image).content
rental.put()
The solution to this question: Image which is stored as a blob in Datastore in a html page
is identical to mine, however the solution suggests to upload the image to the blobstore and uses:
upload_files = self.get_uploads('file')
blob_info = upload_files[0]
This confuses me because I am not sure what exactly 'file' refers to. Would I replace 'file' with the URL of each image? Or would I need to perform some operation to each image prior to replacing?
I have been stuck on this issue for at least two days now and would greatly appreciate any help that's provided. I think the main reason why this is confusing me so much is because of the variety of methods used in each solution. I.e. using Google Cloud Storage, URLFetch, Images API, and the various types of ndb.Blobstore's (BlobKeyProperty vs. BlobProperty) and etc.
Thank you.
Be careful with blob inside Models. A Model cannot be more than 1 Mo including the blobproperty.
If we listen Google, there is no good reason to use the blobstore. If you can, use Google Cloud Storage. It made to store files.