If for some reason the create_blob_from_path method runs into issues, what would be the proper way to handle the exception? As in, when you are writing your 'except' block, it would be except ??what?? as e: ? I want to be able to handle that exception when a blob upload fails so it can email me via sendmail so I know a backup that is attempting to be offloaded to Azure storage has failed.
Thank you!
I looked for & searched some exception names on the source codes of Azure Python Storage SDK, there are not any exceptions defined about failure of uploading blob. However, per my experience, Azure Storage SDK for Python wrapped REST APIs of Storage Services via the python package requests to do related operations. So any failure will cause the requests exceptions, you can try to catch the requests root exception requests.exceptions.RequestException as below to do the next action like sendmail.
import requests
try:
blobservice.create_blob_from_path(container_name, blob_name, file_path)
except requests.exceptions.RequestException:
sendmain(...)
Related
I am trying to upload multiple files at once using the google cloud python api, and am having trouble with it. Below is my code:
client = storage.Client.from_service_account_json("path_to_json")
bucket = client.get_bucket(bucket_name)
with client.batch():
for i in range(10):
try:
blob = bucket.blob("my_blob")
blob.upload_from_filename("path_to_file",content_type="image/jpeg")
except Exception as e:
raise e
However, this is telling me that there are no deferred requests to make. I have tried doing it without client.batch() and it works but is too slow. I was wondering if anyone has ever encountered this.
Thanks
According to this issue, the back-end API doesn't support batching "media" operations
I am developing a django app which communicates with several Amazon Web Services.
So far I am having trouble dealing with and catching exceptions thrown by the boto3 client. What I am doing seems unnecessarily tedious:
Example:
client = boto3.client('sns')
client.create_platform_endpoint(PlatformApplicationArn=SNS_APP_ARN, Token=token)
this might throw an botocore.errorfactory.InvalidParameterException if e.g. the token is bad.
client.get_endpoint_attributes(EndpointArn=endpoint_arn)
might throw an botocore.errorfactory.NotFoundException.
First, I can't find these Errors anywhere in code, so they are probably generated somewhere. Bottom line: I can't import it and catch it as usual.
Second, I found one way to catch the error here using:
try:
# boto3 stuff
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == 'NotFound':
# handle exception
else:
raise e
But I have to remove the Exception part of the error name. Seems very random and I have no clue whether I would remove the Error in botocore.exceptions.ParamValidationError if I wanted to catch that one. So it's hard to generalize.
Another way to catch the error is using the boto3 client object I got:
try:
# boto3 stuff
except client.exceptions.NotFoundException as e:
# handle exception
This seems the cleanest way so far. But I don't always have the boto3 client object at hand where I want to catch the error. Also I am still only trying things out, so it's mostly guess work.
Does anybody know how boto3 errors are supposed to be handled?
Or can point me towards some coherent documentation which mentions the errors above? Thanks
You've summarized the situation well. The old boto had a simple hardcoded approach to supporting AWS APIs. boto3, in what appears to be an attempt to reduce the overhead of keeping Python client synced with evolving features on the various apis, has been more squishy around exceptions, so the ClientError approach you outlined above used to be the canonical way.
In 2017 they introduced the second mechanism you highlight: 'modeled' exceptions available on the client.
I am not familiar with SNS but in my experience with other AWS products, the ClientError naming matches up with the HTTP apis, which tend to be well documented. So I would start here: https://docs.aws.amazon.com/sns/latest/api/Welcome.html
It looks like the new-style modeled exceptions are generated from service definition files that live in botocore module. I can't find any documentation about it, but go browse around the AWS service models in https://github.com/boto/botocore/tree/master/botocore/data.
Also, it's good to know that if you are not (in contrast to OP's code) dealing directly with the low-level client, but instead are using a high-level AWS ServiceResource object, a low-level client is still easily available at my_service_resource.meta.client so you can handle exceptions like this:
try:
my_service_resource.do_stuff()
except my_service_resource.meta.client.exceptions.NotFoundException as e:
# handle exception
Use Boto3 exceptions: https://www.programcreek.com/python/example/97944/boto3.exceptions
client = boto3.client('logs')
try:
client.create_log_group(logGroupName=LOG_GROUP)
except client.exceptions.ResourceAlreadyExistsException:
pass
I am writing python script using the Google Sheets apiAPI. It reads data and writes it to a new file, shares that file with a specified email and returns the id of the new file.
def read_sheet(self,spreadsheetId):
try:
result=self.service.spreadsheets().get(spreadsheetId=spreadsheetId,includeGridData=True,fields='namedRanges,properties,sheets').execute()
return result
except apiclient.errors.HttpError as e:
traceback.print_exc()
print(e)
sys.exit(1)
def create_spreadsheet(self,data,email):
try:
newid=self.service.spreadsheets().create(body=data,fields='spreadsheetId').execute()
newid=newid.get('spreadsheetId')
self.give_permissions(email,newid)
return newid
except apiclient.errors.HttpError as e:
traceback.print_exc()
print(e)
sys.exit(1)
This code works very well, but not with 100% accuracy. Sometimes I get a 500 Internal Server Error, but the file is created in my account. I found a similar Stack Overflow question (Getting 500 Error when using Google Drive API to update permissions), but it didn't help. I want to know the exact reason for this. Can anyone help?
EDIT1:
This is the exact error message
https://www.googleapis.com/drive/v3/files/349hsadfhSindfSIins-rasdfisadfOsa3OQmE/permissions?sendNotificationEmail=true&alt=json&transferOwnership=false
returned "Internal Error. User message: "An internal error has
occurred which prevented the sharing of these item(s): Template"">
As hinted to above in DaimTo's comment, the error is due to Google Drive still processing the create request while you're trying to add the permission to share the (new) file with. Remember, when you add a file to Drive, Google's servers are still working on the file-create as well as making it accessible globally. Once the flurry of activity settles down, then adding additional users to the document shouldn't be a problem.
You can see from this Drive API documentation page a description of the (500) error you received as well as the recommended course of action which is to implement exponential backoff, which is really just saying you should pause a bit before trying again & extending that delay each time you get the same error. He also pointed to another SO Q&A which you can look at. Another resource is this descriptive blog post. If you don't want to implement it yourself, you can try the retrying or backoff packages.
NOTE: You didn't show us all your code, but I changed the title of this question to more accurately reflect that you're using the Drive API for adding the permissions. While you've used the Sheets API to create the Sheet with, realize that you can just do this all with the Drive API (and not use the Sheets API at all unless you're doing spreadsheet-oriented operations. The Drive API is for all file-related operations like sharing, copying, import/export, etc.)
Bottom line is that you can create Sheets using either API, but if you're not doing anything else with the Sheets API, why bother making your app more complex? If you want to see how to create Sheets with both APIs, there's a short segment in my blog post that covers this... you'll see that they're nearly identical but using the Drive API does require one more thing, the MIMEtype.
If you want to learn more about both APIs, see this answer I gave to a related question that features additional learning resources I've created for both Drive and Sheets, most of which are Python-based.
I guess I'm late but just in case: just add a few seconds delay between the create request and the give permissions one. For me it works making the thread sleep for 10 seconds. Try this:
def create_spreadsheet(self,data,email):
try:
newid=self.service.spreadsheets().create(body=data,fields='spreadsheetId').execute()
newid=newid.get('spreadsheetId')
time.sleep(10)
self.give_permissions(email,newid)
return newid
except apiclient.errors.HttpError as e:
traceback.print_exc()
print(e)
sys.exit(1)
I am writing a simple API client in Python and I am wondering what exception should I raise if the remote server is not happy.
The API itself is very poorly documented (I don't know all the possible error messages so I can't just define custom classes for all of them), I am letting Requests handle HTTP-level errors which raises HTTPError in those cases, but what should I raise if the server just supplies an error key in the JSON response?
I am currently using Exception but that feels quite broad, I'd like to know whether there's a better alternative.
Regards.
Yes, just raising Exception is likely too broad.
Usually a module should define his own base exception class:
class MyAPIClientError(Exception):
pass
And then you can subclass that:
class RemoteServerNotHappyError(MyAPIClientError):
pass
RemoteServerNotHappyError should probably mention something about what the server's json returned that was not expected by your API client class. Add in the relevant error messages as you see fit.
This means users of your library can catch these specific exceptions (which they might know how to handle), without having to catch every exception (they surely don't know how to handle every possible failure mode).
I'm trying to push a file (put blob request) to the Azure CDN Blob storage using the python SDK. It works no problem, I just can't figure out how to read the header information in the response. According to the docs, its supposed to send back a 201 status if it is successful.
http://msdn.microsoft.com/en-us/library/azure/dd179451.aspx
http://azure.microsoft.com/en-us/documentation/articles/storage-python-how-to-use-blob-storage/
from azure.storage import BlobService
blob_service = BlobService(account_name='accountnamehere', account_key='apikeyhere')
file_contents = open('path/to/image.jpg').read()
blob_service.put_blob(CONTAINER, 'filename.jpg', file_contents, x_ms_blob_type='BlockBlob', x_ms_blob_content_type='image/jpeg')
Any help is greatly appreciated.
Thanks
You can't read the response code.
Source code for SDK is available on GitHub, and in the current version put_blob() function does not return anything.
Do you need to read it though? If put_blob completes succesfully, then your code continues from the next statement. If it were to fail, then the SDK will raise an exception which you can then catch.
You could verify your exception/error handling by using a wrong access key for example.