I am developing a django app which communicates with several Amazon Web Services.
So far I am having trouble dealing with and catching exceptions thrown by the boto3 client. What I am doing seems unnecessarily tedious:
Example:
client = boto3.client('sns')
client.create_platform_endpoint(PlatformApplicationArn=SNS_APP_ARN, Token=token)
this might throw an botocore.errorfactory.InvalidParameterException if e.g. the token is bad.
client.get_endpoint_attributes(EndpointArn=endpoint_arn)
might throw an botocore.errorfactory.NotFoundException.
First, I can't find these Errors anywhere in code, so they are probably generated somewhere. Bottom line: I can't import it and catch it as usual.
Second, I found one way to catch the error here using:
try:
# boto3 stuff
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == 'NotFound':
# handle exception
else:
raise e
But I have to remove the Exception part of the error name. Seems very random and I have no clue whether I would remove the Error in botocore.exceptions.ParamValidationError if I wanted to catch that one. So it's hard to generalize.
Another way to catch the error is using the boto3 client object I got:
try:
# boto3 stuff
except client.exceptions.NotFoundException as e:
# handle exception
This seems the cleanest way so far. But I don't always have the boto3 client object at hand where I want to catch the error. Also I am still only trying things out, so it's mostly guess work.
Does anybody know how boto3 errors are supposed to be handled?
Or can point me towards some coherent documentation which mentions the errors above? Thanks
You've summarized the situation well. The old boto had a simple hardcoded approach to supporting AWS APIs. boto3, in what appears to be an attempt to reduce the overhead of keeping Python client synced with evolving features on the various apis, has been more squishy around exceptions, so the ClientError approach you outlined above used to be the canonical way.
In 2017 they introduced the second mechanism you highlight: 'modeled' exceptions available on the client.
I am not familiar with SNS but in my experience with other AWS products, the ClientError naming matches up with the HTTP apis, which tend to be well documented. So I would start here: https://docs.aws.amazon.com/sns/latest/api/Welcome.html
It looks like the new-style modeled exceptions are generated from service definition files that live in botocore module. I can't find any documentation about it, but go browse around the AWS service models in https://github.com/boto/botocore/tree/master/botocore/data.
Also, it's good to know that if you are not (in contrast to OP's code) dealing directly with the low-level client, but instead are using a high-level AWS ServiceResource object, a low-level client is still easily available at my_service_resource.meta.client so you can handle exceptions like this:
try:
my_service_resource.do_stuff()
except my_service_resource.meta.client.exceptions.NotFoundException as e:
# handle exception
Use Boto3 exceptions: https://www.programcreek.com/python/example/97944/boto3.exceptions
client = boto3.client('logs')
try:
client.create_log_group(logGroupName=LOG_GROUP)
except client.exceptions.ResourceAlreadyExistsException:
pass
Related
Trying to catch this error:
ERROR: (gcloud.compute.instances.add-labels) HTTPError 404: The resource 'projects/matei-testing-4010-5cbdeeff/zones/us-east1-b/instances/all' was not found
Tried different versions of code and none worked for me.
My current code does not seem to catch an error:
from googleapiclient import discovery, errors
try:
print("Applying labels")
gcloud_value = (f'gcloud compute instances add-labels all --labels="key=value" --zone=us-east1-b')
process = subprocess.run([gcloud_value], shell=True)
except errors.HttpError:
print("Command did not succeed because of the following error: {}".format(errors.HttpError))
How do I catch the error to use it later?
Thank you
Try this:-
import subprocess
gcloud_value = 'gcloud compute instances add-labels all --labels="key=value" --zone=us-east1-b'
process = subprocess.run(gcloud_value, shell=True, capture_output=True)
print(process.stdout.decode('utf-8'))
print(process.stderr.decode('utf-8'))
print(process.returncode)
One would expect gcloud to emit errors to stderr. Therefore by examining process.stderr you should be able to figure out what (if anything) has gone wrong. Also, if process.returncode is non-zero you should be able to deduce that it didn't work but that depends entirely on how the underlying application (gcloud in this case) is written. There's plenty of stuff out there that returns zero even when there was a failure!
I strongly encourage you to consider using Google's client libraries to interact with its services' rather than subprocess'ing.
As you're experiencing calling out to a shell not only limits error handling but it often requires string "munging" which is also imprecise and ad hoc auth.
Google generally provides machine-generated "perfect" SDK implementations of all its services. For Cloud (except Compute!?), there are also Cloud Client libraries.
I encourage you to explore using the Python SDK for Compute Engine:
https://cloud.google.com/compute/docs/tutorials/python-guide
If for some reason the create_blob_from_path method runs into issues, what would be the proper way to handle the exception? As in, when you are writing your 'except' block, it would be except ??what?? as e: ? I want to be able to handle that exception when a blob upload fails so it can email me via sendmail so I know a backup that is attempting to be offloaded to Azure storage has failed.
Thank you!
I looked for & searched some exception names on the source codes of Azure Python Storage SDK, there are not any exceptions defined about failure of uploading blob. However, per my experience, Azure Storage SDK for Python wrapped REST APIs of Storage Services via the python package requests to do related operations. So any failure will cause the requests exceptions, you can try to catch the requests root exception requests.exceptions.RequestException as below to do the next action like sendmail.
import requests
try:
blobservice.create_blob_from_path(container_name, blob_name, file_path)
except requests.exceptions.RequestException:
sendmain(...)
I am using Flask Restful to build an API. I have a number of model classes with methods that may raise custom exceptions (for example: AuthFailed exception on my User model class). I am using the custom error handling, documented here, to handle this (so that when auth fails, an appropriate response is sent). So far so good. However, I notice that when the exception is raised although the correct response JSON and status is sent back, I still get a traceback which is not ideal. Usually, if I handle an error (outside of flask) with a try-except block, the except can catch the error and handle it (preventing the traceback). So what is the correct approach here? Am I misunderstanding how to use the errors feature?
Unfortunately for you, it is handled this way "by design" of the Flask-RESTful APIs errors functionality.
The exceptions which are thrown are logged and the corresponding response defined in the errors dict is returned.
However, you can change the level of log output by modifying the log level of Flask's logger like this:
app = Flask(__name__)
app.logger.setLevel(logging.CRITICAL)
I think you would actually have to set it to CRITICAL because these errors are still getting logged even on log level ERROR as far as I know.
Furthermore, both Flask and Flask-RESTful are open-source. That being said, after looking at the code I found the function of a Flask app that is responsible for adding the exception traceback to the log (Flask version 0.11.1).
Of course you could just create your own App class (extending the original class of Flask) which overrides this method (or a caller of it) and does something else instead. However, you should be careful when updating your Flask version if you make use of undocumented stuff like this.
I am writing python script using the Google Sheets apiAPI. It reads data and writes it to a new file, shares that file with a specified email and returns the id of the new file.
def read_sheet(self,spreadsheetId):
try:
result=self.service.spreadsheets().get(spreadsheetId=spreadsheetId,includeGridData=True,fields='namedRanges,properties,sheets').execute()
return result
except apiclient.errors.HttpError as e:
traceback.print_exc()
print(e)
sys.exit(1)
def create_spreadsheet(self,data,email):
try:
newid=self.service.spreadsheets().create(body=data,fields='spreadsheetId').execute()
newid=newid.get('spreadsheetId')
self.give_permissions(email,newid)
return newid
except apiclient.errors.HttpError as e:
traceback.print_exc()
print(e)
sys.exit(1)
This code works very well, but not with 100% accuracy. Sometimes I get a 500 Internal Server Error, but the file is created in my account. I found a similar Stack Overflow question (Getting 500 Error when using Google Drive API to update permissions), but it didn't help. I want to know the exact reason for this. Can anyone help?
EDIT1:
This is the exact error message
https://www.googleapis.com/drive/v3/files/349hsadfhSindfSIins-rasdfisadfOsa3OQmE/permissions?sendNotificationEmail=true&alt=json&transferOwnership=false
returned "Internal Error. User message: "An internal error has
occurred which prevented the sharing of these item(s): Template"">
As hinted to above in DaimTo's comment, the error is due to Google Drive still processing the create request while you're trying to add the permission to share the (new) file with. Remember, when you add a file to Drive, Google's servers are still working on the file-create as well as making it accessible globally. Once the flurry of activity settles down, then adding additional users to the document shouldn't be a problem.
You can see from this Drive API documentation page a description of the (500) error you received as well as the recommended course of action which is to implement exponential backoff, which is really just saying you should pause a bit before trying again & extending that delay each time you get the same error. He also pointed to another SO Q&A which you can look at. Another resource is this descriptive blog post. If you don't want to implement it yourself, you can try the retrying or backoff packages.
NOTE: You didn't show us all your code, but I changed the title of this question to more accurately reflect that you're using the Drive API for adding the permissions. While you've used the Sheets API to create the Sheet with, realize that you can just do this all with the Drive API (and not use the Sheets API at all unless you're doing spreadsheet-oriented operations. The Drive API is for all file-related operations like sharing, copying, import/export, etc.)
Bottom line is that you can create Sheets using either API, but if you're not doing anything else with the Sheets API, why bother making your app more complex? If you want to see how to create Sheets with both APIs, there's a short segment in my blog post that covers this... you'll see that they're nearly identical but using the Drive API does require one more thing, the MIMEtype.
If you want to learn more about both APIs, see this answer I gave to a related question that features additional learning resources I've created for both Drive and Sheets, most of which are Python-based.
I guess I'm late but just in case: just add a few seconds delay between the create request and the give permissions one. For me it works making the thread sleep for 10 seconds. Try this:
def create_spreadsheet(self,data,email):
try:
newid=self.service.spreadsheets().create(body=data,fields='spreadsheetId').execute()
newid=newid.get('spreadsheetId')
time.sleep(10)
self.give_permissions(email,newid)
return newid
except apiclient.errors.HttpError as e:
traceback.print_exc()
print(e)
sys.exit(1)
I am writing a simple API client in Python and I am wondering what exception should I raise if the remote server is not happy.
The API itself is very poorly documented (I don't know all the possible error messages so I can't just define custom classes for all of them), I am letting Requests handle HTTP-level errors which raises HTTPError in those cases, but what should I raise if the server just supplies an error key in the JSON response?
I am currently using Exception but that feels quite broad, I'd like to know whether there's a better alternative.
Regards.
Yes, just raising Exception is likely too broad.
Usually a module should define his own base exception class:
class MyAPIClientError(Exception):
pass
And then you can subclass that:
class RemoteServerNotHappyError(MyAPIClientError):
pass
RemoteServerNotHappyError should probably mention something about what the server's json returned that was not expected by your API client class. Add in the relevant error messages as you see fit.
This means users of your library can catch these specific exceptions (which they might know how to handle), without having to catch every exception (they surely don't know how to handle every possible failure mode).