I am working with athena from within my python code, using boto3, as follows:
def query_athena(query, output_path):
client = boto3.client('athena')
client.start_query_execution(
ResultConfiguration={'OutputLocation': output_path},
QueryString=query
)
As stated in the docs, start_query_execution may raise InternalServerException, InvalidRequestException or TooManyRequestsException. I'd like to treat this as follows:
def query_athena(query, output_path):
client = boto3.client('athena')
try:
client.start_query_execution(
ResultConfiguration={'OutputLocation': output_path},
QueryString=query
)
except <AthenaException> as e:
deal with e
where <AthenaException> being one of the three exceptions I mentioned or, better yet, their superclass.
My question is how do I import these exceptions? The docs show them as Athena.Client.exceptions.InternalServerException, but I can't seem to find this Athena.Client in any boto3 module.
I ran into the same confusion, but figured it out. The exceptions listed in the docs aren't internal to boto3, but rather contained in the response when boto3 throws a client error.
My first shot at a solution looks like this. It assumes you've handled s3 output location, a boto3 session, etc already:
import boto3
from botocore.exceptions import ClientError
try:
client = session.client('athena')
response = client.start_query_execution(
QueryString=q,
QueryExecutionContext={
'Database': database
},
ResultConfiguration={
'OutputLocation': s3_output,
}
)
filename = response['QueryExecutionId']
print('Execution ID: ' + response['QueryExecutionId'])
except ClientError as e:
response = e.response
code = response['Error']['Code']
message = response['Error']['Message']
if code == 'InvalidRequestException':
print(f'Error in query, {code}:\n{message}')
raise e
elif code == 'InternalServerException':
print(f'AWS {code}:\n{message}')
raise e
elif code == 'TooManyRequestsException':
# Handle a wait, retry, etc here
pass
Related
I am trying to send e-mail campaign mails with the SMTP API of Sendinblue.com, a cloud-based marketing communication software suite. I am running my configuration of the template script provided by Sendinblue as seen on the screenshot:
My initial script was as follows:
# ------------------
# Create a campaign\
# ------------------
# Include the Sendinblue library\
from __future__ import print_function
import time
from datetime import datetime
import sib_api_v3_sdk
from sib_api_v3_sdk.rest import ApiException
from pprint import pprint
myapikey = 'xkeysib-.................................' # full api key
# Instantiate the client\
sib_api_v3_sdk.configuration.api_key['api-sb'] = myapikey
api_instance = sib_api_v3_sdk.EmailCampaignsApi()
# Define the campaign settings\
email_campaigns = sib_api_v3_sdk.CreateEmailCampaign(
name= "Campaign sent via the API",
subject= "My subject",
sender= { "name": "From name", "email": "------#gmail.com"}, # personal email with which I was registered # sendinblue
type= "classic",
# Content that will be sent\
html_content= "Congratulations! You successfully sent this example campaign via the Sendinblue API.",
# Select the recipients\
recipients= {"listIds": [2, 7]},
# Schedule the sending in one hour\
scheduled_at = datetime.now()
)
# Make the call to the client\
try:
api_response = api_instance.create_email_campaign(email_campaigns)
pprint(api_response)
except ApiException as e:
print("Exception when calling EmailCampaignsApi->create_email_campaign: %s\n" % e)
I referred to Sendinblue Documentation in the cases where I had doubt. Most of the process seems self-explanatory, just the line with configuration.api_key['api-key'] = 'YOUR API KEY' was ambiguous as it is not quite clearly stated how exactly the api-key should be passed and the attribute of api_key I assumed api_key was holding the name of the API as defined in the SMTP & API Advanced tab. Even if I assume it should be holding some other values, it is not clear what they should be, what I can tell for sure is that sib_api_v3_sdk.configuration.api_key['api-key'] also resulted in AttributeError: module 'sib_api_v3_sdk.configuration' has no attribute 'api_key'.
After initially getting the error AttributeError: module 'sib_api_v3_sdk.configuration' has no attribute 'api_key' I researched a dozen of articles on StackOverflow, one of them on the Kubernetes topic where the script seemed to throw a similar error like the one I was getting, so I followed the advices described in Python kubernetes module has no attribute 'api_key'. Therefore, I tried reassigning the class attributes as follows:
configuration.api_key['api-sb'] = myapikey
api_instance = sib_api_v3_sdk.EmailCampaignsApi()
The fact that now I wasn't getting the error of a missing key but facing an 'unauthorized' message cheered me up for a short time, until I spent several hours trying to get past this. So the script that's now throwing the HTTP response body: {"message":"Key not found","code":"unauthorized"} is now this:
# ------------------
# Create a campaign\
# ------------------
# Include the Sendinblue library\
from __future__ import print_function
import time
from datetime import datetime
import sib_api_v3_sdk
from sib_api_v3_sdk.rest import ApiException
from pprint import pprint
myapikey = 'xkeysib-c..............' # key
# Instantiate the client\
sib_api_v3_sdk.configuration.api_key['api-sb'] = myapikey
api_instance = sib_api_v3_sdk.EmailCampaignsApi()
# Define the campaign settings\
email_campaigns = sib_api_v3_sdk.CreateEmailCampaign(
name= "Campaign sent via the API",
subject= "My subject",
sender= { "name": "From name", "email": "mail....#gmail.com"}, # personal gmail with which I was initially registered # sendinblue.com
type= "classic",
# Content that will be sent\
html_content= "Congratulations! You successfully sent this example campaign via the Sendinblue API.",
# Select the recipients\
recipients= {"listIds": [2, 7]},
# Schedule the sending in one hour\
scheduled_at = datetime.now()
)
# Make the call to the client\
try:
api_response = api_instance.create_email_campaign(email_campaigns)
pprint(api_response)
except ApiException as e:
print("Exception when calling EmailCampaignsApi->create_email_campaign: %s\n" % e)
My 1st question is: was my 1st script wrong and in what way?
Secondly: Is configuration.api_key['api-sb'] = myapikey the correct syntax, assuming that 'api-sb' is the name of my API key as shown on the screenshot?
And third: In the myapikey variable, when assigning the API key itself, should there be anything else as a prefix to the key?
Try this sample transactional email script. Worked for me in python.
https://developers.sendinblue.com/reference/sendtransacemail
Instead of this:
sib_api_v3_sdk.configuration.api_key['api-sb'] = myapikey
api_instance = sib_api_v3_sdk.EmailCampaignsApi()
do this:
configuration = sib_api_v3_sdk.Configuration()
configuration.api_key['api-sb'] = myapikey
api_instance = sib_api_v3_sdk.EmailCampaignsApi(sib_api_v3_sdk.ApiClient(configuration))
as shown in https://developers.sendinblue.com/reference/createemailcampaign-1.
Based on AWS TimeStream SDK documentation for Python, I have the following code:
import boto3
def list_databases(self):
print("Listing databases")
try:
result = self.client.list_databases(MaxResults=5)
self._print_databases(result['Databases'])
next_token = result.get('NextToken', None)
while next_token:
result = self.client.list_databases(NextToken=next_token, MaxResults=5)
self._print_databases(result['Databases'])
next_token = result.get('NextToken', None)
except Exception as err:
print("List databases failed:", err)
session = boto3.Session(profile_name='superuser', region_name='eu-west-1')
query_client = session.client('timestream-query')
list_databases(query_client)
The authentication for my user superuser seems to work fine, but the boto3 session used for my query_client does not have a client object:
Listing databases
List databases failed: 'TimestreamQuery' object has no attribute 'client'
What am I missing?
This argument name in this method is likely a mistake:
# this name is discouraged in non-OO code:
def list_databases(self):
self is typically used in object oriented python code, which is not the case here.
Rename it as follows:
# this is better:
def list_databases(client):
Then, remove any mention of self in the body of the function, for example:
# this is incorrect:
result = self.client.list_databases(MaxResults=5)
should be:
# this should work:
result = client.list_databases(MaxResults=5)
I am writing a Python script that runs a query through Athena, outputs it to S3 and downloads it into my computer. I am able to run my query through Athena and output the result into S3. So my next step that I can’t seem to figure out is how to download it to my computer without knowing the key name?
Is there a way to lookup the object key within my python script after outputting it to Athena?
What I have completed:
# Output location and DB
s3_output = ‘s3_output_here’
database = ‘database_here’
# Function to run Athena query
def run_query(query, database, s3_output):
while True:
try:
response = client.start_query_execution(
QueryString=query,
QueryExecutionContext={
'Database': database
},
ResultConfiguration={
'OutputLocation': s3_output,
}
)
return response
break
except client.exceptions.TooManyRequestsException as e:
print('Too many requests, trying again after sleep')
time.sleep(100)
# Our SQL Query
query = """
SELECT *
FROM test
”””
print("Running query to Athena...")
res = run_query(query, database, s3_output)
I understand how to download a file with this code:
try:
s3.Bucket(BUCKET_NAME).download_file(KEY, ‘KEY_HERE’)
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == "404":
print("The object does not exist.")
else:
raise
So how can I read the key name after running my first completed code?
You can get the key using the get_key command provided by the boto library. This is how I download things from s3:
with open("path/aws-credentials.json") as f:
data= json.load(f)
conn = boto.connect_s3(data["accessKeyId"], data["secretAccessKey"])
bucket = conn.get_bucket('your_bucket')
file_path = bucket.get_key('path/to/s3/file')
file_path.get_contents_to_filename('path/on/local/computer/filename')
You can hardcode your credentials into the code if you are just testing something out, but if you are planning on putting this into production, it's best to store your credentials externally in something like a json file.
What is the best way to do error handling when getting an object from S3 using Python boto3?
My approach:
from botocore.exceptions import ClientError
import boto3
s3_client = boto3.client('s3')
try:
s3_object = s3_client.get_object("MY_BUCKET", "MY_KEY")
except ClientError, e:
error_code = e.response["Error"]["Code"]
# do error code checks here
I am not sure if ClientError is the best Exception to use here. I know there is a Boto3Error class, but I do not think you can do error code checks similarly to ClientError.
I think your approach is sufficient. If you can narrow your errors to a few, you can break it down into if blocks, and handle accordingly.
except ClientError as e:
error_code = e.response["Error"]["Code"]
if error_code == "AccessDenied":
# do code
elif error_code == "InvalidLocationConstraint":
# do more code
This is just an experimental approach. Because most error responses are API-driven, I don't think you'll find them anywhere directly in the code (ie: doing except AccessDenied:). You can find all error responses for Amazon S3 here.
I have some code in a python-eve app that retrieves some data from a device and populates a resource when that resource is requested for the first time. Sometimes the code can't successfully connect to the device. In this case, I would like to return an error message that explains this better, rather than just a plain 500 error. Here is the on_fetch_item hook:
def before_returning_item(resource, _id, document):
if resource == "switches":
if "interfaces" not in document.keys():
# retrieve and store switch config if it hasn't been stored yet
r = app.data.driver.db[resource]
try:
switch = prepare_switch_from_document(document)
except socket.timeout:
# raise some more meaningful error with message
pass
interface_list = switch.get_formatted_interfaces()
r.update({'_id': _id}, {'interfaces': interface_list})
document['interfaces'] = interface_list
app.on_fetch_item += before_returning_item
Thanks in advance.
All you have to do is take advantage of Flask's abort method:
from flask import abort
def before_returning_item(resource, _id, document):
if resource == "switches":
if "interfaces" not in document.keys():
# retrieve and store switch config if it hasn't been stored yet
r = app.data.driver.db[resource]
try:
switch = prepare_switch_from_document(document)
except socket.timeout:
# raise some more meaningful error with message
abort(500)
interface_list = switch.get_formatted_interfaces()
r.update({'_id': _id}, {'interfaces': interface_list})
document['interfaces'] = interface_list
app.on_fetch_item += before_returning_item
If you want to add a custom description:
abort(500, description='My custom abort description')
I like to create custom exceptions, and raise those with meaning comments, Eg:
class MyExcept(Exception): pass
def before_returning_item():
...
if error_condition:
raise MyException('detailed explanation')
...
try:
before_returning_item()
except MyException, exc:
if 'device not ready' in str(exc):
print("Device was not rdy...Try agin?")
else:
raise