For my Data Pipeline where I am scraping data out of the web and send it to Kafka, I am using the Dead-Letter-Queue Strategy for professional Error Handling. I was wondering how I can define the except-Statement if I don't know what Exception might occur. I am sending the exception to an error topic in which I want to extract the Error details. Atm it looks like this:
from setup import producer as p
def scraper():
try:
# ... scraping data
except Exception as e:
# Send NaN-Value to topic
p.produce(topic, NaN-message, callback=delivery_report)
p.flush()
# Send error message to error topic
error_message = json.dumps({
"topic": topic,
"error": repr(e),
"error_type": str(type(e)),
"timestamp": str(datetime.datetime.now())
})
p.produce(error_topic, error_message, callback=delivery_report)
p.flush()
Anyone an idea on how to improve the error handling that it takes whatever Exception occurs?
Related
How can i make an exception on "A wait of 61 seconds is required (caused by SendMultiMediaRequest)" ? I can't find anything in the official documentation.
The fact is that when sending photos to some users, this error pops up.
Basically sends well, but sometimes there are users who simply do not send (throws an exception "Exception"), although they exist and, in principle, you can send a message to them. I need to trace this exception
from telethon import TelegramClient, errors
client = TelegramClient('session', api_id=API_ID, api_hash=API_HASH)
message = "Hello"
files = ['img/11.jpg', 'img/22.jpg', 'img/33.jpg', 'img/44.jpg', 'img/55.jpg']
async def main():
for user in db.get_users(name_table=table):
try:
if db.check_invited(name_table=table, user_name=user) != "TRUE":
await client.send_message(entity=user, message=message, file=files)
print('Ok')
except errors.BadRequestError as e:
print(e)
except Exception as e:
print(e)
I have a python file called main.py and I am trying to make a Prometheus connection with a specific token. However, if the token is expired, instead of the error message printing out prometheus_api_client.exceptions.PrometheusApiClientException, how can I get the error message to print our like status_code: 500, reason: Invalid token using a try and except block.
Code:
#token="V0aksn-as9ckcnblqc6bi3ans9cj1nsk" #example, expired token
token ="c0ams7bnskd9dk1ndk7aKNYTVOVRBajs" #example, valid token
pc = PrometheusConnect(url = url, headers={"Authorization": "bearer {}".format(token)}, disable_ssl=True)
try:
#Not entirely sure what to put here and the except block
except:
I've tested out a couple code in the try and except blocks and could not get rid of the long error from Prometheus. Any suggestions?
How about putting your pc variable in try and PrometheusApiClientException for the exception. If that doesn't work, go to the source file and use whatever exception developers used while making authorization.
This is how you catch that exception in a try/except block
try:
# Interact with prometheus here
pass
except prometheus_api_client.exceptions.PrometheusApiClientException as e:
print('status_code: 500, reason: Invalid token')
I make batch operations in my lambda function on a huge number of csv files. I want to have the content of error/exception in my completion reports. I have only 5% of error files so lambda works fine, but it doesn't write errors in the report.
When I test my lambda on a file that leads to errors, I see that "ResultMessage" is the same as error or exception. I tried adding a string with exception to report but the last column is always Null.
Can you help me?
except ClientError as e:
# If request timed out, mark as a temp failure
# and S3 Batch Operations will make the task for retry. If
# any other exceptions are received, mark as permanent failure.
errorCode = e.response['Error']['Code']
errorMessage = e.response['Error']['Message']
if errorCode == 'RequestTimeout':
resultCode = 'TemporaryFailure'
resultString = 'Retry request to Amazon S3 due to timeout.'
else:
resultCode = 'PermanentFailure'
resultString = '{}: {}'.format(errorCode, errorMessage)
except Exception as e:
# Catch all exceptions to permanently fail the task
resultCode = 'PermanentFailure'
resultString = 'Exception: {}'.format(e)
finally:
results.append({
'taskId': taskId,
'resultCode': resultCode,
'ResultMessage': resultString
})
return {
'invocationSchemaVersion': invocationSchemaVersion,
'invocationId': invocationId,
'results': results
}
Example rows of my report with failed csv
There's nothing obviously wrong with your code.
I checked the docs:
Response and result codes
There are two levels of codes that S3 Batch Operations expect from Lambda functions. The first is the response code for the entire request, and the second is a per-task result code. The following table contains the response codes.
Response code
Description
Succeeded
The task completed normally. If you requested a job completion report, the task's result string is included in the report.
TemporaryFailure
The task suffered a temporary failure and will be redriven before the job completes. The result string is ignored. If this is the final redrive, the error message is included in the final report.
PermanentFailure
The task suffered a permanent failure. If you requested a job-completion report, the task is marked as Failed and includes the error message string.
Sounds to me like you'd need to look into the Job Completion Report to get more details.
I am using tweepy to make a twitter application.
When users tweet/update profile, etc, they will get some errors. I want to classify error and give user more information.
try:
tweet/update profile/ follow....
except tweepy.TweepError, e:
if tweepy.TweepError is "Account update failed: Description is too long (maximum is 160 characters)"
Do something
if tweepy.TweepError is "Failed to send request: Invalid request URL: http://api.twitter.com/1/account/update_profile.json?location=%E5%85%B5%E5%BA%A"
Do something
if tweepy.TweepError is "[{u'message': u'Over capacity', u'code': 130}]"
Do something
Is the only way to classify error is to compare e with string, for example, Account update failed: Description is too long (maximum is 160 characters)?
Right, it is the only way now. There is only one TweepError exception defined. It's raised throughout the app with different text.
Here's the relevant open issue on github. So there is a chance that it'll be improved in the future.
In python 2.6.6, how can I capture the error message of an exception.
IE:
response_dict = {} # contains info to response under a django view.
try:
plan.save()
response_dict.update({'plan_id': plan.id})
except IntegrityError, e: #contains my own custom exception raising with custom messages.
response_dict.update({'error': e})
return HttpResponse(json.dumps(response_dict), mimetype="application/json")
This doesnt seem to work. I get:
IntegrityError('Conflicts are not allowed.',) is not JSON serializable
Pass it through str() first.
response_dict.update({'error': str(e)})
Also note that certain exception classes may have specific attributes that give the exact error.
Everything about str is correct, yet another answer: an Exception instance has message attribute, and you may want to use it (if your customized IntegrityError doesn't do something special):
except IntegrityError, e: #contains my own custom exception raising with custom messages.
response_dict.update({'error': e.message})
You should use unicode instead of string if you are going to translate your application.
BTW, Im case you're using json because of an Ajax request, I suggest you to send errors back with HttpResponseServerError rather than HttpResponse:
from django.http import HttpResponse, HttpResponseServerError
response_dict = {} # contains info to response under a django view.
try:
plan.save()
response_dict.update({'plan_id': plan.id})
except IntegrityError, e: #contains my own custom exception raising with custom messages.
return HttpResponseServerError(unicode(e))
return HttpResponse(json.dumps(response_dict), mimetype="application/json")
and then manage errors in your Ajax procedure.
If you wish I can post some sample code.
Suppose you raise error like this
raise someError("some error message")
and 'e' is catched error instance
str(e) returns:
[ErrorDetail(string='some error message', code='invalid')]
but if you want "some error message" only
e.detail
will gives you that (actually gives you a list of str which includes "some error message")
This works for me:
def getExceptionMessageFromResponse( oResponse ):
#
'''
exception message is burried in the response object,
here is my struggle to get it out
'''
#
l = oResponse.__dict__['context']
#
oLast = l[-1]
#
dLast = oLast.dicts[-1]
#
return dLast.get( 'exception' )