I'm confused as to when to return self inside a class and when to return a value which may or may not possibly be used to check the method ran correctly.
def api_request(self, data):
#api web request code
return response.text
def connect(self):
#login to api, set some vars defined in __init__
return self
def send_message(self, message):
#send msg code
return self
So above theres a few examples. api_request I know having the text response is a must. But with send_message what should I return?
which is then converted to a dict to check a key exists, else raise error).
Should it return True, the response->dict, or self?
Thanks in advance
Since errors tend to be delivered as exceptions and hence success/fail return values are rarely useful, a lot of object-modifier functions wind up with no return value at all—or more precisely, return None, since you can't return nothing-at-all. (Consider some of Python's built-in objects, like list, where append and extend return None, and dict, where dict.update returns None.)
Still, returning self is convenient for chaining method calls, even if some Pythonistas don't like it. See kindall's answer in Should internal class methods returnvalues or just modify instance variables in python? for example.
Edit to add some examples based on comment:
What you "should" return—or raise an exception, in which case, "what exception"—depends on the problem. Do you want send_message() to wait for a response, validate that response, and verify that it was good? If so, do you want it to raise an error if there is no response, the validation fails, or the response was valid but says "message rejected"? If so, do you want different errors for each failure, etc? One reasonable (for some value of reasonable) method is to capture all failures with a "base" exception, and make each "type" of failure a derivative of that:
class ZorgError(Exception): # catch-all "can't talk via the Zorg-brand XML API"
pass
class ZorgRemoteDown(ZorgError): # connect or send failed, or no response/timeout
pass
class ZorgNuts(ZorgError): # remote response incomprehensible
pass
class ZorgDenied(ZorgError): # remote says "permission denied"
pass
# add more if needed
Now some of your functions might look something like this (note, none of this is tested):
def connect(self):
"""connect to server, log in"""
... # do some prep work
addr = self._addr
try:
self._sock.connect(addr)
except socket.error as err:
if err.errno == errno.ECONNREFUSED: # server is down
raise ZorgRemoteDown(addr) # translate that to our Zorg error
# add more special translation here if needed
raise # some other problem, propagate it
... # do other stuff now that we're connected, including trying to log in
response = self._get_response()
if response == 'login denied' # or whatever that looks like
raise ZorgDenied() # maybe say what exactly was denied, maybe not
# all went well, return None by not returning anything
def send_message(self, msg):
"""encode the message in the way the remote likes, send it, and wait for
a response from the remote."""
response = self._send_and_wait(self._encode(msg))
if response == 'ok':
return
if response == 'permission denied':
raise ZorgDenied()
# don't understand what we got back, so say the remote is crazy
raise ZorgNuts(response)
Then you need some "internal" functions like these:
def _send_and_wait(self, raw_xml):
"""send raw XML to server"""
try:
self._sock.sendall(raw_xml)
except socket.error as err:
if err.errno in (errno.EHOSTDOWN, errno.ENETDOWN) # add more if needed
raise ZorgRemoteDown(self._addr)
raise
return self._get_response()
def _get_response(self):
"""wait for a response, which is supposedly XML-encoded"""
... some code here ...
if we_got_a_timeout_while_waiting:
raise ZorgRemoteDown(self._addr)
try:
return some_xml_decoding_stuff(raw_xml)
except SomeXMLDecodeError:
raise ZorgNuts(raw_xml) # or something else suitable for debug
You might choose not to translate socket.errors at all, and not have all your own errors; perhaps you can squeeze your errors into ValueError and KeyError and so on, for instance.
These choices are what programming is all about!
Generally, objects in python are mutable. You therefore do not return self, as the modifications you make in a method are reflected in the object itself.
To use your example:
api = API() # initialise the API
if api.connect(): # perhaps return a bool, indicating that the connection succeeded
api.send_message() # you now know that this API instance is connected, and can send messages
Related
I'm writing some tests, and I want to assert that no request was sent, preferably using responses library.
def my_method():
if something:
raise AnException("foo")
session.post()
I need to make sure that if someone changes my_method, if something will be checked first and an exception will be raised
def test_my_method_fail():
my_class.something = True
with pytest.raises(AnException) as exc:
my_class.my_method()
assert "foo" in str(exc.value) and (no message was sent)
I'm calling Udemy external api to build a simple REST service for experimental purpose.
https://www.udemy.com/developers/affiliate/
Here is my get_all() courses method.
class Courses(object):
"""
Handles all requests related to courses.
ie; gets the courses-list, courses-detail, coursesreviews-list
"""
def __init__(self, api):
self.api = api
logger.debug("courses initialized")
def get_all(self):
page = 1
per_page = 20
while True:
res = self._get_courses(page, per_page)
if not res['results']:
break
try:
for one in res['results']:
yield one
except Exception as e: -->>>handling exception
print(e)
break
page += 1
def _get_courses_detail(self, page, per_page):
resource = "courses"
params = {'page': page, 'per_page': per_page,
# 'fields[course]': '#all'
}
res = self.api.get(resource, params)
return res
Now, is it reasonable to handle a exception(in get_all() method) assuming that there could some error in the returning data of the api?
Or handling the exception(in get_all) is not needed here and it should be handled by the calling function?
Most of the open source projects that I see don't handle this exception.
I'm sharing the opinion in this answer. So catch the exception as soon as possible and rethrow it if needed to the next layer.
With practice and experience with your code base it becomes quite easy to judge when to add additional context to errors, and where it's most sensible to actually, finally handle the errors.
Catch → Rethrow
Do this where you can usefully add more information that would save a developer having to work through all the layers to understand the problem.
Catch → Handle
Do this where you can make final decisions on what is an appropriate, but different execution flow through the software.
Catch → Error Return
Relevant Code
def start_requests( self ):
requests = [ Request( url['url'], meta=url['meta'], callback=self.parse, errback=self.handle_error ) for url in self.start_urls if valid_url( url['url'] )]
return requests
def handle_error( self, err ):
# Errors being saved in DB
# So I don't want them displayed in the logs
I've got my own code for saving error codes in DB. I don't want them displayed in the log output. How can I suppress these errors?
Note that I don't want to suppress all errors - just the ones being handled here.
Try to use self.skipped.add, self.failed.add with isinstance condition in your handle_error method.
Here is an example
def on_error(self, failure):
if isinstance(failure.value, HttpError):
response = failure.value.response
if response.status in self.bypass_status_codes:
self.skipped.add(response.url[-3:])
return self.parse(response)
# it assumes there is a response attached to failure
self.failed.add(failure.value.response.url[-3:])
return failure
Answer by #Daniil Mashkin seems to be the most comprehensive solution.
For simple cases, you can add http error codes Spider.handle_httpstatus_list or HTTPERROR_ALLOWED_CODES in Settings.py.
This will send some erroneous answers to your callback function, thus skipping logging as well
Use a simple try-except in your function. As long as you handle the exception by yourself (adding rows to the db, just "pass", ...), twisted won't recognize the error.
e.g.
def handle_error( self, err ):
try:
#do something that raises an exception
#twisted won't log this as long as you handle it yourself
myvar = 14 / 0
except:
pass
I would like to improve my coding style with a more robust grasp of try, except and raise in designing API, and less verbose code.
I have nested functions, and when one catches an execption, I am passing the exception to the other one and so on.
But like this, I could propagate multiple checks of a same error.
I am referring to:
[Using try vs if in python
for considering cost of try operation.
How would you handle an error only once across nested functions ?
E.g.
I have a function f(key) doing some operations on key; result is
passed to other functions g(), h()
if result comply with
expected data structure, g() .. h() will manipulate and return
updated result
a decorator will return final result or return the
first error that was met, that is pointing out in which method it was raised (f(),g() or h()).
I am doing something like this:
def f(key):
try:
#do something
return {'data' : 'data_structure'}
except:
return {'error': 'there is an error'}
#application.route('/')
def api_f(key):
data = f(k)
try:
# do something on data
return jsonify(data)
except:
return jsonify({'error':'error in key'})
IMO try/except is the best way to go for this use case. Whenever you want to handle an exceptional case, put in a try/except. If you can’t (or don’t want to) handle the exception in some sane way, let it bubble up to be handled further up the stack. Of course there are various reasons to take different approaches (e.g. you don’t really care about an error and can return something else without disrupting normal operation; you expect “exceptional” cases to happen more often than not; etc.), but here try/except seems to make the most sense:
In your example, it’d be best to leave the try/except out of f() unless you want to…
raise a different error (be careful with this, as this will reset your stack trace):
try:
### Do some stuff
except:
raise CustomError('Bad things')
do some error handling (e.g. logging; cleanup; etc.):
try:
### Do some stuff
except:
logger.exception('Bad things')
cleanup()
### Re-raise the same error
raise
Otherwise, just let the error bubble up.
Subsequent functions (e.g. g(); h()) would operate the same way. In your case, you’d probably want to have some jsonify helper function that jsonifies when possible but also handles non-json data:
def handle_json(data):
try:
return json.dumps(data)
except TypeError, e:
logger.exception('Could not decode json from %s: %s', data, e)
# Could also re-raise the same error
raise CustomJSONError('Bad things')
Then, you would have handler(s) further up the stack to handle either the original error or the custom error, ending with a global handler that can handle any error. In my Flask application, I created custom error classes that my global handler is able to parse and do something with. Of course, the global handler is configured to handle unexpected errors as well.
For instance, I might have a base class for all http errors…
### Not to be raised directly; raise sub-class instances instead
class BaseHTTPError(Exception):
def __init__(self, message=None, payload=None):
Exception.__init__(self)
if message is not None:
self.message = message
else:
self.message = self.default_message
self.payload = payload
def to_dict(self):
"""
Call this in the the error handler to serialize the
error for the json-encoded http response body.
"""
payload = dict(self.payload or ())
payload['message'] = self.message
payload['code'] = self.code
return payload
…which is extended for various http errors:
class NotFoundError(BaseHTTPError):
code = 404
default_message = 'Resource not found'
class BadRequestError(BaseHTTPError):
code = 400
default_message = 'Bad Request'
class NotFoundError(BaseHTTPError):
code = 500
default_message = 'Internal Server Error'
### Whatever other http errors you want
And my global handler looks like this (I am using flask_restful, so this gets defined as a method on my extended flask_restful.Api class):
class RestAPI(flask_restful.Api):
def handle_error(self, e):
code = getattr(e, 'code', 500)
message = getattr(e, 'message', 'Internal Server Error')
to_dict = getattr(e, 'to_dict', None)
if code == 500:
logger.exception(e)
if to_dict:
data = to_dict()
else:
data = {'code': code, 'message': message}
return self.make_response(data, code)
With flask_restful, you may also just define your error classes and pass them as a dictionary to the flask_restful.Api constructor, but I prefer the flexibility of defining my own handler that can add payload data dynamically. flask_restful automatically passes any unhandled errors to handle_error. As such, this is the only place I’ve needed to convert the error to json data because that is what flask_restful needs in order to return an https status and payload to the client. Notice that even if the error type is unknown (e.g. to_dict not defined), I can return a sane http status and payload to the client without having had to convert errors lower down the stack.
Again, there are reasons to convert errors to some useful return value at other places in your app, but for the above, try/except works well.
Let's say your processing a message in an overridden class like:
class MailProcessorServer(smtpd.SMTPServer):
def process_message(self, peer, sender, rcpttos, data):
badrecipients = []
for rcpt in rcpttos:
badrecipients.append(rcpt)
#Here I want to warn the sender via a bounced email
# that the recipient does not exist
raise smtplib.SMTPRecipientsRefused(badrecipients)
#but this just crashes the process and eventually the sender times out,
# not good enough
I just want to bounce back to the sender immediately. Instead the sending service (say, GMail) just gives up eventually and warns the user many hours later. The documentation seems pretty sparse.
As documented only in the sources (sorry!), process_message's specs include:
This function should return None, for
a normal `250 Ok' response; otherwise
it returns the desired response string
in RFC 821 format.
So you could "return '554 bad recipients %s' % badrecipients" instead of using that raise statement -- not entirely satisfactory (doesn't properly account for a mix of good and bad, which by RFC 821 should return a '250 Ok' but also send a warning mail later) but it does seem to be the "bounce back immediately" effect that you're looking for with that raise.
The way to reject a message is to return a string with the error code from your process_message method; e.g.
return '550 No such user here'
However, RFC 821 doesn't allow error code 550 to be returned after the message data has been transfered (it should be returned after the RCPT command), and the smtpd module unfortunately doesn't provide an easy way to return an error code at that stage. Furthermore, smtpd.py makes it difficult to subclass its classes by using auto-mangling "private" double-underscore attributes.
You may be able to use the following custom subclasses of smtpd classes, but I haven't tested this code:
class RecipientValidatingSMTPChannel(smtpd.SMTPChannel):
def smtp_RCPT(self, arg):
print >> smtpd.DEBUGSTREAM, '===> RCPT', arg
if not self._SMTPChannel__mailfrom:
self.push('503 Error: need MAIL command')
return
address = self._SMTPChannel__getaddr('TO:', arg)
if not address:
self.push('501 Syntax: RCPT TO: <address>')
return
if self._SMTPChannel__server.is_valid_recipient(address):
self._SMTPChannel__rcpttos.append(address)
print >> smtpd.DEBUGSTREAM, 'recips:', self._SMTPChannel__rcpttos
self.push('250 Ok')
else:
self.push('550 No such user here')
class MailProcessorServer(smtpd.SMTPServer):
def handle_accept(self):
conn, addr = self.accept()
print >> smtpd.DEBUGSTREAM, 'Incoming connection from %s' % repr(addr)
channel = RecipientValidatingSMTPChannel(self, conn, addr)
def is_valid_recipient(self, address):
# insert your own tests here, return True if it's valid
return False
The following will discard the mail without bouncing it.
return '554-5.7.1'
Problem: The sender MTA's will try to resend the mail again and again if you reject a mail without bouncing.
Error code 550 will bounce the email which can be a bad idea as you don't want to give a spamer any information concerning your mail server. Be careful with that.
return '550'
Both errors will raise an smtplib.SMTPException. Here is the simplified code I use to handle such exceptions.
try:
if bounce:
return '550 Bad address'
else:
self.send_and_quit(sender, recipients, data)
except smtplib.SMTPException as e:
raise e
except Exception as e:
# Catch any other exception
logging.error(traceback.format_exc())
if not isinstance(e, smtplib.SMTPException):
self.send_and_quit(sender, recipients, data)
else:
raise e