I'm writing a small production level Flask application that runs on IIS. I've wrapped all of my functions inside try catch blocks and it looks like this.
try:
#Do Something
except Exception,e:
logger.error('Exception in Function X of type : %s ,for Image %s : %s'%(str(type(e)),path,str(e.args)))
I just need to log the problem in most of the cases and use python's builtin logging module to achieve this. I even log the type of the exception sometimes.
Now the thing I'm really concerned about is that although in my specific case, I don't have to handle or recover from any exception and even If I handle specific exceptions with a stack of different except cases, I'll just be logging the error in each block. So,
Is it still necessary for me to catch specific exceptions instead of
the generic Exception?
If the goal is to log all exceptions, then no, you don't have to catch specific ones.
As you noted, there'd be no point as you'd only repeat the same piece of logging.
Related
Python tutorial have a section called Errors and Exceptions which use the structure as;
try:
[statement]
except [Built-in ExceptionType]:
do something when an exception has been captured.
finally:
do something whether exception occurred or not.
This can also handle by raise an Error directly, too.
try:
raise [Built-in ExceptionType]
except [Built-in ExceptionType above] as e:
print(f'foo error: {e}')
There are several built-in ExceptionType which the good practice is developer should catch every specific exceptions type that developer should look for. (refer from Why is "except: pass" a bad programming practice?)
However, after I reading logging section. I'm thinking about I want to log the error into the log file and let user to aware the error (maybe just print the text and user can inform IT support) rather than throwing an exception on screen. Therefore, instead of try / except combo above then I can use the following error message and logged it.
if len(symbol) != 1:
error_message = f'Created box with symbol={symbol}, width={width} and height={height}, Symbol needs to be a string of length 1'
logger.error(error_message)
print(f'\nError! symbol={symbol}, width={width} and height={height} -- Symbol needs to be a string of length 1')
return
I doubt that what the better current practice is between:
a) raise Error on screen and
b) logging the Error for further investigation?
c) Other method (please suggest me)
I hope that I'm not try to comparing two different things.For me, I want to choose b) and store the error in log. this is easier for investigation and does not give unnecessary information to the users. but I do not sure I'm on the right track or not. Thank you very much.
Logging and raising exceptions are two different things for two different purposes. Logs let you inspect what your program did after the fact. Raising exceptions has important effects on the program flow right now. Sometimes you want one, sometimes you want the other, sometimes you want both.
The question is always whether an error is expected or unexpected, whether you have some plan what to do in case it occurs, and whether it is useful to notify anyone about the occurrence of the error or not.
Expected errors that you have a "backup plan" for should be caught and handled, that is regular program control flow. Unexpected errors probably should halt the program, or at least the particular function in which they occurred. If a higher up caller feels like handling the exception, let them. And whether to log an error or not (in addition to handling or not handling it) depends on whether anyone can glean any useful insight from that log entry or whether it would just be noise.
As deceze already mentions, logging and exceptions are two totally distinct things. As a general rule:
you raise an exception when you encounter an unexpected / invalid condition that you cannot handle at this point of the code. What will became of this exception (whether someone up in the call stack will handle it or not) is none of your concern at this point
you only catch an exception when you can handle it or when you want to log it (eventually with additionnal context informations) THEN reraise it
at your application's top-level, you eventually add a catchall exception handler that can log the exception and decide on the best way to deal with the situation depending on the type of application (command-line script, GUI app, webserver etc...).
Logging is mostly a developper/admin tool used for post-mortem inspection, program behaviour analysis etc, it's neither an error handling tool nor an end-user UI feature. Your example:
if len(symbol) != 1:
error_message = f'Created box with symbol={symbol}, width={width} and height={height}, Symbol needs to be a string of length 1'
logger.error(error_message)
print(f'\nError! symbol={symbol}, width={width} and height={height} -- Symbol needs to be a string of length 1')
return
is a perfect antipattern. If your function expects a one character string and the caller passed anything else then the only sane solution is to raise an exception and let the caller deal with it. How it does is, once again, none of your concerns.
You should log all errors no matter if they are thrown or handled for log analysis and refactoring amongst other purposes.
Whether to throw or to handle the error usually depends on the intended purpose of the code and the severity of the error.
Although "throwing" should be only used in debugging and handled
'gracefully' by dedicated exception code in the production version of the
application. No user likes crashes.
If the error impacts the business logic, end result of the code, or can cause damages to the user of the software, you want to terminate the execution. You do it either by throwing the error or handling it through a dedicated error handling procedure that shutdowns the program.
If the error is not severe and does not impact the normal functionality of the program, you can choose whether you want to handle it or throw it based on the best practices in your team and requirements of the project.
From the Docs:
Returns a new instance of the SMTPHandler class. The instance is initialized with the from and to addresses and subject line of the email. The toaddrs should be a list of strings.
logging.handlers.SMTPHandler can be used to send logged error message.
import logging
import logging.handlers
smtp_handler = logging.handlers.SMTPHandler(mailhost=("example.com", 25),
fromaddr="from#example.com",
toaddrs="to#example.com",
subject="Exception notification")
logger = logging.getLogger()
logger.addHandler(smtp_handler)
you can break the logic here if the code execution needs to be stopped.
Also look around this answer i have referred earlier collate-output-in-python-logging-memoryhandler-with-smtphandler
According to Python's Logging tutorial, logging.error is to be used to "report suppression of an error without raising an exception.", whereas if I wanted to report an error I'm supposed to just raise an exception and not use logging at all.
What if I want to report an error and then raise an exception?
For example:
try:
os.path.getsize('/nonexistent')
except os.error as error:
logging.error(str(error))
logging.error('something went wrong')
raise SystemExit(1)
This results in the following lines being printed to the standard error:
ERROR:root:[Errno 2] No such file or directory: '/nonexistent'
ERROR:root:something went wrong
It seems sensible. Or:
try:
os.path.getsize('/nonexistent')
except os.error as error:
logging.error(str(error))
raise SomeUserDefinedException('something went wrong')
Is this bad practice?
logging is used to report.
raise is an action which interrupts the normal flow of the program.
Typically, you raise exceptions where there is an actual flow to interrupt, and when you want to report, you can use logging, which will log errors and exceptions by default.
The logging part usually comes as a wrapper around a module, class, or middleware in your program.
logging within the code should only be used for DEBUG purposes, or for INFO purpose if you wanted to keep track of some information in your server logs for example. Not for ERROR.
I do not believe that is a bad practice. Exceptions and logging serve two different purposes and in any given situation you may want one, the other or both. Exceptions are for programmatic consumption and control flow within your code, while logging is for external consumption, by either a human or some sort of automated log watcher (Splunk for example). For instance, you may log the values of certain variables to help you figure out what happened and improve the code, though in that case, a level of debug would be more appropriate. Or you may want a systems administrator to be aware of the issue and potentially take some action. In that case, setting the log level to error and using something like a syslog logging handler makes sense.
Consider you have this kind of code for flask api:
#app.route('/doing/foo')
def foo():
try:
...
except Exception as ex:
doing some stuff
Now doing some stuff accidentally threw another exception. How do you prevent the server from showing another generic internal server error without any clue what happened?
In other words, how do I catch exceptions that may have originated from the except clause?
Aha! That is a 100 years question.
First option is to move to Python 3.
Python 3 has exception chaining and will print both of the exceptions - the one being handled and the one that was thrown due to the mishandling.
Second option is to implement exception chaining by yourself:
Manually inject the new exception into the one being handled using another try-except clause inside the current except:.
Keep in mind it WILL cause circular reference. Since it's a one-time exception, I wouldn't bother dealing with it. Further usage of this method will require cleaning the traceback using traceback.clear_frames().
In a request handler, log the error and return an HTTP 500 error and some HTML noting that an "Internal error has occurred" and containing a link back to some safe point for users to restart whatever they were doing.
You can attempt to catch more specific exceptions (from most specific and common towards Exception (which is the most general of the general exceptions). You can also attempt to introspect on the exception/traceback object (ex in your example).
Mostly you just want to ensure that you don't display internal application state to the user through their browser (possible security implications) and that you don't simply drop the user's request on the floor.
def process_item(...):
try:
#some hacking
#...
#lots of buggy code
except Exception as e:
#do logging
#record complete/incomplete state
Yes I know that I should generally intercept specific exceptions. But if the code is applied to many items which come from input data. So there might be unpredictable errors. Programming errors are also possible. I have special mechanisms to log/report error states. I don't want to stop the overall process on an unexpected error, whichever the error nature is. I rather want to process as many items as possible. Can I catch base Exception as in above?
Of course, and the reasons you provided are excellent cases in which you would do just that. Sometimes you don't have anything special to do with the exception, but you still don't want to let it go on, whatever type of exception it is (this is also very common in apps that need to run forever, such as servers. Many errors would be caught and logged, and the system will just try its best to continue running as best as it can.
Do, however, note the difference between except Exception [as e]: and except:, as described in this Q&A: Difference between except: and except Exception as e: in Python
I've got several methods which can raise my custom exception. After the exception has been raised, I need to handle it, let's say log message to console and save it into database.
I was thinking about a crazy workaround described in post title - I could move that custom code with logging and DB-saving into __init__ method of my custom exception, so everytime the exception is raised, I would just silence it, since all needed stuff would be done on exception initialization.
I'm aware of that the exception __init__ itself can raise another exception, but that can be handled too :)
Did anyone tried that?
And why is it crazy idea? :)
-
edit:
I know it's kinda crazy, I'm just curious of your opinion. I will try to enclose what I want to achieve:
I'm working with remote data and when communicating to other servers through network, some problems could appear, every one at some other place:
1. network error - on creating connection.
2. HTTP errors (404, 500, etc.) - after connection;
3. remote server can return some other errors too
Since these problems occur in several different places, I created my custom exception:
class CustomException(Exception):
pass
and raise it everywhere, when I can catch them, for example:
try:
conn.open(url)
except HTTPException as e:
raise CustomException('http')
That's just a pseudo-example.
This CustomException is catched somewher higher and in almost every place I handle this the same way, i.e.:
try:
place.populate()
except CustomException as e:
handle_exception(e)
return False
And handle_exception saves info about problem to database, does other things also like saving place's object status and acccess date, but always updates the same things.
So I was just wondering if putting that handle_exception code inside __init__ would be really crazy idea, since it is done everytime the exception is raised.
Thanks for your opinions!
If you were to do something like this, which as others have said is a questionable goal, I'd suggest routing the messages through a logging.Logger in your exception's __init__, then attaching logging.Handlers for logging to the DB, printing to the console, or what ever you want. Then users of your libraries can control (and extend) the system in a much easier, more standardized way.
It seems like you're suggesting two separate things. Firstly, there's the idea of having some custom code within an exception, which is far from crazy and is quite doable.
Example:
class CustomException(Exception):
def __init__(self, message, Errors):
# do stuff here
However, you alluded to automatically catching the exception every time it's raised. The reason that isn't done is pretty simple, if it's not an error then don't make it an exception. I have trouble imagining that there would be any reason to make a class an exception if you don't plan on actually doing anything with it at any point.
Personally if you want substantial logic on raising an exception (a little hard to comment on this without knowing your particular goal) I would suggest putting a logger in the __init__ method and then put your logic directly in a try/except which is afterall, exactly what they are for.