Exception handling for 'soft' errors - python

I'm running means2 from scipy, and although I'm getting an error message:
/usr/lib/python2.7/dist-packages/scipy/cluster/vq.py:600: UserWarning:
One of the clusters is empty. Re-run kmean with a different
initialization. warnings.warn("One of the clusters is empty. "
when running the following code in a loop:
training_2= numpy.random.random_integers(0, 400, size=[50,1]).astype(numpy.float32)
cent, clus= kmeans2(training_2, 3, minit='points')
the loop doesn't terminate. I've noticed this behavior with numpy and OpenCV as well.
The usual try except block doesn't catch these (for lack of a better term) 'soft' errors; is there another way to handle these errors?
If this isn't possible directly, then is there a way to have a python script read it's own output as it runs?

It is not an exception, it is a warning:
Warning messages are typically issued in situations where it is useful
to alert the user of some condition in a program, where that condition
(normally) doesn’t warrant raising an exception and terminating the
program.
How to handle it, depends on the warning and whether it affects your program execution, you can:
suppress warnings
see Temporarily Suppressing Warnings
treat warnings as errors
see How do I catch a numpy warning like it's an exception (not just for testing)?

Related

python exception not reported

Python version 2.7.13
I was debugging an issue in my program. I use pycharm debug to find that there is a key error, dict try to get a key not exist in it, but there is no traceback reported. In my program, we use sys.excepthook = <my_callback_func> to catch the uncaught trace, it usually works, but in my issue, it didn't work, the code didn't enter <my_callback_func>.
Maybe there is some other try catch code which catch this exception but I can't find it, because the program is quite large and complex.
So here is my question, is there a way to find which try catch code capture this exception, I use pycharm debug tools, but when I step debug it, the code never go to the catch code. or maybe there is some other reason that the sys.excepthook not working.

Between raise Error and logging the Error, which one is a better practice in Python?

Python tutorial have a section called Errors and Exceptions which use the structure as;
try:
[statement]
except [Built-in ExceptionType]:
do something when an exception has been captured.
finally:
do something whether exception occurred or not.
This can also handle by raise an Error directly, too.
try:
raise [Built-in ExceptionType]
except [Built-in ExceptionType above] as e:
print(f'foo error: {e}')
There are several built-in ExceptionType which the good practice is developer should catch every specific exceptions type that developer should look for. (refer from Why is "except: pass" a bad programming practice?)
However, after I reading logging section. I'm thinking about I want to log the error into the log file and let user to aware the error (maybe just print the text and user can inform IT support) rather than throwing an exception on screen. Therefore, instead of try / except combo above then I can use the following error message and logged it.
if len(symbol) != 1:
error_message = f'Created box with symbol={symbol}, width={width} and height={height}, Symbol needs to be a string of length 1'
logger.error(error_message)
print(f'\nError! symbol={symbol}, width={width} and height={height} -- Symbol needs to be a string of length 1')
return
I doubt that what the better current practice is between:
a) raise Error on screen and
b) logging the Error for further investigation?
c) Other method (please suggest me)
I hope that I'm not try to comparing two different things.For me, I want to choose b) and store the error in log. this is easier for investigation and does not give unnecessary information to the users. but I do not sure I'm on the right track or not. Thank you very much.
Logging and raising exceptions are two different things for two different purposes. Logs let you inspect what your program did after the fact. Raising exceptions has important effects on the program flow right now. Sometimes you want one, sometimes you want the other, sometimes you want both.
The question is always whether an error is expected or unexpected, whether you have some plan what to do in case it occurs, and whether it is useful to notify anyone about the occurrence of the error or not.
Expected errors that you have a "backup plan" for should be caught and handled, that is regular program control flow. Unexpected errors probably should halt the program, or at least the particular function in which they occurred. If a higher up caller feels like handling the exception, let them. And whether to log an error or not (in addition to handling or not handling it) depends on whether anyone can glean any useful insight from that log entry or whether it would just be noise.
As deceze already mentions, logging and exceptions are two totally distinct things. As a general rule:
you raise an exception when you encounter an unexpected / invalid condition that you cannot handle at this point of the code. What will became of this exception (whether someone up in the call stack will handle it or not) is none of your concern at this point
you only catch an exception when you can handle it or when you want to log it (eventually with additionnal context informations) THEN reraise it
at your application's top-level, you eventually add a catchall exception handler that can log the exception and decide on the best way to deal with the situation depending on the type of application (command-line script, GUI app, webserver etc...).
Logging is mostly a developper/admin tool used for post-mortem inspection, program behaviour analysis etc, it's neither an error handling tool nor an end-user UI feature. Your example:
if len(symbol) != 1:
error_message = f'Created box with symbol={symbol}, width={width} and height={height}, Symbol needs to be a string of length 1'
logger.error(error_message)
print(f'\nError! symbol={symbol}, width={width} and height={height} -- Symbol needs to be a string of length 1')
return
is a perfect antipattern. If your function expects a one character string and the caller passed anything else then the only sane solution is to raise an exception and let the caller deal with it. How it does is, once again, none of your concerns.
You should log all errors no matter if they are thrown or handled for log analysis and refactoring amongst other purposes.
Whether to throw or to handle the error usually depends on the intended purpose of the code and the severity of the error.
Although "throwing" should be only used in debugging and handled
'gracefully' by dedicated exception code in the production version of the
application. No user likes crashes.
If the error impacts the business logic, end result of the code, or can cause damages to the user of the software, you want to terminate the execution. You do it either by throwing the error or handling it through a dedicated error handling procedure that shutdowns the program.
If the error is not severe and does not impact the normal functionality of the program, you can choose whether you want to handle it or throw it based on the best practices in your team and requirements of the project.
From the Docs:
Returns a new instance of the SMTPHandler class. The instance is initialized with the from and to addresses and subject line of the email. The toaddrs should be a list of strings.
logging.handlers.SMTPHandler can be used to send logged error message.
import logging
import logging.handlers
smtp_handler = logging.handlers.SMTPHandler(mailhost=("example.com", 25),
fromaddr="from#example.com",
toaddrs="to#example.com",
subject="Exception notification")
logger = logging.getLogger()
logger.addHandler(smtp_handler)
you can break the logic here if the code execution needs to be stopped.
Also look around this answer i have referred earlier collate-output-in-python-logging-memoryhandler-with-smtphandler

proper way of displaying error messages

According to Python's Logging tutorial, logging.error is to be used to "report suppression of an error without raising an exception.", whereas if I wanted to report an error I'm supposed to just raise an exception and not use logging at all.
What if I want to report an error and then raise an exception?
For example:
try:
os.path.getsize('/nonexistent')
except os.error as error:
logging.error(str(error))
logging.error('something went wrong')
raise SystemExit(1)
This results in the following lines being printed to the standard error:
ERROR:root:[Errno 2] No such file or directory: '/nonexistent'
ERROR:root:something went wrong
It seems sensible. Or:
try:
os.path.getsize('/nonexistent')
except os.error as error:
logging.error(str(error))
raise SomeUserDefinedException('something went wrong')
Is this bad practice?
logging is used to report.
raise is an action which interrupts the normal flow of the program.
Typically, you raise exceptions where there is an actual flow to interrupt, and when you want to report, you can use logging, which will log errors and exceptions by default.
The logging part usually comes as a wrapper around a module, class, or middleware in your program.
logging within the code should only be used for DEBUG purposes, or for INFO purpose if you wanted to keep track of some information in your server logs for example. Not for ERROR.
I do not believe that is a bad practice. Exceptions and logging serve two different purposes and in any given situation you may want one, the other or both. Exceptions are for programmatic consumption and control flow within your code, while logging is for external consumption, by either a human or some sort of automated log watcher (Splunk for example). For instance, you may log the values of certain variables to help you figure out what happened and improve the code, though in that case, a level of debug would be more appropriate. Or you may want a systems administrator to be aware of the issue and potentially take some action. In that case, setting the log level to error and using something like a syslog logging handler makes sense.

Python -- when to choose logging.error() over raise Exception()?

I am using logging throughout my code for easier control over logging level. One thing that's been confusing me is
when do you prefer logging.error() to raise Exception()?
To me, logging.error() doesn't really make sense, since the program should stop when encountered with an error, like what raise Exception() does.
In what scenarios, do we put up an error message with logging.error() and let the program keep running?
Logging is merely to leave a trace of events for later inspection but does absolutely nothing to influence the program execution; raising an exception allows you to signal an error condition to higher up callers and let them handle it. Typically you use both together, it's not an either-or.
That purely depends on your process flow but it basically boils down to whether you want to deal with an encountered error (and how), or do you want to propagate it to the user of your code - or both.
logging.error() is typically used to log when an error occurs - that doesn't mean that your code should halt and raise an exception as there are recoverable errors. For example, your code might have been attempting to load a remote resource - you can log an error and try again at later time without raising an exception.
btw. you can use logging to log an exception (full, with a stack trace) by utilizing logging.exception().

Force a unit test case to ERROR in Python

I went through the Unit Test Case module of python and found that there is way to force a test case to fail using the TestCase.fail() object. However I did not find anything that would force a test case to error. Any ideas on how this could be done?
EDIT:
More explaination
There are few test scripts which were written long ago and which I'm using right now for testing a firmware. These test scripts use the UnitTest module of python. The test scripts also import a bunch of other user-written modules. All these user written modules throw few exceptions. So whenever a test case doesn't call methods from user-written modules with correct inputs exception are thrown. the UnitTest module rightly flags all such test cases as errors. But by looking at the output of UnitTest module, which is just a Traceback to the line where exception was generated, it is not immediately clear to me because of which input the exception was generated. So I used try-except construct in the test scripts to catch the exceptions and print out the input which was the reason for exception. But since I handled the exceptions UnitTest was flagging these test cases as pass. Using the raise statement in the except block, as pointed out by #alecxe, solved this issue.
Just use raise to raise an exception:
raise Exception('Manually raised exception')

Categories