I am running different sets of data to identify best modeling algorithm for each dataset. I loop through each datasets to check various algorithms and select the best models based on test score. I know that some of my datasets not going to converge for specific models (i.e: LogisticRegression)
and getting converging warning (i.e:"lbfgs failed to converge (status=1):"). I don't want to ignore the warning. My goal is to return score for models that converge and don't return any value if I get this convergence warning.
I am able to work around this by turning this warning into error using "warnings.filterwarnings('error',category=ConvergenceWarning, module='sklearn')" and then go through try and except to get what I want. The problem with this method is that if there is any other error beside sklearn convergance warning it will bypass the try line and I wouldn't be able to know what cause the error. Is there any other way to capture this warning beside turning it to error?
Here is the simplified overview of my code ( data not included as its a big datasets and I don't think is relevant to the question). Most of stackoverflow questions that I was able to find is about how to supress the error(How to disable ConvergenceWarning using sklearn?)or to turn this warning into error and I didn't find any other method to capture the warning without turning it to error.
from sklearn.linear_model import LogisticRegression
from sklearn.exceptions import ConvergenceWarning
warnings.filterwarnings('error',category=ConvergenceWarning, module='sklearn')
try:
model=LogisticRegression().fit(x_train,y_train)
predict=model.predict(x_test)
except:
print('model didnt converge')
There are a couple things that can help you here.
First, you can specify what kind of Exception you are looking for, any you can specify multiple except clauses. Here is an example from the docs:
import sys
try:
f = open('myfile.txt')
s = f.readline()
i = int(s.strip())
except OSError as err:
print("OS error: {0}".format(err))
except ValueError:
print("Could not convert data to an integer.")
except:
print("Unexpected error:", sys.exc_info()[0])
raise
The other thing to notice in the above is the except OSError as err. Using this syntax, you can print the error message associated with the error.
Related
I am trying to loop over a series of pages and extract some info. However, in certain pages some exceptions occur and I need to deal with them. I created the following function to try to deal with them. See below:
def iferr(x):
try:
x
except (Exception, TypeError, AttributeError) as e:
pass
I intend to use as part of code like this:
articles = [[iferr(dp[0].find('span', class_='citation')),\
iferr(dp[0].find('div', class_='abstract')),\
iferr(dp[0].find('a', rel='nofollow')['href'])] for dp in data]
The idea is that if, for example, dp[0].find('a', rel='nofollow')['href'] leads to an error (fails), it will simply ignore it (fill it with a blank or a None).
However, whenever an error/exception occurs in one of the three elements it does not 'pass'. It just tells me that the error has occurred. There errors it displays are those I listed in the 'except' command which I assume would be dealt with.
EDIT:
Per Michael's suggestion, I was able to see that the order in which iferr processes what is going on would always prompt the error before he try. So I worked on workaround:
def fndwoerr(d,x,y,z,h):
try:
if not h:
d.find('x',y = 'z')
else:
d.find('x',y = 'z')['h']
except (Exception, TypeError, AttributeError) as e:
pass
...
articles = [[fndwoerr(dp[0],'span','class_','citation',None),\
fndwoerr(dp[0],'div','class_','abstract',None),\
fndwoerr(dp[0], 'a', 'rel','nofollow','href')] for dp in data]
Now it runs without prompting an error. However, everything returned becomes None. I am pretty sure it has to do with he way the parameters are entered. y should not be displayed as a string in the find function, whereas z has. However, I input both as string when i call the function. How can I go about this?
Example looks a bit strange, so it would be a good idea to improve the question, so that we can reproduce your issue easily. May read how to create minimal, reproducible example
The idea is that if, for example, dp[0].find('a',
rel='nofollow')['href'] leads to an error (fails), it will simply
ignore it (fill it with a blank or a None).
What about checking if element is available with an if-statement?
dp[0].find('a', rel='nofollow').get('href']) if dp[0].find('a', rel='nofollow') else None
or with walrus operator from python 3.8:
l.get('href']) if (l:=dp[0].find('a', rel='nofollow')) else None
Example
from bs4 import BeautifulSoup
soup = BeautifulSoup('<h1>This is a Heading</h1>', 'html.parser')
for e in soup.select('h1'):
print(e.find('a').get('href') if e.find('a') else None)
I'm working on a convoluted FOSS project that utilizes GTK+3. When a flow graph is generated and attempted to run it, it generates the following error:
'Page' object has no attribute 'get_flow_graph'
There are 30 different files that have the generic "...object has no attribute..." exception listed in the code, and there are 4 files that call the function get_flow_graph().
So what I want to figure out is which of the 30 files that generate that particular error message is being executed, and preferably which of the 4 files with the function are causing the error in the first place.
I'm trying to use Python's traceback module to figure out where, specifically, the exception is being generated. I think I figured out the file that is calling the function that ultimately errors out, but I can't seem to get the traceback module to provide much more.
For example, if I wrap the function like this:
try:
fg = self.page.get_flow_graph()
except Exception:
traceback.print_exc()
then I just get
File "<redacted>", line 66, in _popen
fg = self.page.get_flow_graph()
AttributeError: 'Page' object has no attribute 'get_flow_graph'
'Page' object has no attribute 'get_proc'
as the output. So I get the original exception but a new get_proc error that doesn't help me but is obviously associated with trying to use traceback.
Maybe I'm not putting the trace in the correct file/location, or maybe I'm asking too much, but how should I write it to figure out the actual stack trace for the original AttributeError?
Does using
except AttributeError as e:
print(e.__traceback__.tb_lineno)
print(e.__traceback__.tb_frame)
instead, helps you further? (really asking, not being ironic)
I was just trying to catch an OptimizeWarning thrown by the scipy.optimize.curve_fit function, but I realized it was not recognized as a valid exception.
This is a non-working simple idea of what I'm doing:
from scipy.optimize import curve_fit
try:
popt, pcov = curve_fit(some parameters)
except OptimizeWarning:
print 'Maxed out calls.'
# do something
I looked around the docs but there was nothing there.
Am I missing something obvious or is it simply not defined for some reason?
BTW, this is the full warning I get and that I want to catch:
/usr/local/lib/python2.7/dist-packages/scipy/optimize/minpack.py:604: OptimizeWarning: Covariance of the parameters could not be estimated
category=OptimizeWarning)
You can require that Python raise this warning as an exception using the following code:
import warnings
from scipy.optimize import OptimizeWarning
warnings.simplefilter("error", OptimizeWarning)
# Your code here
Issues with warnings
Unfortunately, warnings in Python have a few issues you need to be aware of.
Multiple filters
First, there can be multiple filters, so your warning filter can be overridden by something else. This is not too bad and can be worked around with the catch_warnings context manager:
import warnings
from scipy.optimize import OptimizeWarning
with warnings.catch_warnings():
warnings.simplefilter("error", OptimizeWarning)
try:
# Do your thing
except OptimizeWarning:
# Do your other thing
Raised Once
Second, warnings are only raised once by default. If your warning has already been raised before you set the filter, you can change the filter, it won't raise the warning again.
To my knowledge, there unfortunately is not much you can do about this. You'll want to make sure you run the warnings.simplefilter("error", OptimizeWarning) as early as possible.
It seems to me that commonly you may want a Python program to print (usually to standard error) only the last line of the stack trace, e.g.:
IOError: Error reading file 'b'plunk'': b'failed to load external entity "plunk"'
I've got this solution:
def print_error(ex:Exception, file) -> None:
print('{0}: {1}'.format(ex.__class__.__name__, ex), file=file)
Example usage:
try:
crash in some manner
except Exception as ex:
print_error(ex, sys.stderr)
There is nothing particularly wrong with this, but this feature seems so basic that I can't help but wonder if there isn't a simpler way to do it. Am I missing something? Or is this a good solution?
I don't know if there's a better way of doing it, but considering that it's all of 6 lines of code and I can't imagine it takes long to carry out at all, I don't think that you need a better one.
Here is part of a stack-trace from a recent run of an unreliable application written in Python which controls another application written in Excel:
pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, None, None, None, 0, -2146788248), None)
Obviously something has gone wrong ... but what?[1] These COM error codes seem to be excessively cryptic.
How can I decode this error message? Is there a table somewhere that allows me to convert this numerical error code into something more meaningful?
[1] I actually know what went wrong in this case, it was attempting to access a Name prperty on a Range object which did not have a Name property... not all bugs are this easy to find!
You are not doing anything wrong. The first item in your stack trace (the number) is the error code returned by the COM object. The second item is the description associated with the error code which in this case is "Exception Occurred". pywintypes.com_error already called the equivalent of win32api.FormatMessage(errCode) for you. We'll look at the second number in a minute.
By the way, you can use the "Error Lookup" utility that comes in Visual Studio (C:\Program Files\Microsoft Visual Studio 9.0\Common7\Tools\ErrLook.exe) as a quick launching pad to check COM error codes. That utility also calls FormatMessage for you and displays the result. Not all error codes will work with this mechanism, but many will. That's usually my first stop.
Error handling and reporting in COM is a bit messy. I'll try to give you some background.
All COM method calls will return a numeric code called an HRESULT that can indicate success or failure. All forms of error reporting in COM build on top of that.
The codes are commonly expressed in hex, although sometimes you will see them as large 32-bit numbers, like in your stack trace. There are all kinds of predefined return codes for common results and problems, or the object can return custom numeric codes for special situations. For example, the value 0 (called S_OK) universally means "No error" and 0x80000002 is E_OUTOFMEMORY. Sometimes the HRESULT codes are returned by the object, sometimes by the COM infrastructure.
A COM object can also choose to provide much richer error information by implementing an interface called IErrorInfo. When an object implements IErrorInfo, it can provide all kinds of detail about what happened, such as a detailed custom error message and even the name of a help file that describes the problem. In VB6 and VBA. the Err object allows you to access all that information (Err.Description, etc).
To complicate matters, late bound COM objects (which use a mechanism called COM Automation or IDispatch) add some layers that need to be peeled off to get information out. Excel is usually manipulated via late binding.
Now let's look at your situation again. What you are getting as the first number is a fairly generic error code: DISP_E_EXCEPTION. Note: you can usually figure out the official name of an HRESULT by googling the number, although sometimes you will have to use the hex version to find anything useful.
Errors that begin with DISP_ are IDISPATCH error codes. The error loosely means "There was a COM exception thrown by the object", with more information packed elsewhere (although I don't quite know where; I'll have to look it up).
From what I understand of pywintypes.com_error, the last number in your message is the actual error code that was returned by the object during the exception. It's the actual numeric code that you would get out of VBA's Err.Number.
Unfortunately, that second code -2146788248 (0x800A9C68) is in the range reserved for custom application-defined error messages (in VBA: VbObjectError + someCustomErrorNumber), so there is no centralized meaning. The same number can mean entirely different things for different programs.
In this case, we have reached a dead end:
The error code is "custom", and the application needs to document what it is, except that Excel doesn't. Also, Excel (or the actual source of the error) doesn't seem to be providing any more information via IErrorInfo.
Excel is notorious (to me at least) for cryptic error codes from automation and obscure situations that cause them. This is especially so for errors that one could consider "design-time errors" ("you should have known better than calling a method that doesn't exist in the object"). Instead of a nice "Could not read the Name property", you get "Run-time error '1004': Application defined or object-defined error" (which I just got by trying to access a Name property on a Range, from VBA in Excel). That is NOT very helpful.
The problem is not routed on Python or it's interface to Excel. Excel itself doesn't explain what happened, even to VBA.
However, the general procedure above remains valid. If you get an error from Excel in the future, you might get a better error message that you can track the same way.
Good luck!
Do it like this:
try:
[whatever code]
except pythoncom.com_error as error:
print(win32api.FormatMessage(error.excepinfo[5]))
More information on digesting the pythoncom.com_error object here: https://web.archive.org/web/20170831073447/http://docs.activestate.com/activepython/3.2/pywin32/com_error.html
Yes try the win32api module:
import win32api
e_msg = win32api.FormatMessage(-2147352567)
You can grab any codes returned from the exception and pass them to FormatMessage. Your example had 2 error codes.
Specifically for pythoncom, the errors codes that result are more than cryptic. This is because pythoncom represents them internally as a 32bit signed integer, when the correct representation is a 32bit unsigned integer. As a result, the conversion that you end up seeing in the stack trace is incorrect.
In particular, your exception, according to pythoncom, is -2147352567, and your (for lack of a better word) Err.Number is -2146788248.
This however causes some issues when watching for specific errors, like below:
DISP_E_EXCEPTION = 0x80020009
#...
#except pywintypes.com_error as e:
# print repr(e)
# #pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, None, None, None, 0, -2146788248), None)
# hr = e.hresult
hr = -2147352567
if hr == DISP_E_EXCEPTION:
pass #This never occurs
else:
raise
To see why this has issues, lets look into these error codes:
>>> DISP_E_EXCEPTION = 0x80020009
>>> DISP_E_EXCEPTION
2147614729L
>>> my_hr = -2147352567
>>> my_hr == DISP_E_EXCEPTION
False
Again, this is because python sees the constant declared as positive, and pythoncom's incorrect declaration interpreted it as negative. Of course, the most obvious solution fails:
>>> hex(my_hr)
'-0x7ffdfff7'
The solution is to properly interpret the number. Luckily, pythoncom's representation is reversible. We need to interpret the negative number as a 32 bit signed integer, then interpret that as an unsigned integer:
def fix_com_hresult(hr):
import struct
return struct.unpack("L", struct.pack("l", hr))[0]
>>> DISP_E_EXCEPTION = 0x80020009
>>> my_hr = -2147352567
>>> my_hr == DISP_E_EXCEPTION
False
>>> fixed_hr = fix_com_hresult(my_hr)
>>> fixed_hr
2147614729L
>>> fixed_hr == DISP_E_EXCEPTION
True
So, putting it all together, you need to run fix_com_hresult() on that result from pythoncom, essentially all the time.
Since normally you need to do this when checking for exceptions, I created these functions:
def fix_com_exception(e):
e.hresult = fix_com_hresult(e.hresult)
e.args = [e.hresult] + list(e.args[1:])
return e
def fix_com_hresult(hr):
import struct
return struct.unpack("L", struct.pack("l", hr))[0]
which can then be used how you expect:
DISP_E_EXCEPTION = 0x80020009
try:
#failing call
except pywintypes.com_error as e:
print repr(e)
#pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, None, None, None, 0, -2146788248), None)
fix_com_exception(e)
print repr(e)
#pywintypes.com_error: (2147614729L, 'Exception occurred.', (0, None, None, None, 0, -2146788248), None)
if e.hresult == DISP_E_EXCEPTION:
print "Got expected failure"
else:
raise
I was unable to find a MSDN document listing all HRESULTs, but I found this: http://www.megos.ch/support/doserrors_e.txt
Also, since you have it, fix_com_hresult() should also be run on your extended error code (-2146788248), but as Euro Micelli said, it doesn't help you in this particular instance :)
No-one has yet mentioned the strerror attribute of the pywintypes.com_error Exception. This returns the result of FormatMessage for the error code. So instead of doing it yourself like this
try:
[whatever code]
except pythoncom.com_error as error:
print(win32api.FormatMessage(error.excepinfo[5]))
You can just do this:
try:
[whatever code]
except pythoncom.com_error as error:
print(error.strerror)
Note it will return None if you have a non-standard HRESULT :(