It seems to me that commonly you may want a Python program to print (usually to standard error) only the last line of the stack trace, e.g.:
IOError: Error reading file 'b'plunk'': b'failed to load external entity "plunk"'
I've got this solution:
def print_error(ex:Exception, file) -> None:
print('{0}: {1}'.format(ex.__class__.__name__, ex), file=file)
Example usage:
try:
crash in some manner
except Exception as ex:
print_error(ex, sys.stderr)
There is nothing particularly wrong with this, but this feature seems so basic that I can't help but wonder if there isn't a simpler way to do it. Am I missing something? Or is this a good solution?
I don't know if there's a better way of doing it, but considering that it's all of 6 lines of code and I can't imagine it takes long to carry out at all, I don't think that you need a better one.
Related
I am trying to loop over a series of pages and extract some info. However, in certain pages some exceptions occur and I need to deal with them. I created the following function to try to deal with them. See below:
def iferr(x):
try:
x
except (Exception, TypeError, AttributeError) as e:
pass
I intend to use as part of code like this:
articles = [[iferr(dp[0].find('span', class_='citation')),\
iferr(dp[0].find('div', class_='abstract')),\
iferr(dp[0].find('a', rel='nofollow')['href'])] for dp in data]
The idea is that if, for example, dp[0].find('a', rel='nofollow')['href'] leads to an error (fails), it will simply ignore it (fill it with a blank or a None).
However, whenever an error/exception occurs in one of the three elements it does not 'pass'. It just tells me that the error has occurred. There errors it displays are those I listed in the 'except' command which I assume would be dealt with.
EDIT:
Per Michael's suggestion, I was able to see that the order in which iferr processes what is going on would always prompt the error before he try. So I worked on workaround:
def fndwoerr(d,x,y,z,h):
try:
if not h:
d.find('x',y = 'z')
else:
d.find('x',y = 'z')['h']
except (Exception, TypeError, AttributeError) as e:
pass
...
articles = [[fndwoerr(dp[0],'span','class_','citation',None),\
fndwoerr(dp[0],'div','class_','abstract',None),\
fndwoerr(dp[0], 'a', 'rel','nofollow','href')] for dp in data]
Now it runs without prompting an error. However, everything returned becomes None. I am pretty sure it has to do with he way the parameters are entered. y should not be displayed as a string in the find function, whereas z has. However, I input both as string when i call the function. How can I go about this?
Example looks a bit strange, so it would be a good idea to improve the question, so that we can reproduce your issue easily. May read how to create minimal, reproducible example
The idea is that if, for example, dp[0].find('a',
rel='nofollow')['href'] leads to an error (fails), it will simply
ignore it (fill it with a blank or a None).
What about checking if element is available with an if-statement?
dp[0].find('a', rel='nofollow').get('href']) if dp[0].find('a', rel='nofollow') else None
or with walrus operator from python 3.8:
l.get('href']) if (l:=dp[0].find('a', rel='nofollow')) else None
Example
from bs4 import BeautifulSoup
soup = BeautifulSoup('<h1>This is a Heading</h1>', 'html.parser')
for e in soup.select('h1'):
print(e.find('a').get('href') if e.find('a') else None)
I'm working on a convoluted FOSS project that utilizes GTK+3. When a flow graph is generated and attempted to run it, it generates the following error:
'Page' object has no attribute 'get_flow_graph'
There are 30 different files that have the generic "...object has no attribute..." exception listed in the code, and there are 4 files that call the function get_flow_graph().
So what I want to figure out is which of the 30 files that generate that particular error message is being executed, and preferably which of the 4 files with the function are causing the error in the first place.
I'm trying to use Python's traceback module to figure out where, specifically, the exception is being generated. I think I figured out the file that is calling the function that ultimately errors out, but I can't seem to get the traceback module to provide much more.
For example, if I wrap the function like this:
try:
fg = self.page.get_flow_graph()
except Exception:
traceback.print_exc()
then I just get
File "<redacted>", line 66, in _popen
fg = self.page.get_flow_graph()
AttributeError: 'Page' object has no attribute 'get_flow_graph'
'Page' object has no attribute 'get_proc'
as the output. So I get the original exception but a new get_proc error that doesn't help me but is obviously associated with trying to use traceback.
Maybe I'm not putting the trace in the correct file/location, or maybe I'm asking too much, but how should I write it to figure out the actual stack trace for the original AttributeError?
Does using
except AttributeError as e:
print(e.__traceback__.tb_lineno)
print(e.__traceback__.tb_frame)
instead, helps you further? (really asking, not being ironic)
Necessarily within a python program and given an str variable that contains C code, I want to check fast if this code is syntactically correct, or not. Essentially, I only need to pass it through the compiler's front end.
My current implementation uses a temp file to dump the string and calls a clang process with subprocess (non-working code below to illustrate my solution). This is very slow for my needs.
src = "int main(){printf("This is a C program\n"); return 0;}"
with open(temp_file, 'w') as f:
f.write(src)
cmd = ["clang", abs_path(f), flags]
subprocess.Popen(cmd)
## etc..
After looking around, I found out about clang.cindex module (pip clang), which I tried out. After reading a bit the main module, lines 2763-2837 (specifically line 2828) led me to the conclusion that the following code snippet will do what I need:
import clang.cindex
......
try:
unit = clang.cindex.TranslationUnit.from_source(temp_code_file, ##args, etc.)
print("Compiled!")
except clang.cindex.TranslationUnitLoadError:
print("Did not compile!")
However, it seems that even if the source file contains obvious syntactic errors, an exception is not raised. Anyone knows what am I missing to make this work ?
On a general context, any suggestions on how to do this task as fast as possible would be more than welcome. Even with clang.cindex, I cannot get away from writing my string-represented code to a temp file, which may be an additional overhead. Writing a python parser could solve this but is an overkill at the moment, no matter how much I need speed.
The compilation itself succeeds even if the file has syntax errors. Consider the following example:
import clang.cindex
with open('broken.c', 'w') as f:
f.write('foo bar baz')
unit = clang.cindex.TranslationUnit.from_source('broken.c')
for d in unit.diagnostics:
print(d.severity, d)
Run it and you will get
3 broken.c:1:1: error: unknown type name 'foo'
3 broken.c:1:8: error: expected ';' after top level declarator
The severity member of is an int, with the value from the enum CXDiagnosticSeverity with values
CXDiagnostic_Ignored = 0
CXDiagnostic_Note = 1
CXDiagnostic_Warning = 2
CXDiagnostic_Error = 3
CXDiagnostic_Fatal = 4
I think I have the opposite problem as described here. I have one process writing data to a log, and I want a second process to read it, but I don't want the 2nd process to be able to modify the contents. This is potentially a large file, and I need random access, so I'm using python's mmap module.
If I create the mmap as read/write (for the 2nd process), I have no problem creating ctypes object as a "view" of the mmap object using from_buffer. From a cursory look at the c-code, it looks like this is a cast, not a copy, which is what I want. However, this breaks if I make the mmap ACCESS_READ, throwing an exception that from_buffer requires write privileges.
I think I want to use ctypes from_address() method instead, which doesn't appear to need write access. I'm probably missing something simple, but I'm not sure how to get the address of the location within an mmap. I know I can use ACCESS_COPY (so write operations show up in memory, but aren't persisted to disk), but I'd rather keep things read only.
Any suggestions?
I ran into a similar issue (unable to setup a readonly mmap) but I was using only the python mmap module. Python mmap 'Permission denied' on Linux
I'm not sure it is of any help to you since you don't want the mmap to be private?
Ok, from looking at the mmap .c code, I don't believe it supports this use case. Also, I found that the performance pretty much sucks - for my use case. I'd be curious what kind performance others see, but I found that it took about 40 sec to walk through a binary file of 500 MB in Python. This is creating a mmap, then turning the location into a ctype object with from_buffer(), and using the ctypes object to decipher the size of the object so I could step to the next object. I tried doing the same thing directly in c++ from msvc. Obviously here I could cast directly into an object of the correct type, and it was fast - less than a second (this is with a core 2 quad and ssd).
I did find that I could get a pointer with the following
firstHeader = CEL_HEADER.from_buffer(map, 0) #CEL_HEADER is a ctypes Structure
pHeader = pointer(firstHeader)
#Now I can use pHeader[ind] to get a CEL_HEADER object
#at an arbitrary point in the file
This doesn't get around the original problem - the mmap isn't read-only, since I still need to use from_buffer for the first call. In this config, it still took around 40 sec to process the whole file, so it looks like the conversion from a pointer into ctypes structs is killing the performance. That's just a guess, but I don't see a lot of value in tracking it down further.
I'm not sure my plan will help anyone else, but I'm going to try to create a c module specific to my needs based on the mmap code. I think I can use the fast c-code handling to index the binary file, then expose only small parts of the file at a time through calls into ctypes/python objects. Wish me luck.
Also, as a side note, Python 2.7.2 was released today (6/12/11), and one of the changes is an update to the mmap code so that you can use a python long to set the file offset. This lets you use mmap for files over 4GB on 32-bit systems. See Issue #4681 here
Ran into this same problem, we needed the from_buffer interface and wanted read only access. From the python docs https://docs.python.org/3/library/mmap.html "Assignment to an ACCESS_COPY memory map affects memory but does not update the underlying file."
If it's acceptable for you to use an anonymous file backing you can use ACCESS_COPY
An example: open two cmd.exe or terminals and in one terminal:
mm_file_write = mmap.mmap(-1, 4096, access=mmap.ACCESS_WRITE, tagname="shmem")
mm_file_read = mmap.mmap(-1, 4096, access=mmap.ACCESS_COPY, tagname="shmem")
write = ctypes.c_int.from_buffer(mm_file_write)
read = ctypes.c_int.from_buffer(mm_file_read)
try:
while True:
value = int(input('enter an integer using mm_file_write: '))
write.value = value
print('updated value')
value = int(input('enter an integer using mm_file_read: '))
#read.value assignment doesnt update anonymous backed file
read.value = value
print('updated value')
except KeyboardInterrupt:
print('got exit event')
In the other terminal do:
mm_file = mmap.mmap(-1, 4096, access=mmap.ACCESS_WRITE, tagname="shmem")
i = None
try:
while True:
new_i = struct.unpack('i', mm_file[:4])
if i != new_i:
print('i: {} => {}'.format(i, new_i))
i = new_i
time.sleep(0.1)
except KeyboardInterrupt:
print('Stopped . . .')
And you will see that the second process does not receive updates when the first process writes using ACCESS_COPY
Here is part of a stack-trace from a recent run of an unreliable application written in Python which controls another application written in Excel:
pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, None, None, None, 0, -2146788248), None)
Obviously something has gone wrong ... but what?[1] These COM error codes seem to be excessively cryptic.
How can I decode this error message? Is there a table somewhere that allows me to convert this numerical error code into something more meaningful?
[1] I actually know what went wrong in this case, it was attempting to access a Name prperty on a Range object which did not have a Name property... not all bugs are this easy to find!
You are not doing anything wrong. The first item in your stack trace (the number) is the error code returned by the COM object. The second item is the description associated with the error code which in this case is "Exception Occurred". pywintypes.com_error already called the equivalent of win32api.FormatMessage(errCode) for you. We'll look at the second number in a minute.
By the way, you can use the "Error Lookup" utility that comes in Visual Studio (C:\Program Files\Microsoft Visual Studio 9.0\Common7\Tools\ErrLook.exe) as a quick launching pad to check COM error codes. That utility also calls FormatMessage for you and displays the result. Not all error codes will work with this mechanism, but many will. That's usually my first stop.
Error handling and reporting in COM is a bit messy. I'll try to give you some background.
All COM method calls will return a numeric code called an HRESULT that can indicate success or failure. All forms of error reporting in COM build on top of that.
The codes are commonly expressed in hex, although sometimes you will see them as large 32-bit numbers, like in your stack trace. There are all kinds of predefined return codes for common results and problems, or the object can return custom numeric codes for special situations. For example, the value 0 (called S_OK) universally means "No error" and 0x80000002 is E_OUTOFMEMORY. Sometimes the HRESULT codes are returned by the object, sometimes by the COM infrastructure.
A COM object can also choose to provide much richer error information by implementing an interface called IErrorInfo. When an object implements IErrorInfo, it can provide all kinds of detail about what happened, such as a detailed custom error message and even the name of a help file that describes the problem. In VB6 and VBA. the Err object allows you to access all that information (Err.Description, etc).
To complicate matters, late bound COM objects (which use a mechanism called COM Automation or IDispatch) add some layers that need to be peeled off to get information out. Excel is usually manipulated via late binding.
Now let's look at your situation again. What you are getting as the first number is a fairly generic error code: DISP_E_EXCEPTION. Note: you can usually figure out the official name of an HRESULT by googling the number, although sometimes you will have to use the hex version to find anything useful.
Errors that begin with DISP_ are IDISPATCH error codes. The error loosely means "There was a COM exception thrown by the object", with more information packed elsewhere (although I don't quite know where; I'll have to look it up).
From what I understand of pywintypes.com_error, the last number in your message is the actual error code that was returned by the object during the exception. It's the actual numeric code that you would get out of VBA's Err.Number.
Unfortunately, that second code -2146788248 (0x800A9C68) is in the range reserved for custom application-defined error messages (in VBA: VbObjectError + someCustomErrorNumber), so there is no centralized meaning. The same number can mean entirely different things for different programs.
In this case, we have reached a dead end:
The error code is "custom", and the application needs to document what it is, except that Excel doesn't. Also, Excel (or the actual source of the error) doesn't seem to be providing any more information via IErrorInfo.
Excel is notorious (to me at least) for cryptic error codes from automation and obscure situations that cause them. This is especially so for errors that one could consider "design-time errors" ("you should have known better than calling a method that doesn't exist in the object"). Instead of a nice "Could not read the Name property", you get "Run-time error '1004': Application defined or object-defined error" (which I just got by trying to access a Name property on a Range, from VBA in Excel). That is NOT very helpful.
The problem is not routed on Python or it's interface to Excel. Excel itself doesn't explain what happened, even to VBA.
However, the general procedure above remains valid. If you get an error from Excel in the future, you might get a better error message that you can track the same way.
Good luck!
Do it like this:
try:
[whatever code]
except pythoncom.com_error as error:
print(win32api.FormatMessage(error.excepinfo[5]))
More information on digesting the pythoncom.com_error object here: https://web.archive.org/web/20170831073447/http://docs.activestate.com/activepython/3.2/pywin32/com_error.html
Yes try the win32api module:
import win32api
e_msg = win32api.FormatMessage(-2147352567)
You can grab any codes returned from the exception and pass them to FormatMessage. Your example had 2 error codes.
Specifically for pythoncom, the errors codes that result are more than cryptic. This is because pythoncom represents them internally as a 32bit signed integer, when the correct representation is a 32bit unsigned integer. As a result, the conversion that you end up seeing in the stack trace is incorrect.
In particular, your exception, according to pythoncom, is -2147352567, and your (for lack of a better word) Err.Number is -2146788248.
This however causes some issues when watching for specific errors, like below:
DISP_E_EXCEPTION = 0x80020009
#...
#except pywintypes.com_error as e:
# print repr(e)
# #pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, None, None, None, 0, -2146788248), None)
# hr = e.hresult
hr = -2147352567
if hr == DISP_E_EXCEPTION:
pass #This never occurs
else:
raise
To see why this has issues, lets look into these error codes:
>>> DISP_E_EXCEPTION = 0x80020009
>>> DISP_E_EXCEPTION
2147614729L
>>> my_hr = -2147352567
>>> my_hr == DISP_E_EXCEPTION
False
Again, this is because python sees the constant declared as positive, and pythoncom's incorrect declaration interpreted it as negative. Of course, the most obvious solution fails:
>>> hex(my_hr)
'-0x7ffdfff7'
The solution is to properly interpret the number. Luckily, pythoncom's representation is reversible. We need to interpret the negative number as a 32 bit signed integer, then interpret that as an unsigned integer:
def fix_com_hresult(hr):
import struct
return struct.unpack("L", struct.pack("l", hr))[0]
>>> DISP_E_EXCEPTION = 0x80020009
>>> my_hr = -2147352567
>>> my_hr == DISP_E_EXCEPTION
False
>>> fixed_hr = fix_com_hresult(my_hr)
>>> fixed_hr
2147614729L
>>> fixed_hr == DISP_E_EXCEPTION
True
So, putting it all together, you need to run fix_com_hresult() on that result from pythoncom, essentially all the time.
Since normally you need to do this when checking for exceptions, I created these functions:
def fix_com_exception(e):
e.hresult = fix_com_hresult(e.hresult)
e.args = [e.hresult] + list(e.args[1:])
return e
def fix_com_hresult(hr):
import struct
return struct.unpack("L", struct.pack("l", hr))[0]
which can then be used how you expect:
DISP_E_EXCEPTION = 0x80020009
try:
#failing call
except pywintypes.com_error as e:
print repr(e)
#pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, None, None, None, 0, -2146788248), None)
fix_com_exception(e)
print repr(e)
#pywintypes.com_error: (2147614729L, 'Exception occurred.', (0, None, None, None, 0, -2146788248), None)
if e.hresult == DISP_E_EXCEPTION:
print "Got expected failure"
else:
raise
I was unable to find a MSDN document listing all HRESULTs, but I found this: http://www.megos.ch/support/doserrors_e.txt
Also, since you have it, fix_com_hresult() should also be run on your extended error code (-2146788248), but as Euro Micelli said, it doesn't help you in this particular instance :)
No-one has yet mentioned the strerror attribute of the pywintypes.com_error Exception. This returns the result of FormatMessage for the error code. So instead of doing it yourself like this
try:
[whatever code]
except pythoncom.com_error as error:
print(win32api.FormatMessage(error.excepinfo[5]))
You can just do this:
try:
[whatever code]
except pythoncom.com_error as error:
print(error.strerror)
Note it will return None if you have a non-standard HRESULT :(