I'm using Twister to build a server. I am also maintaining a server error log. The issue is that if I let an exception run all the way up the stack, it'll crash the current connection and disconnect the user, so obviously I attach a bare except to grab everything else.
Once I've caught something, is there a way to get the traceback as a string so that I can store it somewhere/print it myself without raising it and letting Python print it for me once the program crashes?
The traceback module contains some helper functions for printing and inspecting the traceback (for exameble, traceback.print_tb ) - but the important thing is that the traceback information itself is stored in a "interpreter global" variable - sys.exc_traceback, on the module sys.
Quoting from:
http://docs.python.org/reference/compound_stmts.html#try
Before an except clause’s suite is executed, details about the
exception are assigned to three variables in the sys module:
sys.exc_type receives the object identifying the exception;
sys.exc_value receives the exception’s parameter; sys.exc_traceback
receives a traceback object...
You can pass the sys.exc_traceback object as a parameter to traceback.print_tb to have the traceback printed to stdout within the except clause.
Using the logging module, you could log the traceback to a file:
import logging
logging.basicConfig(level = logging.DEBUG, filename = logfile)
logger = logging.getLogger(__name__)
try:
1/0
except ZeroDivisionError as err:
logger.exception(err)
Try this:
import traceback, sys
try:
# Do something that might raise an exception
open("/does not exist",'rb')
except:
traceback.print_exc( file=sys.stderr )
# Or
traceback.print_exc( file=your_open_log_file )
That should do the trick and print full stack traces too.
Yep, there's a module for that. The traceback module contains functions to print or format exception information, or to return the raw stack frames so you can do whatever you want with them.
For a reasonably sophisticated application, though, I would actually recommend using the logging system instead of plain old traceback functions. In particular the method logging.Logger.exception will write information about the exception to whatever logging destination you (or your software's user) has configured. The default formatter will just print out a traceback, as Python would on the console, but you can customize the display of exceptions in your log by creating a Formatter and overriding the format_exception method. Your overridden method is the place to call traceback functions if you need them to format the exception output.
Related
I'm trying to make a debugger function, which is called when an error is raised, and let me access a console so I can check what happened in my program.
Here is the basic function:
def DEBUGGER(error):
print(error)
print("[DEBUGGER] Your program has failed, here is the debugger. Enter EXIT to end program.")
while True:
line = input(">>> ").lower()
if line == 'exit':
sys.exit(0)
else:
try:
exec(line)
except Exception as e:
print(str(e))
The problem is that I can't enter something like print(var) because it's referenced in another function.
Globals functions don't help me since I want to be able to call any variable in my program, and I can't globalize them all.. I know I can resolve it by putting all my functions in classes but I can't for many reasons.
Is there a way to get local variables of the running functions ? (When I call DEBUGGER(), the mother function is still running)
If no, can I export the local variables of the current function and pass it as an argument to DEBUGGER() ?
Thanks for your answers.
You are basically re-implementing the Python debugger pdb. If you want to go this route, you probably want to study the source code. pdb itself is a user-interface around the lower-level bdb (basic debugger) module, and the source code for that is also available.
To answer your direct question: when you catch an exception you have access to a traceback object (either via exception.__traceback__ or via sys.exc_info()), and tracebacks have access to both the local and global namespace of each frame in the stack, via the tb_frame attribute. That attribute is set to a frame object, which has f_locals and f_globals attributes.
The bdb.Bdb.get_stack() method could be an interesting example on how to treat a traceback, and the internal pdb.Pdb._select_frame() method then is used to pick a frame from the stack to use the locals and globals from.
If you don't want to re-implement the full debugger, you can use the pdb.pm() or pdb.port_mortem() functions. These take the last traceback raised and let you inspect the stack frame in an interactive environment:
try:
exec(line)
except Exception as e:
pdb.post_mortem(e.__traceback__)
The correct way to "write" your "DEBUGGER" function is:
import pdb
DEBUGGER = pdb.set_trace
Now you can call DEBUGGER() wherever you want, you will be in an interactive environment with access not only to local vars but also to whole call stack, and the ability to execute the remaining code step by step (including stepping into other functions etc), change the control flow to continue executing from another line etc etc etc.
Oh and yes: you can of course just write import pdb; pdb.set_trace() instead ;-)
TL;DR: The Standard Library fails to close a file when an exception is raised. I'm looking for the best way to handle this situation. Feel free to read from the paragraph beginning with "Upon closer inspection of CPython's source code". Also scroll down to the end of the question to grab a self-contained script that reproduces this issue on Windows.
I'm writing a Python package in which I use STL's ConfigParser (2.x) or configparser (3.x) to parse user config file (I will refer to both as ConfigParser since the problem mainly lies in the 2.x implementation). From now on my relevant lines of code on GitHub will be linked when appropriate throughout. ConfigParser.ConfigParser.read(filenames) (used in my code here) raises a ConfigParser.Error exception when the config file is malformed. I have some code in my test suite targeting this situation, using unittest.TestCase.assertRaises(ConfigParser.Error). The malformed config file is properly generated with tempfile.mkstemp (the returned fd is closed with os.close before anything) and I attempt to remove the temp file with os.remove.
The os.remove is where trouble begins. My tests fail on Windows (while working on both OS X and Ubuntu) with Python 2.7 (see this AppVeyor build):
Traceback (most recent call last):
File "C:\projects\storyboard\tests\test_util.py", line 235, in test_option_reader
os.remove(malformed_conf_file)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\appveyor\\appdata\\local\\temp\\1\\storyboard-test-3clysg.conf'
Note that as I said above, malformed_conf_file is generated with tempfile.mkstemp and immediately closed with os.close, so the only time it is opened is when I call ConfigParser.ConfigParser.read([malformed_conf_file]) here inside the unittest.TestCase.assertRaises(ConfigParser.Error) context. So the culprit seems to be the STL rather than my own code.
Upon closer inspection of CPython's source code, I found that ConfigParser.ConfigPaser.read indeed doesn't close the file properly when an exception is raised. The read method from 2.7 (here on CPython's Mercurial) has the following lines:
for filename in filenames:
try:
fp = open(filename)
except IOError:
continue
self._read(fp, filename)
fp.close()
read_ok.append(filename)
The exception (if any) is raised by self._read(fp, filename), but as you can see, if self._read raises, then fp won't be closed, since fp.close() is only called after self._read returns.
Meanwhile, the read method from 3.4 (here) does not suffer from the same problem since this time they properly embedded file handling in a context:
for filename in filenames:
try:
with open(filename, encoding=encoding) as fp:
self._read(fp, filename)
except OSError:
continue
read_ok.append(filename)
So I think it's pretty clear that the problem is a defect in 2.7's STL. And what's the best way to handle this situation? Specifically:
Is there anything I can do from my side to close that file?
Is it worth reporting to bugs.python.org?
For now I guess I'll just add a try .. except OSError .. to that os.remove (any suggestions?).
Update: A self-contained script that can be used to reproduce this issue on Windows:
#!/usr/bin/env python2.7
import ConfigParser
import os
import tempfile
def main():
fd, path = tempfile.mkstemp()
os.close(fd)
with open(path, 'w') as f:
f.write("malformed\n")
config = ConfigParser.ConfigParser()
try:
config.read(path)
except ConfigParser.Error:
pass
os.remove(path)
if __name__ == '__main__':
main()
When I run it with Python 2.7 interpreter:
Traceback (most recent call last):
File ".\example.py", line 19, in <module>
main()
File ".\example.py", line 16, in main
os.remove(path)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\redacted\\appdata\\local\\temp\\tmp07yq2v'
This is an interesting problem. As Lukas Graf noted in a comment, the problem seems to be that the exception traceback object holds a reference to the call frame where the exception was raised. This call frame includes the local variables that existed at that time, one of which is a reference to the open file. So that file object still has a reference to it and is not properly closed.
For your self-contained example, simply removing the try/except ConfigParser.Error "works": the exception about the malformed config file is uncaught and stops the program. However, in your actual application, assertRaises is catching the exception in order to check that it is the one you're testing for. I'm not 100% sure why the traceback persists even after the with assertRaises block, but apparently it does.
For your example, another more promising fix is to change the pass in your except clause to sys.exc_clear():
try:
config.read(path)
except ConfigParser.Error:
sys.exc_clear()
This will get rid of the pesky traceback object and allow the file to be closed.
However, it's not clear exactly how to do that in your real application, because the offending except clause there is inside unittest. I think the easiest thing might be to not use assertRaises directly. Instead, write a helper function that does your test, checks for the exception you want, cleans up with the sys.exc_clear() trick, and then raises another custom exception. Then wrap a call to that helper method in assertRaises. This way you gain control of the problematic exception raised by ConfigParser and can properly clean it up (which unittest isn't doing).
Here's a sketch of what I mean:
# in your test method
assertRaises(CleanedUpConfigError, helperMethod, conf_file, malformed_conf_file)
# helper method that you add to your test case class
def helperMethod(self, conf_file, malformed_conf_file):
gotRightError = False
try:
or6 = OptionReader(
config_files=[conf_file, malformed_conf_file],
section='sec',
)
except ConfigParser.Error:
gotRightError = True
sys.exc_clear()
if gotRightError:
raise CleanedUpConfigError("Config error was raised and cleaned up!")
I haven't actually tested this, of course, because I don't have the whole unittest thing set up with your code. You might have to tweak it a bit. (Come to think of it, you might not even need exc_clear() if you do this, because since the exception handler is now in a separate function, the traceback should be correctly cleared when helperMethod exits.) However, I think this idea may get you somewhere. Basically you need to make sure the except clause that catches this particular ConfigParser.Error is written by you, so that it can be cleaned up before you try to remove your test file.
Addendum: it seems that if the context manager handles an exception, the traceback is actually stored until the function containing the with block ends, as this example shows:
class CM(object):
def __init__(self):
pass
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, tb):
return True
def foo():
with CM():
raise ValueError
print(sys.exc_info())
Even though the with block has ended by the time the print occurs, so the exception handling should be finished, sys.exc_info still returns exception info as if there were an active exception. This is what is happening in your code too: the with assertRaises block causes the traceback to persist all the way to the end of that function, interfering with your os.remove. This seems like buggy behavior, and I notice that it no longer works this way in Python 3 (the print prints (None, None None)), so I imagine this was a wart that was fixed with Python 3.
Based on this, I suspect it may be sufficient to just insert a sys.exc_clear() right before your os.remove (after the with assertRaises block).
I embed Python into my GUI application, when the script throws syntax errors, I want to redirect the error message into a textbox, so I can know where the error is.
My codes is very simple:
Py_Initialize()
PyRun_SimpleString( "execfile('my.py')" );
Py_Finalize();
If the file contains syntax error, it may looks like:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "my.py", line 3, in <module>
app=everedit.Ap
AttributeError: 'module' object has no attribute 'Ap'
The above messages can be seen in a console window, but now I want to cache these messages in a GUI window.
A GUI window don't have a console. I want to output such messages into a text box.
Please note that I embed Python into C++, I want to cache Python's Syntax Error in C++.
As the documentation for PyRun_SimpleString clearly says:
Returns 0 on success or -1 if an exception was raised. If there was an error, there is no way to get the exception information.
So, if you want to get the exception information, you have to use slightly lower-level functions.
Meanwhile, once you're using the right function, and it returns NULL or -1 to tell you an exception occurred, how do you get the exception information (and distinguish SyntaxError from other exceptions, for whatever reason you want to do that)?
There's a whole chapter in the C-API docs on Exception Handling. But briefly: You call PyErr_Occurred to get the type of the exception. Then you call PyErr_ExceptionMatches to check whether it's the type you want. If so, use PyErr_Fetch to get the value and traceback so you can format them yourself, or PyErr_Format to get a simple formatted string, or whatever else you want. Then you just extract the Unicode or ASCII bytes from the string and put them in your GUI window.
Use a try/catch block and the traceback module (docs).
import traceback
try:
PyParser_SimpleParseString(input())
except SyntaxError as e:
# There will always be one line, but we're better off still looping over the list
# To be extra safe!
for line in traceback.format_exception_only(e.__class__, str(e)):
print(line) # Replace this with what write to the GUI
This will only give you the SyntaxError: x part, if you also want the traceback, use traceback.format_exception.
I am running an IronPython script inside a c# application, i am catching exceptions within the script and i wish to find out the script line at which the exception is thrown. This has to be done while the script is running ie. i do not wish the script to terminate in order to print the exception.
Is this even possible?
If inspect is working as expected under IronPython (not really sure) this could do the trick:
import inspect
filename, linenum, funcname = inspect.getframeinfo(inspect.currentframe())[:3]
print linenum
Edit: alternate solution:
import sys
frame = sys._getframe()
print frame.f_lineno
Haven't tried this on ironpython but:
import traceback
try:
# something that raises exception
except YourException, _:
traceback.print_exc()
This should show you the stack trace of the place where the exception was raised. You can also do other stuff than just print, like print to string, or get the stack frames.
The traceback module documentation will tell you more.
When an exception occurs in Python, can you inspect the stack? Can you determine its depth? I've looked at the traceback module, but I can't figure out how to use it.
My goal is to catch any exceptions that occur during the parsing of an eval expression, without catching exceptions thrown by any functions it may have called. Don't berate me for using eval. It wasn't my decision.
NOTE: I want to do this programmatically, not interactively.
traceback is enough - and I suppose that documentation describes it rather well. Simplified example:
import sys
import traceback
try:
eval('a')
except NameError:
traceback.print_exc(file=sys.stdout)
You can use the inspect module which has some utility functions for tracing. Have a look at the overview of properties of the frame objects.
I like the traceback module.
You can get a traceback object using sys.exc_info(). Then you can use that object to get a list preprocessed list of traceback entries using traceback.extract_tb(). Then you can get a readable list using traceback.format_list() as follows:
import sys
import traceback, inspect
try:
f = open("nonExistant file",'r')
except:
(exc_type, exc_value, exc_traceback) = sys.exc_info()
#print exception type
print exc_type
tb_list = traceback.extract_tb(sys.exc_info()[2])
tb_list = traceback.format_list(tb_list)
for elt in tb_list:
print elt
#Do any processing you need here.
See the sys Module: http://docs.python.org/library/sys.html
and the traceback Module: http://docs.python.org/library/traceback.html
You define such a function (doc here):
def raiseErr():
for f in inspect.stack(): print '-', inspect.getframeinfo(f[0])
and call it from your modules so:
raiseErr()
The function raiseErr will print info about the place you called it.
More elaborate, you can do so:
import inspect, traceback
A = [inspect.getframeinfo(f[0]) for f in inspect.stack()]
print "traceback structure fields:", filter(lambda s: s[0] != '_', dir(A[0]))
print A[0].filename, A[0].lineno
for f in inspect.stack():
F = inspect.getframeinfo(f[0])
print '-', F.filename, F.lineno, '\t', F.code_context[0].strip()
Other possibility is to define this function:
def tr():
print '* - '*10,
print sys._getframe(1).f_code.co_name
And call it in the place where you want the trace. If you want all the trace, make an iterator from 1 up in _getframe(1).
In addition to AndiDog's answer about inspect, note that pdb lets you navigate up and down the stack, inspecting locals and such things. The source in the standard library pdb.py could be helpful to you in learning how to do such things.