I have a .py file that has a number of functions. Now I am debugging the codes and find that sometimes the program is stuck somewhere.
If I just leave it there and wait for a super long time, error message shows up:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 9, in generate
File "<stdin>", line 7, in choosePath
MemoryError
I have no idea where it is stuck, as I have several while and for loops. Is there any easy way to figure out that easily? I feel really reluctant to debug one loop by another.
Hit CTRL-C and look at the traceback.
For example, the following code will hit a never-ending loop:
import time
def foo():
while True:
time.sleep(1)
def bar():
for i in range(10):
foo()
bar()
and when I interrupt it I see:
$ bin/python endless_demo.py
^CTraceback (most recent call last):
File "test.py", line 11, in <module>
bar()
File "test.py", line 9, in bar
foo()
File "test.py", line 5, in foo
time.sleep(1)
KeyboardInterrupt
The traceback ends in foo, on line 5. This is where Python was busy when I interrupted the program. The traceback also tells me that first bar() was called, which called foo(), so I can see how we got there.
Note that if you have a bare except handler this won't necessarily work; catching all exceptions with try: except: catches KeyboardInterrupt too. Always use except Exception: at a minimum to prevent catching system exceptions.
Related
I have a very simple Python program.
def square(a):
for i in a:
yield i*i
def main():
a = [1, 2, 3]
b = square(a)
print(b)
print(next(b))
print(next(b))
print(next(b))
print(next(b))
if __name__ == "__main__":
main()
When I run it from the command line, I got the following result, which is what I was expecting.
c:\test1>python main.py
<generator object square at 0x000002982E07A4A0>
1
4
9
Traceback (most recent call last):
File "main.py", line 16, in <module>
main()
File "main.py", line 12, in main
print(next(b))
StopIteration
c:\test1>
I run it many times and the result is always the same. However, when I use IntelliJ IDEA with Python plugin to run the same program, I got this:
C:\Python\python.exe c:/test1/main.py
Traceback (most recent call last):
File "c:/my_projects/python1/main.py", line 16, in <module>
main()
File "c:/my_projects/python1/main.py", line 12, in main
print(next(b))
StopIteration
<generator object square at 0x000001799923C580>
1
4
9
Process finished with exit code 1
Or this:
C:\Python\python.exe c:/test1/main.py
<generator object square at 0x00000230F551C580>
1
4
9
Traceback (most recent call last):
File "c:/my_projects/python1/main.py", line 16, in <module>
main()
File "c:/my_projects/python1/main.py", line 12, in main
print(next(b))
StopIteration
Process finished with exit code 1
Why does this happen? How to get a stable result in IntelliJ IDEA?
The Kemp's explanation is correct.
There is a corresponding ticket in PyCharm issue tracker https://youtrack.jetbrains.com/issue/PY-37396
See also https://youtrack.jetbrains.com/issue/PY-32776
As a possible workaround, you can enable Emulate terminal in output console option for your run configuration (Run | Edit Configurations...).
The issue here is that the print output goes to stdout while the exception message goes to stderr. In your terminal they get output as-is, but the IDE is likely doing some buffering on stdout for efficiency reasons. The same buffer won't apply to stderr because it needs to make sure that output gets on the screen asap and without risking the buffer being prematurely cleared out. This means that sometimes (quite often in my experience) exception messages like yours will appear before normal output that you'd expect to already have been shown.
I created a library with decorator. It allows to run seamlessly any function in a separate thread and get timeout exception even that function stuck in TASK_UNINTERRUPTIBLE due to IO to the broken NFS or other nasty problem.
Now I'm working with exception handling. I'm passing exception from the thread back to the caller thread and reraising it again.
Right now trace (if it happens inside decorated function) looks like this:
Traceback (most recent call last):
File "1.py", line 11, in <module>
foo()
File "1.py", line 9, in foo
boo()
File "thread_timeout/__init__.py", line 116, in inner_worker
result = wrapped(*args, **kwargs)
File "1.py", line 6, in boo
file('/fdf','r')
If I remove decorator it looks like this:
Traceback (most recent call last):
File "1.py", line 11, in <module>
foo()
File "1.py", line 9, in foo
boo()
File "1.py", line 6, in boo
file('/fdf','r')
IOError: [Errno 2] No such file or directory: '/fdf'
Question: should I try to hide decorator from the trace or not?
Library code is available here: https://github.com/amarao/thread_timeout/blob/master/thread_timeout/init.py
Thanks.
I'm trying to make an automated test framework for a side-project and could use some help creating the assertion checks.
Running this in python...
assert(False)
Gives you this...
Traceback (most recent call last):
File "test.py", line 1, in <module>
assert(False)
AssertionError
As you can see the traceback lowest level is assert(False). So I made my custom assert that prints when the assert succeeds.
def custom_assert(condition):
if condition:
print("Yay! It werks!")
else:
raise Exception("Nay, it don't werks...")
custom_assert(False)
But instead of what assert gives, custom_assert gives me this.
Traceback (most recent call last):
File "test.py", line 14, in <module>
custom_assert(False)
File "test.py", line 12, in custom_assert
raise Exception("Nay, it don't werks...")
Exception: Nay, it don't werks...
Which is of course the default behavior. Perfectly useful 99.9999% of the time, but this is that one time it could be improved. It's not useful to know that the method I called to raise an error when the condition is false raised the error.
How can I make my custom_assert raise an exception with a traceback starting from the caller, the same way assert does?
P.S.: I don't want to print it, I want the exception to have properly modified traceback so it works properly with debuggers and other tools too!
Edit
To clarify, the traceback I want would be like this.
Traceback (most recent call last):
File "test.py", line 14, in <module>
custom_assert(False)
Exception: Nay, it don't werks...
Essentially what you want to do is something similar to this:
tb = None
try:
raise Exception('foo')
except Exception:
tb = sys.exc_info()[2]
tb.tb_frame = tb.tb_frame.f_back # This line doesn't work
raise Exception('Nay it doesnt werks').with_traceback(tb)
but you can't assign tb_frame, and from mucking around in the CPython code, this is C-generated data structures (not python) (see sys._getframe())
So your only option left is to mock the entire machinery and then convince python to use your stack. This looks like what jinja2 is doing. If that's what you choose to do, good luck! (It's out of my scope at that point)
Suppose I have the following:
def foo():
bar()
def bar():
baz()
def baz():
raise ValueError("hello")
foo()
unsurprisingly, the trace is
$ python x.py
Traceback (most recent call last):
File "x.py", line 10, in <module>
foo()
File "x.py", line 2, in foo
bar()
File "x.py", line 5, in bar
baz()
File "x.py", line 8, in baz
raise ValueError("hello")
ValueError: hello
Now suppose I want the traceback to look like this
Traceback (most recent call last):
File "x.py", line 10, in <module>
foo()
ValueError: hello
That is, I want to be able to remove the topmost two entries in the stack trace, and I want to do it within baz(), that is, at the raising of the exception, either by mangling the stack, or by crafting the raised exception in a special way.
Is it possible?
Edit: one option would be to raise inside baz and catch immediately, then re-raise with a trimmed traceback, but I don't know how to do it.
Edit 2:
what I need is this, in pseudocode
def baz():
raise ValueError, "hello", <traceback built from current frame stack, except the last two>
When I get a runtime exception from the standard library, it's almost always a problem in my code and not in the library code. Is there a way to truncate the exception stack trace so that it doesn't show the guts of the library package?
For example, I would like to get this:
Traceback (most recent call last):
File "./lmd3-mkhead.py", line 71, in <module>
main()
File "./lmd3-mkhead.py", line 66, in main
create()
File "./lmd3-mkhead.py", line 41, in create
headver1[depotFile]=rev
TypeError: Data values must be of type string or None.
and not this:
Traceback (most recent call last):
File "./lmd3-mkhead.py", line 71, in <module>
main()
File "./lmd3-mkhead.py", line 66, in main
create()
File "./lmd3-mkhead.py", line 41, in create
headver1[depotFile]=rev
File "/usr/anim/modsquad/oses/fc11/lib/python2.6/bsddb/__init__.py", line 276, in __setitem__
_DeadlockWrap(wrapF) # self.db[key] = value
File "/usr/anim/modsquad/oses/fc11/lib/python2.6/bsddb/dbutils.py", line 68, in DeadlockWrap
return function(*_args, **_kwargs)
File "/usr/anim/modsquad/oses/fc11/lib/python2.6/bsddb/__init__.py", line 275, in wrapF
self.db[key] = value
TypeError: Data values must be of type string or None.
update: added an answer with the code, thanks to the pointer from Alex.
The traceback module in Python's standard library lets you emit error tracebacks in a way that accords to your liking, while an exception is propagating. You can use this power either in the except leg of a try/except statement, or in a function you've installed as sys.excepthook, which gets called if and when an exception propagates all the way; quoting the docs:
In an interactive session this happens
just before control is returned to the
prompt; in a Python program this
happens just before the program exits.
The handling of such top-level
exceptions can be customized by
assigning another three-argument
function to sys.excepthook.
Here's a simple, artificial example:
>>> import sys
>>> import traceback
>>> def f(n):
... if n<=0: raise ZeroDivisionError
... f(n-1)
...
>>> def excepthook(type, value, tb):
... traceback.print_exception(type, value, tb, 3)
...
>>> sys.excepthook = excepthook
>>> f(8)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in f
File "<stdin>", line 3, in f
ZeroDivisionError
as you see, without needing a try/except, you can easily limit the traceback to (for example) the first three levels -- even though we know by design that there were 9 nested levels when the exception was raised.
You want something more sophisticated than a simple limit on levels, so you'll need to call traceback.format_exception, which gives you a list of lines rather than printing it, then "prune" from that list the lines that are about modules you never want to see in your tracebacks, and finally emit the remaining lines (typically to sys.stderr, but, whatever!-).
Thanks to the pointer from Alex, here's teh codez:
def trimmedexceptions(type, value, tb, pylibdir=None, lev=None):
"""trim system packages from the exception printout"""
if pylibdir is None:
import traceback, distutils.sysconfig
pylibdir = distutils.sysconfig.get_python_lib(1,1)
nlev = trimmedexceptions(type, value, tb, pylibdir, 0)
traceback.print_exception(type, value, tb, nlev)
else:
fn = tb.tb_frame.f_code.co_filename
if tb.tb_next is None or fn.startswith(pylibdir):
return lev
else:
return trimmedexceptions(type, value, tb.tb_next, pylibdir, lev+1)
import sys
sys.excepthook=trimmedexceptions
# --- test code ---
def f1(): f2()
def f2(): f3()
def f3():
import xmlrpclib
proxy = xmlrpclib.ServerProxy('http://nosuchserver')
proxy.f()
f1()
Which yields this stack trace:
Traceback (most recent call last):
File "./tsttraceback.py", line 47, in <module>
f1()
File "./tsttraceback.py", line 40, in f1
def f1(): f2()
File "./tsttraceback.py", line 41, in f2
def f2(): f3()
File "./tsttraceback.py", line 45, in f3
proxy.f()
gaierror: [Errno -2] Name or service not known
The Traceback library is probably what you want. Here's one example that might help:
import traceback
try:
your_main()
except:
lines = traceback.format_exc()
print lines[:lines.find('File "/usr')]
(This obviously won't work if there's an exception outside the library, and might not exactly fit your needs, but it's one way of using the traceback library)
Put an unqualified try...except at the top of your code (ie: in your "main") or set sys.excepthook. You can then format the stack trace however you'd like.