Imagine I am debugging the following script:
import ipdb
def slow_function(something):
# I'm a very slow function
return something_else
def fast_function(something_else):
# There's a bug here
return final_output
something = 1
something_else = slow_function(something)
ipdb.set_trace()
final_output = fast_function(something_else)
ipdb.set_trace()
When the ipdb.set_trace() line is met the debugger shell is prompted and I can now execute the final_output = fast_function(something_else) statement to check whether fast_function is behaving as intended. I see there's a bug so I go into the source code and I fix it. Now I want to see whether the fix is correct but I don't want to run the script a second time (because it's slow), nor I want to save something_else on disk (because, maybe, it's very large).
Is there a way to update fast_function() in the debugger shell so that the new source code is used?
You can execute a single-line python statement inside pdb by preceding it with an exclamation sign. The output of help exec as produced by pdb follows:
(!) statement
Execute the (one-line) statement in the context of the current
stack frame. The exclamation point can be omitted unless the
first word of the statement resembles a debugger command. To
assign to a global variable you must always prefix the command
with a 'global' command, e.g.:
(Pdb) global list_options; list_options = ['-l']
(Pdb)
Using this facility, you can save the source code of the function in a file and update that function inside pdb as follows:
!exec(open("fast_function.py", "r").read())
Demonstration:
$ cat test.py
import pdb;
def foo():
print('Foo');
foo()
pdb.set_trace()
foo()
$ python3 test.py
Foo
> test.py(8)<module>()
-> foo()
(Pdb) cont
Foo
$ cat foo.py
def foo():
print('Modified Foo');
$ python3 test.py
Foo
> test.py(8)<module>()
-> foo()
(Pdb) !exec(open("foo.py", "r").read())
(Pdb) cont
Modified Foo
If it's a short function you can just overwrite the existing function in one pdb line. Note that the exclamation sign is optional there.
(pdb)...
(pdb)!def fast_function(something_else): print("Hello world");return True
If the function is a bit larger in code length then you can leverage the normal interactive shell(see this)
(Pdb) !import code; code.interact(local=vars())
(InteractiveConsole)
In : def fast_function(something_else):
...: print 'hello in pdb'
...:
In : # use ctrl+d here to return to pdb shell...
(Pdb) !fast_function(arg)
hello in pdb
If the code of the function is not easily manageable with the interactive shell either, Leons' recommendation will be better I guess.
Just keep in mind that, every pdb line can work like executing a line of normal python code and you can achieve pretty much anything on the fly! pdb is more powerful than some other graphical debugging tools in this sense.
P.S. seems like PyCharm does support Evaluating expression feature according to ADR
edit 19/01/2018
You can write result in a file on memory. for example /dev/shm is a tmpfs partition. size could be optimized with protocol dump's keyargs.
# save result
with open('/dev/shm/data.pk', 'w' ) as data:
pickle.dump(something_else, data, protocole=3)
You can use pickle to store result in file the first time and reload it to debug second function
import pickle
def slow_function(something):
# I'm a very slow function
return something + 42
def fast_function(something_else):
# There's a bug here
return something_else + 42
something = 1
something_else = slow_function(something)
# save result
with open('data.pk', 'w' ) as data:
pickle.dump(something_else, data)
second launch
import ipdb
# load result from disk
with open('data.pk', 'r' ) as data:
something_else = pickle.load(data)
ipdb.set_trace()
fast_function(something_else)
Related
Is it possible to intercept interpreter's code before it executes?
Let's say I want to handle a case like:
>>> for f in glob.glob('*.*'): # I'd like to intercept this code before it executes
... something_to_do(f) # and play with it in some dangerous fashion :)
...
ERROR: Using glob not allowed. # e.g.
But there are tons of other examples (like altering the code, or sending it somewhere).
I can write my own interpreter, but that's not really the point.
Ok, solved it by creating new module which starts new interpreter instance and do whatever.
I just put the code below in the module and import it.
import code
class GlobeFilterConsole(code.InteractiveConsole):
def push(self, line):
self.buffer.append(line)
source = "\n".join(self.buffer)
if 'glob' in source: # do whatever you want with the source
print('glob usage not allowed')
more = False
else:
more = self.runsource(source, self.filename)
if not more:
self.resetbuffer()
return more
console = GlobeFilterConsole()
console.interact()
I sometimes run into the following problem. I have a function, which returns something I am interested in and prints something I do not care about. E.g.
def f(x):
print('Some complicated printing stuff')
important_result = 42
return important_result
I want to write a doctest, which checks, that it indeed returns the correct result. But whose code is not obfuscated by the complicated printing stuff.
Something along the following lines would be cool:
def f(x):
"""
>>> f(0)
...
42
"""
print('Some complicated printing stuff')
important_result = 42
return important_result
Is there an elegant way to acomplish this?
Given that, your question has tag 'doctest' - I am assuming you want to run doctest for your function (enlighten me in the comment section, for any presumption or trivia) - as the text is ambiguous.
doctest basically works by looking for text in the form of Python interactive session. This text can be written in the docstring (like you have in the second code block/sample) or in a separate file.
using docstrings - all you have to do is specify few (or at-least one) examples of function i.e. passing the required parameters and the expected result, exactly in the same format as of Python interactive session. A good practice is to run your function in the interactive session and copy paste it in the docstring of the function.
To run the doctest, first specify the following code (for your mentioned code example, you just have to type the exact lines):
if __name__ = "__main__":
import doctest
doctest.testmod()
Line 1 - run the following code, only if the module (your *.py file)
is run as script i.e. not imported, etc. (a more detailed answer here)
Line 2 - importing the doctest module.
Line 3 - looks for any interactive session style text in the
docstring and runs it.
Once you have the above code included in your module (*.py file), just run in as a script:
python yourmodule.py
OR,
You can run the doctest directly (without the above 3 lines of code):
pyhton -m doctest yourmodule.py
using a separate file - add the following line in your file:
if __name__ = "__main__"
import doctest
doctest.testfile("somefile.txt")
it will identify and execute any interactive Python text in the file. By default the testfile() will look for 'somefile.txt' in the same directory where the module (.py file) is located (using options we can look for files in other locations).
Coming back to your question
I want to write a doctest, which checks, that it indeed returns the
correct result. But whose code is not obfuscated by the complicated
printing stuff. Something along the following lines would be cool:
NO (directly not possible) - Scenarios for doctest are set/written by specifying examples in form of Python interactive sessions in your docstring - exactly the i.e. as mentioned above a good practice is to run your function with various inputs in the interactive session and copy paste those lines in the docstring - and all the print statements will definitely be there, and part of the out for doctest to mark test as passed.
Indirect ways:
use an optional parameter in function, for example printing - the overhead 'd be you'll need to make changes to the function i.e. moving the print statements under if.
def f(x, printing=True):
"""
f(0, printing=False)
...
42
"""
if printing:
print('Some complicated printing stuff')
important_result = 42
return important_result
pass the None argument to print function - beware, it'll be passed to all the print calls in the module i.e. the whole *.py file.
def f(x):
"""
f(0)
...
42
"""
print('Some complicated printing stuff')
important_result = 42
return important_result
if name == 'main':
import doctest
print = lambda *args, **kwargs: None
doctest.testmod()
Source: jonrsharpe's answer
To get some feedback, I ended up printing the result of doctest.testmod()
I have a program like this:
if __name__=="__main__":
foo = expensiveDataProcessClass(filepath)
y = foo.doStuff()
y = foo.doOtherStuff()
I'm testing things out as a I build it in ipython with the %run myprogram command.
After it's running, since it takes forever, I'll break it with ctrl+C and go rewrite some stuff in the file.
Even after I break it, though, IPython has foo stored.
>type(foo)
__main__.expensiveDataProcessClass
I'm never having to edit anything in foo, so it would be cool if I could update my program to first check for the existence of this foo variable and just continue to use it in IPython rather than doing the whole creation process again.
You could first check for the variable's existence, and only assign to it if it doesn't exist. Example:
if __name__=="__main__":
if not "foo" in globals()
foo = expensiveDataProcessClass(filepath)
However, this won't actually work (in the sense of saving a foo assignment). If you read IPython's doc on the %run magic, it clearly states that the executed program is run in its own namespace, and only after program execution are its globals loaded into IPython's interactive namespace. Every time you use %run it will always not have foo defined from the program's prospective.
Sounds like I'm missing something extremely simple, I'm trying to set a breakpoint in my python code using:
if(some condition):
pdb.set_trace()
My error in the code comes after a large number of iterations..difficult to debug using print etc.
I am able to print stuff when the condition hits but I would like to set brk-pt.
--EDIT--
Actual code:
import pdb
if (node_num == 16):
print node_num
pdb.set_trace()
I see you found your solution Sanjay. But for those who arrived here looking for a means to set a conditional breakpoint with pdb read on:
Instead of hard coding conditions such as if node_num == 16:, run pdb in interactive mode. Sample code:
import pdb
for node_num in range(50):
do_something(node_num)
...
In the shell start the script in debug mode using -m pdb:
[rick#rolled ~]$ python -m pdb abc.py
> /home/dcadm/abc.py(1)<module>()
-> import pdb
(Pdb) l
1 -> import pdb
2
3 for node_num in range(50) :
4 foo = 2**node_num
[EOF]
(Pdb) b 4, node_num > 4
Breakpoint 1 at /home/dcadm/abc.py:4
(Pdb) c
> /home/dcadm/abc.py(4)<module>()
-> foo = 2**node_num
(Pdb) node_num
5
(Pdb)
The pdb shell command b 4, node_num > 4 breaks at line 4 when node_num is greater than 4.
To actually use conditional breakpoints in pdb, you can do the following:
b(reak) [([filename:]lineno | function) [, condition]]
https://docs.python.org/3/library/pdb.html#pdbcommand-break
Eg I'm running some test code that iterates over django views. I want to break only when the particular view I'm interested in is reached:
b C:\Users\powlo\project\tests\TestCase.py:350, view.view_name
== 'app.views.export'
I am not sure as to why your code isn't working, but what you can do is on your local machine, create a new file for your minimum example to see if you can do what you want to do
import pdb
for node_num in range(50):
if node_num == 16:
print(node_num)
pdb.set_trace()
Now running it:
16
> /tmp/tmp.py(3)<module>()
-> for node_num in range(50):
(Pdb) p node_num
16
As you can see this worked as intended with this trivial example, it's up to you to figure out how to adapt this to your code, and/or figure out what else did you do to your code/environment that prevented that prompt from showing up.
Alternatively, if you have a function that is dying in an exception and you want to know the exact line that caused it, you should use post_mortem instead. Wrap the problematic section of code with this
try:
problem_function()
except Exception: # or the specific exception type thrown
pdb.post_mortem()
raise
What post_mortem would do is dump a break point right at the point where the exception happened (specifically in that stack frame), and so this allows all values to be inspected and then let you continue execution. However I also put a raise at the end to allow the exception to continue as normal, and this is intended as execution doesn't normally from where it die but just pause at that exception handling block due to the post_mortem call. Might as well just give up after inspecting what went wrong.
I have a MOTD-type message which prints on invocation of the interpreter. Currently this is printed up in sitecustomize. I'd like to suppress the message if the interpreter is not in interactive mode; unfortunately all of the checks in
Tell if Python is in interactive mode do not work in sitecustomize. (sys.argv, sys.ps1, __main__.__file__ are not populated.) Are there checks which work in sitecustomize?
JAB got me looking at the code and I ultimately came up with this:
import ctypes
import getopt
ctypes.pythonapi.Py_GetArgcArgv.restype = None
ctypes.pythonapi.Py_GetArgcArgv.argtypes = [
ctypes.POINTER(ctypes.c_int),
ctypes.POINTER(ctypes.POINTER(ctypes.c_char_p))]
count = ctypes.c_int()
args = ctypes.pointer(ctypes.c_char_p())
ctypes.pythonapi.Py_GetArgcArgv(ctypes.byref(count), ctypes.byref(args))
argc = count.value
argv = [args[i] for i in range(count.value)]
if argc > 1:
interactive = False
opts, args = getopt.getopt(argv[1:], 'i')
for o, a in opts:
if o == '-i':
interactive = True
else:
interactive = True
Kinda ugly (and for Py3k the c_char_p need to be c_wchar_p) but does the job.
Perhaps this idea for checking interpreter interactivity that utilizes the inspect module and checks stack frames might be of some use to you:
http://mail.python.org/pipermail/pythonmac-sig/2002-February/005054.html
You could also try looking directly at the source of pydoc.help(), which the above-linked code snippets were inspired by.
Just realized that you could simply utilize a file containing your interactive prompt with the PYTHONSTARTUP environment variable. The commands in the file pointed to by PYTHONSTARTUP will only be executed when the interpreter is run interactively.
http://docs.python.org/tutorial/interpreter.html#the-interactive-startup-file
If you don't want to set the environment variable outside of Python, you might be able to set the variable to the desired file in sitecustomize.py, but when I tried looking into it to find the loading order it took me right back to the link from the first part of my answer.
Checking the sys.flags is a cleaner way.
>>> import sys
>>> sys.flags.interactive
1
Note, the IDLE is also interactive in its nature, but the flag is not set. I would do below:
>>> if sys.flags.interactive or sys.modules.has_key('idlelib'):
>>> pass # do stuff specific to interactive.