We Keep Coding

How to debug a function that has been passed as an argument to another function

I'm new to Python and used to debugging in R. I'm trying to debug my_function when it has been passed as an argument to another function
df.apply(my_function, axis=1)
When I use the debugger (I'm in Spyder V5) I can jump down into the apply function ("down" or "crtl + f11"), but I can't seem to navigate into my_function when it is called from being passed as an argument.
In R I would add browser() to my_function and go from there, but when I try adding ipdb.set_trace(), which I assume is the equivalent, my console hangs without any option to supply input.
I need to examine it in this context because it is breaking when passed as an arg to a specific function, but not others (it actually works in the example function "pd.apply").

You could add breakpoints in Spyder by double-clicking the line number. Then you could run the code using the Debug option or Ctrl+F5.
In order to move to the next break point: Continue or Ctrl+F12.
To move to the next line: Step or Ctrl+F10
To move along the function execution: Step Into or Ctrl+F11

how to disable pdb.set_trace() without stopping python program and edit the code

I suspect that I have issue in one of my loops, so I setup a break points with pdb.set_trace()
import pdb
for i in range(100):
print("a")
pdb.set_trace()
print("b")
after check variable in this loop for a few times, I decide continue this programming without further breaks. So I try to get the break number with b command, no breaks listed. I guess this line of code don't setup a break point. but How Do I get ride of this "break points" without stopping the program and change the code?

to my knowledge, you could not bypass set_trace, but you could neutralize it, once debugger stopped, type:
pdb.set_trace = lambda: 1
then continue, it wont break again.

Setting a breakpoint (requires Python 3.7):
breakpoint()
Disabling breakpoints set with the breakpoint() function:
import os
os.environ["PYTHONBREAKPOINT"] = "0"
Long story:
In the 3.7 version of Python, the breakpoint() built-in function for setting breakpoints was introduced. By default, it will call pdb.set_trace(). Also, since the 3.7 version of Python the PYTHONBREAKPOINT environment variable is available. It is considered when the breakpoint() function is used.
So, in order to disable these breakpoints (set with the breakpoint() function), one can just set the PYTHONBREAKPOINT environment variable like this:
import os
os.environ["PYTHONBREAKPOINT"] = "0"
It may be useful to mention here sys.breakpointhook() which was also added in the 3.7 version of Python and allows to customize breakpoints behavior.

Unfortunately pdb is missing a bunch of functionality (even basic stuff like display lists), and you've found another example of that here. The good news is that pdb++ is a great drop-in replacement for pdb, and one of the things it solves is exactly the problem of disabling set_trace. So you can simply do:
pip install pdbpp
and then at the (Pdb++) prompt, type
pdb.disable()
Easy! And you will get lots of other useful goodies on top of that.

It is possible to start a Python script without PDB control but then hit a stray set_trace() left there. To prevent breaking into debugger every time set_trace() is encountered, a similar trick as above (changing the symbol's reference to point to a harmless function) can be applied.
However, the namespace of the debuggee has to be modified, not one of the debugger itself. Simply overwriting pdb.set_trace = lambda:1 or set_trace = lambda:1 did not work for me.
The following trick worked from the pdb prompt:
pdb> globals()['set_trace'] = lambda:1
This line first calls globals() to get access to a dict of the program under debugging, and then modifies the reference of set_trace there.

One way around this is to not write the breakpoints in the script itself, but rather set breakpoints when you start python with python -m pdb my_script.py
You then get into a prompt first, before execution of the script starts, and you can write for example
(Pdb) b 321
to set a breakpoint on line 321 (you can also specify file and specify conditions b 321, i == 50)
Then
(Pdb) c
(for continue) to start the execution of the actual script. The breakpoints you set in this way, you can clear when you're done with them with:
(Pdb) cl
(for clear)

Is it possible to modify python code while debugging? with Pydev, PDB, etc

Let's say I am stepping through code with Eclipse's PyDev.
Is there a way to add some new code below the breakpoint... modify the code on the fly / at runtime?

I found interesting project on github: pyrasite
Inject code into running Python processes http://pyrasite.com

In PDB, you can use p to evaluate and print expressions and ! to execute atements:
p expression
Evaluate the expression in the current context and print its value.
[!]statement
Execute the (one-line) statement in the context of the current stack frame. The exclamation point can be omitted unless the first word of the statement resembles a debugger command. To set a global variable, you can prefix the assignment command with a global command on the same line, e.g.:
(Pdb) global list_options; list_options = ['-l']
(Pdb)
Docs: https://docs.python.org/2/library/pdb.html
As for how to do this in PyDev, I don't use PyDev so I'm not quite sure; but this functionality should be available to any BDB-based debuggers (which I believe PyDev is also based on BDB?).

Can I put a breakpoint in a running Python program that drops to the interactive terminal?

I'm not sure if what I'm asking is possible at all, but since python is an interpreter it might be. I'm trying to make changes in an open-source project but because there are no types in python it's difficult to know what the variables have as data and what they do. You can't just look up the documentation on the var's type since you can't be sure what type it is. I want to drop to the terminal so I can quickly examine the types of the variables and what they do by typing help(var) or print(var). I could do this by changing the code and then re-running the program each time but that would be much slower.
Let's say I have a program:
def foo():
a = 5
my_debug_shell()
print a
foo()
my_debug_shell is the function I'm asking about. It would drop me to the '>>>' shell of the python interpreter where I can type help(a), and it would tell me that a is an integer. Then I type 'a=7', and some 'continue' command, and the program goes on to print 7, not 5, because I changed it.

http://docs.python.org/library/pdb.html
import pdb
pdb.set_trace()

Here is a solution that doesn't require code changes:
python -m pdb prog.py <prog_args>
(pdb) b 3
Breakpoint 1 at prog.py:3
(pdb) c
...
(pdb) p a
5
(pdb) a=7
(pdb) ...
In short:
start your program under debugger control
set a break point at a given line of code
let the program run up to that point
you get an interactive prompt that let's you do what you want (type 'help' for all options)

Python 3.7 has a new builtin way of setting breakpoints.
breakpoint()
The implementation of breakpoint() will import pdb and call pdb.set_trace().
Remember to include the braces (), since breakpoint is a function, not a keyword.

A one-line partial solution is simply to put 1/0 where you want the breakpoint: this will raise an exception, which will be caught by the debugger. Two advantages of this approach are:
Breakpoints set this way are robust against code modification (no dependence on a particular line number);
One does not need to import pdb in every program to be debugged; one can instead directly insert "breakpoints" where needed.
In order to catch the exception automatically, you can simply do python -m pdb prog.py… and then type c(ontinue) in order to start the program. When the 1/0 is reached, the program exits, but variables can be inspected as usual with the pdb debugger (p my_var). Now, this does not allow you to fix things and keep running the program. Instead you can try to fix the bug and run the program again.
If you want to use the powerful IPython shell, ipython -pdb prog.py… does the same thing, but leads to IPython's better debugger interface. Alternatively, you can do everything from within the IPython shell:
In IPython, set up the "debug on exception" mode of IPython (%pdb).
Run the program from IPython with %run prog.py…. When an exception occurs, the debugger is automatically activated and you can inspect variables, etc.
The advantage of this latter approach is that (1) the IPython shell is almost a must; and (2) once it is installed, debugging can easily be done through it (instead of directly through the pdb module). The full documentation is available on the IPython pages.

You can run the program using pdb, and add breakpoints before starting execution.
In reality though, it's usually just as fast to edit the code and put in the set_trace() call, as another user stated.

Not sure what the real question is. Python gives you the 'pdb' debugger (google yourself) and in addition you can add logging and debug output as needed.

Jump into a Python Interactive Session mid-program?

Hey I was wondering... I am using the pydev with eclipse and I'm really enjoying the powerful debugging features, but I was wondering:
Is it possible to set a breakpoint in eclipse and jump into the interactive python interpreter during execution?
I think that would be pretty handy ;)
edit: I want to emphasize that my goal is not to jump into a debugger. pydev/eclipse have a great debugger, and I can just look at the traceback and set break points.
What I want is to execute a script and jump into an interactive python interpreter during execution so I can do things like...
poke around
check the values of things
manipulate variables
figure out some code before I add it to the app
I know you can do this all with a debugger, but I can do it faster in the interactive interpreter because I can try something, see that it didn't work, and try something else without having get the app back to the point of executing that code again.

So roughly a year on from the OP's question, PyDev has this capability built in. I am not sure when this feature was introduced, but all I know is I've spent the last ~2hrs Googling... configuring iPython and whatever (which was looking like it would have done the job), but only to realise Eclipse/PyDev has what I want ootb.
As soon as you hit a breakpoint in debug mode, the console is right there ready and waiting!
I only didn't notice this as there is no prompt or blinking cursor; I had wrongly assumed it was a standard, output-only, console... but it's not. It even has code-completion.
Great stuff, see http://pydev.org/manual_adv_debug_console.html for more details.

This is from an old project, and I didn't write it, but it does something similar to what you want using ipython.
'''Start an IPython shell (for debugging) with current environment.
Runs Call db() to start a shell, e.g.
def foo(bar):
for x in bar:
if baz(x):
import ipydb; ipydb.db() # <-- start IPython here, with current value of x (ipydb is the name of this module).
.
'''
import inspect,IPython
def db():
'''Start IPython shell with callers environment.'''
# find callers
__up_frame = inspect.currentframe().f_back
eval('IPython.Shell.IPShellEmbed([])()', # Empty list arg is
# ipythons argv later args to dict take precedence, so
# f_globals() shadows globals(). Need globals() for IPython
# module.
dict(globals().items() + __up_frame.f_globals.items()),
__up_frame.f_locals)
edit by Jim Robert (question owner): If you place the above code into a file called my_debug.py for the sake of this example. Then place that file in your python path, and you can insert the following lines anywhere in your code to jump into a debugger (as long as you execute from a shell):
import my_debug
my_debug.db()

I've long been using this code in my sitecustomize.py to start a debugger on an exception. This can also be triggered by Ctrl+C. It works beautifully in the shell, don't know about eclipse.
import sys
def info(exception_type, value, tb):
if hasattr(sys, 'ps1') or not sys.stderr.isatty() or not sys.stdin.isatty() or not sys.stdout.isatty() or type==SyntaxError:
# we are in interactive mode or we don't have a tty-like
# device, so we call the default hook
sys.__excepthook__(exception_type, value, tb)
else:
import traceback
import pdb
if exception_type != KeyboardInterrupt:
try:
import growlnotify
growlnotify.growlNotify("Script crashed", sticky = False)
except ImportError:
pass
# we are NOT in interactive mode, print the exception...
traceback.print_exception(exception_type, value, tb)
print
# ...then start the debugger in post-mortem mode.
pdb.pm()
sys.excepthook = info
Here's the source and more discussion on StackOverflow.

You can jump into an interactive session using code.InteractiveConsole as described here; however I don't know how to trigger this from Eclipse.
A solution might be to intercept Ctrl+C to jump into this interactive console (using the signal module: signal.signal(signal.SIGINT, my_handler)), but it would probably change the execution context and you probably don't want this.

If you are already running in debug mode you can set an additional breakpoint if the program execution is currently paused (e.g. because you are already at a breakpoint). I just tried it out now with the latest Pydev - it works just fine.
If you are running normally (i.e. not in debug mode) all breakpoints will be ignored. No changes to breakpoints will alter the way a non-debug run works.