I've made a bunch of tweaks to NPP following this guide, and so far it's gone over well.
But I have a somewhat annoying problem. I want to use input commands in my code.
If the console is closed, this isn't a problem—it pops up and I can type right away in the input field just fine, no clicking.
But when I edit the code and re-run the script (without closing the console), the console clears and runs the program, but I have to click over to be able to interact.
I don't want to close the NppExec console every time I finish the script.
I don't want to have to click over on the console every time I run the script.
As a bonus, I don't want to have to kill my script every time I run it again, either.
I just want to run, type required inputs, go back to editing, re-run, type, etc. without interruptions, if possible..
Can anyone help me with this?
Thank you....
Please check the NppExec Manual for NppExec ver. 0.6 RC2. It includes a lovely script in the section "4.6.4. Running Python & wxPython":
npp_console - // disable any output to the Console
npp_save // save current file (a .py file is expected)
cd "$(CURRENT_DIRECTORY)" // use the current file's dir
set local #exit_cmd_silent = exit() // allows to exit Python automatically
set local PATH_0 = $(SYS.PATH) // current value of %PATH%
env_set PATH = $(SYS.PATH);C:\Python27 // use Python 2.7
npp_setfocus con // set the focus to the Console
npp_console + // enable output to the Console
python -i -u "$(FILE_NAME)" // run Python's program interactively
npp_console - // disable any output to the Console
env_set PATH = $(PATH_0) // restore the value of %PATH%
npp_console + // enable output to the Console
The command
npp_setfocus con
looks like the one you are looking for.
I'm trying to debug a Python program using pdb. The program could be like this:
def main():
a = 1
print(b)
c = 2
d = 3
Apparently, print(b) is a typo which should be print(a) but it is not important and I can fix it with the text editor but I want to bypass this error and continue debugging.
I tried jump, like jump 4(assuming "c=2" is line 4) but I was given error "Jump failed: f_lineno can only be set by a line trace function", which means I need to give a line trace function when I'm programming.
So, is there a way to deal with this problem, or is there some other way to bypass the error line when using pdb?
TLDR: this is pdb's post-mortem mode in which jumping is not supposed to work. But it's still very useful.
Painting by Rembrandt (public domain)
I reproduce it with python 3.8.2 as *** Jump failed: can only jump from a 'line' trace event by running the script "under pdb" like so: python3 -m pdb -c c script.py and trying to jump to another line in pdb prompt which then appears.
What's happened: an unhandled exception, in this case NameError: name 'b' is not defined caused python to stop interpreting the script; pdb intercepted this situation and entered its post-mortem mode.
As Almar Klein nicely put it in his blog post,
Post-mortem debugging refers to the concept of entering debug mode after something has broken. There is no setting of breakpoints involved, so it's very quick and you can inspect the full stack trace, making it an effective way of tracing errors.
Although jump, next, return won't work in post-mortem, bt, up, down, ll and pp, along with the possibility to import modules and run arbitrary python code directly in the pdb's interactive shell can be very effective ways to get the root cause. In our simple example the root cause of the NameError is shown immediately upon a quick ll: pdb prefixes the offending line of code with >>.
Had we not passed -c c (meaning continue), pdb would have shown its prompt and paused before the first line of your program is interpreted, so you'd have a chance to step through the whole program or set a breakpoint before or at the offending line, and jump over it, never entering the post-mortem.
Even in post-mortem, you can prepare a breakpoint anywhere in the program, e.g. break 2 for line 2, and say c or continue so pdb will finish post-mortem, reload the file, and restart the program with the updated set of breakpoints.
Another way to deal with it is to import pdb and pdb.set_trace() in suspicious code - or since python 3.7, simply breakpoint() - and run the python program normally (not "under" pdb anymore) which allows then to jump, next, return etc, as well as everything else - when the breakpoint is hit.
If your Python program is started through behave:
prefer to run behave with --no-capture whenever using pdb or similar debuggers (whether post-mortem mode or not), to avoid problems with behave's stdin/stdout capturing making pdb unresponsive and/or its prompt invisible.
best of all, if you want to end up in the pdb post-mortem mode automatically while potentially still supporting capturing, set the post_mortem environment variable (can also name it differently) to any value (but only on dev machine, not for automated CI or production!) and commit the following permanently into environment.py:
def after_step(context, step):
import os
if 'post_mortem' in os.environ and step.status == 'failed':
import pdb
# Similar to "behave --no-capture" calling stop_capture() ensures visibility of pdb's prompts,
# while still supporting capture until an uncaught error occurs.
# Warning: this does rely on behave's internals which might change
context._runner.stop_capture() # pylint: disable=protected-access
pdb.post_mortem(step.exc_traceback)
I'm not sure, but this may be a bug that was fixed in Mar 2018, so you may need to (patch, upgrade, reinstall?) your Python.
I can debug Python code using ddd -pydb prog.py. All the python command line arguments can be passed too after prog.py. In my case, many classes have been implemented in C++ that are exposed to python using boost-python. I wish I could debug python code and C++ together. For example I want to set break points like this :
break my_python.py:123
break my_cpp.cpp:456
cont
Of course I am trying it after compiling c++ codes with debug option but the debugger does not cross boost boundary. Is there any way?
EDIT:
I saw http://www.boost.org/doc/libs/1_61_0/libs/python/doc/html/faq/how_do_i_debug_my_python_extensi.html.
I followed it and I can do debugging both for python and C++. But I preferably want to do visual debugging with DDD but I don't know how to give 'target exec python' command inside DDD. If not (just using gdb as in the link) I should be able to debug for a Python script not interactively giving python commands as in the link.
I found out how to debug the C++ part while running python. (realized it while reading about process ID detection in Python book..).
First you run the python program which includes C++ programs. At the start of the python program, use raw_input() to make the program wait for you input. But just before that do print os.getpid() (of course you should have imported os package). When you run the python program, it will have printed the pid of the python program you are running and will be waiting for your keyboard input.
python stop code :
import os
def w1(str):
print (str)
wait = raw_input()
return
print os.getpid()
w1('starting main..press a key')
result :
27352
starting main..press a key
Or, you can use import pdb, pdb.set_trace() as comment below.(thanks #AndyG) and see EDIT* to get pid using ps -aux.
Now, suppose the C++ shared library is _caffe.so (which is my case. This _caffe.so library has all the C++ codes and boost python wrapper functions). 27352 is the pid. Then in another shell start gdb like
gdb caffe-fast-rcnn/python/caffe/_caffe.so 27352
or if you want to use graphical debugging using like DDD, do
ddd caffe-fast-rcnn/python/caffe/_caffe.so 27352
Then you'll see gdb starts and wait with prompt. The python program is interrupted by gdb and waits in stopped mode (it was waiting for your key input but now it's really in stopeed mode, and it needs gdb continue command from the second debugger to proceed with the key waiting).
Now you can give break point command in gdb like
br solver.cpp:225
and you can see message like
Breakpoint 1 at 0x7f2cccf70397: file src/caffe/solver.cpp, line 226. (2 locations)
When you give continue command in the second gdb window(that was holding the program), the python code runs again. Of course you should give a key input in the first gdb window to make it proceed.
Now at least you can debug the C++ code while running python program(that's what I wanted to do)!
I later checked if I can do python and C++ debugging at the same time and it works. You start the debugger(DDD) like ddd -pydb prog1.py options.. and attach another DDD using method explained above. Now you can set breakpoints for python and C++ and using other debug functions in each window(I wish I had known this a couple of months earlier.. It should have helped tons.).
EDIT : to get the pid, you can do ps -aux | grep python instead. This pid is the next of ddd's pid.
I had a similar problem, but failed to get the solutions in Chan's answer to work (on MAC OS X 10.12.4). Instead the following worked for me.
Write a python script test.py that imports and uses the boost.Python module.
start python in the debugger
lldb python3 test.py
giving
> lldb python3 test.py
(lldb) target create "python3"
Current executable set to 'python3' (x86_64).
(lldb) settings set -- target.run-args "test.py"
(lldb) run
Process 46189 launched: '/Users/me/anaconda/bin/python3' (x86_64)
test.cpython-36m-darwin.so was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 46189 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10d4b3000)
frame #0: 0x00000001019f49c2 test.cpython-36m-darwin.so`std::__1::enable_if<true, void>::type (anonymous namespace)::Render2D<double>::add_particle<true, 5ul>(float*, float, float, float, float) const [inlined] mylib::SSE::packed<8ul, float>::loadu(
944 { return {_mm256_load_ps(p)}; }
945 /// load from unaligned memory location
946 static __always__inline packed loadu(const element_type*p) noexcept
-> 947 { return {_mm256_loadu_ps(p)}; }
948 /// load from aligned memory location, using template arg for alignment
949 template<bool aligned>
950 static __always_inline enable_if_t< aligned, packed>
No need to obtain the pid and start the debugger from a separate window or set any breakpoints.
I am invoking the bash script from python script.
I want the bash script to add an element to dictionary "d" in the python script
abc3.sh:
#!/bin/bash
rank=1
echo "plugin"
function reg()
{
if [ "$1" == "what" ]; then
python -c 'from framework import data;data(rank)'
echo "iamin"
else
plugin
fi
}
plugin()
{
echo "i am plugin one"
}
reg $1
python file:
import sys,os,subprocess
from collections import *
subprocess.call(["./abc3.sh what"],shell=True,executable='/bin/bash')
def data(rank,check):
d[rank]["CHECK"]=check
print d[1]["CHECK"]
If I understand correctly, you have a python script that runs a shell script, that in turn runs a new python script. And you'd want the second Python script to update a dictionnary in the first script. That will not work like that.
When you run your first python script, it will create a new python process, which will interpret each instruction from your source script.
When it reaches the instruction subprocess.call(["./abc3.sh what"],shell=True,executable='/bin/bash'), it will spawn a new shell (bash) process which will in turn interpret your shell script.
When the shell script reaches python -c <commands>, it invokes a new python process. This process is independant from the initial python process (even if you run the same script file).
Because each of theses scripts will run in a different process, they don't have access to each other data (the OS makes sure that each process is independant from each other, excepted for specific inter-process communications methods).
What you need to do: use some kind of interprocess mechanism, so that the initial python script gets data from the shell script. You may for example read data from the shell standard output, using https://docs.python.org/3/library/subprocess.html#subprocess.check_output
Let's suppose that you have a shell plugin that echoes the value:
echo $1 12
The mockup python script looks like (I'm on windows/MSYS2 BTW, hence the strange paths for a Linux user):
import subprocess
p = subprocess.Popen(args=[r'C:\msys64\usr\bin\sh.exe',"-c","C:/users/jotd/myplugin.sh myarg"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
o,e= p.communicate()
p.wait()
if len(e):
print("Warning: error found: "+e.decode())
result = o.strip()
d=dict()
d["TEST"] = result
print(d)
it prints the dictionary, proving that argument has been passed to the shell, and went back processed.
Note that stderr has been filtered out to avoid been mixed up with the results, but is printed to the console if occurs.
{'TEST': b'myarg 12'}
From what I have read, there are two ways to debug code in Python:
With a traditional debugger such as pdb or ipdb. This supports commands such as c for continue, n for step-over, s for step-into etc.), but you don't have direct access to an IPython shell which can be extremely useful for object inspection.
Using IPython by embedding an IPython shell in your code. You can do from IPython import embed, and then use embed() in your code. When your program/script hits an embed() statement, you are dropped into an IPython shell. This allows the full inspection of objects and testing of Python code using all the IPython goodies. However, when using embed() you can't step-by-step through the code anymore with handy keyboard shortcuts.
Is there any way to combine the best of both worlds? I.e.
Be able to step-by-step through your code with handy pdb/ipdb keyboard shortcuts.
At any such step (e.g. on a given statement), have access to a full-fledged IPython shell.
IPython debugging as in MATLAB:
An example of this type of "enhanced debugging" can be found in MATLAB, where the user always has full access to the MATLAB engine/shell, and she can still step-by-step through her code, define conditional breakpoints, etc. From what I have discussed with other users, this is the debugging feature that people miss the most when moving from MATLAB to IPython.
IPython debugging in Emacs and other editors:
I don't want to make the question too specific, but I work mostly in Emacs, so I wonder if there is any way to bring this functionality into it. Ideally, Emacs (or the editor) would allow the programmer to set breakpoints anywhere on the code and communicate with the interpreter or debugger to have it stop in the location of your choice, and bring to a full IPython interpreter on that location.
What about ipdb.set_trace() ? In your code :
import ipdb; ipdb.set_trace()
update: now in Python 3.7, we can write breakpoint(). It works the same, but it also obeys to the PYTHONBREAKPOINT environment variable. This feature comes from this PEP.
This allows for full inspection of your code, and you have access to commands such as c (continue), n (execute next line), s (step into the method at point) and so on.
See the ipdb repo and a list of commands. IPython is now called (edit: part of) Jupyter.
ps: note that an ipdb command takes precedence over python code. So in order to write list(foo) you'd need print(list(foo)), or !list(foo) .
Also, if you like the ipython prompt (its emacs and vim modes, history, completions,…) it's easy to get the same for your project since it's based on the python prompt toolkit.
You can use IPython's %pdb magic. Just call %pdb in IPython and when an error occurs, you're automatically dropped to ipdb. While you don't have the stepping immediately, you're in ipdb afterwards.
This makes debugging individual functions easy, as you can just load a file with %load and then run a function. You could force an error with an assert at the right position.
%pdb is a line magic. Call it as %pdb on, %pdb 1, %pdb off or %pdb 0. If called without argument it works as a toggle.
(Update on May 28, 2016) Using RealGUD in Emacs
For anyone in Emacs, this thread shows how to accomplish everything described in the OP (and more) using
a new important debugger in Emacs called RealGUD which can operate with any debugger (including ipdb).
The Emacs package isend-mode.
The combination of these two packages is extremely powerful and allows one to recreate exactly the behavior described in the OP and do even more.
More info on the wiki article of RealGUD for ipdb.
Original answer:
After having tried many different methods for debugging Python, including everything mentioned in this thread, one of my preferred ways of debugging Python with IPython is with embedded shells.
Defining a custom embedded IPython shell:
Add the following on a script to your PYTHONPATH, so that the method ipsh() becomes available.
import inspect
# First import the embed function
from IPython.terminal.embed import InteractiveShellEmbed
from IPython.config.loader import Config
# Configure the prompt so that I know I am in a nested (embedded) shell
cfg = Config()
prompt_config = cfg.PromptManager
prompt_config.in_template = 'N.In <\\#>: '
prompt_config.in2_template = ' .\\D.: '
prompt_config.out_template = 'N.Out<\\#>: '
# Messages displayed when I drop into and exit the shell.
banner_msg = ("\n**Nested Interpreter:\n"
"Hit Ctrl-D to exit interpreter and continue program.\n"
"Note that if you use %kill_embedded, you can fully deactivate\n"
"This embedded instance so it will never turn on again")
exit_msg = '**Leaving Nested interpreter'
# Wrap it in a function that gives me more context:
def ipsh():
ipshell = InteractiveShellEmbed(config=cfg, banner1=banner_msg, exit_msg=exit_msg)
frame = inspect.currentframe().f_back
msg = 'Stopped at {0.f_code.co_filename} at line {0.f_lineno}'.format(frame)
# Go back one level!
# This is needed because the call to ipshell is inside the function ipsh()
ipshell(msg,stack_depth=2)
Then, whenever I want to debug something in my code, I place ipsh() right at the location where I need to do object inspection, etc. For example, say I want to debug my_function below
Using it:
def my_function(b):
a = b
ipsh() # <- This will embed a full-fledged IPython interpreter
a = 4
and then I invoke my_function(2) in one of the following ways:
Either by running a Python program that invokes this function from a Unix shell
Or by invoking it directly from IPython
Regardless of how I invoke it, the interpreter stops at the line that says ipsh(). Once you are done, you can do Ctrl-D and Python will resume execution (with any variable updates that you made). Note that, if you run the code from a regular IPython the IPython shell (case 2 above), the new IPython shell will be nested inside the one from which you invoked it, which is perfectly fine, but it's good to be aware of. Eitherway, once the interpreter stops on the location of ipsh, I can inspect the value of a (which be 2), see what functions and objects are defined, etc.
The problem:
The solution above can be used to have Python stop anywhere you want in your code, and then drop you into a fully-fledged IPython interpreter. Unfortunately it does not let you add or remove breakpoints once you invoke the script, which is highly frustrating. In my opinion, this is the only thing that is preventing IPython from becoming a great debugging tool for Python.
The best you can do for now:
A workaround is to place ipsh() a priori at the different locations where you want the Python interpreter to launch an IPython shell (i.e. a breakpoint). You can then "jump" between different pre-defined, hard-coded "breakpoints" with Ctrl-D, which would exit the current embedded IPython shell and stop again whenever the interpreter hits the next call to ipsh().
If you go this route, one way to exit "debugging mode" and ignore all subsequent breakpoints, is to use ipshell.dummy_mode = True which will make Python ignore any subsequent instantiations of the ipshell object that we created above.
You can start IPython session from pudb and go back to the debugging session as you like.
BTW, ipdb is using IPython behind the scenes and you can actually use IPython functionality such as TAB completion and magic commands (the one starts with %). If you are OK with ipdb you can start it from IPython using commands such as %run and %debug. ipdb session is actually better than plain IPython one in the sense you can go up and down in the stack trace etc. What is missing in ipdb for "object inspection"?
Also, python.el bundled with Emacs >= 24.3 has nice ipdb support.
Looks like the approach in #gaborous's answer is deprecated.
The new approach seems to be:
from IPython.core import debugger
debug = debugger.Pdb().set_trace
def buggy_method():
debug()
Prefixing an "!" symbol to commands you type in pdb seems to have the same effect as doing something in an IPython shell. This works for accessing help for a certain function, or even variable names. Maybe this will help you to some extent. For example,
ipdb> help(numpy.transpose)
*** No help on (numpy.transpose)
But !help(numpy.transpose) will give you the expected help page on numpy.transpose. Similarly for variable names, say you have a variable l, typing "l" in pdb lists the code, but !l prints the value of l.
You can start IPython from within ipdb.
Induce the ipdb debugger1:
import idpb; ipdb.set_trace()
Enter IPython from within in the ipdb> console2:
from IPython import embed; embed()
Return to the ipdb> console from within IPython:
exit
If you're lucky enough to be using Emacs, things can be made even more convenient.
This requires using M-x shell. Using yasnippet and bm, define the following snippet. This will replace the text ipdb in the editor with the set-trace line. After inserting the snippet, the line will be highlighted so that it is easily noticeable and navigable. Use M-x bm-next to navigate.
# -*- mode: snippet -*-
# name: ipdb
# key: ipdb
# expand-env: ((yas-after-exit-snippet-hook #'bm-toggle))
# --
import ipdb; ipdb.set_trace()
1 All on one line for easy deletion. Since imports only happen once, this form ensures ipdb will be imported when you need it with no extra overhead.
2 You can save yourself some typing by importing IPython within your .pdbrc file:
try:
from IPython import embed
except:
pass
This allows you to simply call embed() from within ipdb (of course, only when IPython is installed).
Did you try this tip?
Or better still, use ipython, and call:
from IPython.Debugger import Tracer; debug_here = Tracer()
then you can just use
debug_here()
whenever you want to set a breakpoint
the right, easy, cool, exact answer for the question is to use %run macro with -d flag.
In [4]: run -d myscript.py
NOTE: Enter 'c' at the ipdb> prompt to continue execution.
> /cygdrive/c/Users/mycodefolder/myscript.py(4)<module>()
2
3
----> 4 a=1
5 b=2
One option is to use an IDE like Spyder which should allow you to interact with your code while debugging (using an IPython console, in fact). In fact, Spyder is very MATLAB-like, which I presume was intentional. That includes variable inspectors, variable editing, built-in access to documentation, etc.
If you type exit() in embed() console the code continue and go to the next embed() line.
The Pyzo IDE has similar capabilities as the OP asked for. You don't have to start in debug mode. Similarly to MATLAB, the commands are executed in the shell. When you set up a break-point in some source code line, the IDE stops the execution there and you can debug and issue regular IPython commands as well.
It does seem however that step-into doesn't (yet?) work well (i.e. stopping in one line and then stepping into another function) unless you set up another break-point.
Still, coming from MATLAB, this seems the best solution I've found.
From python 3.2, you have the interact command, which gives you access to the full python/ipython command space.
Running from inside Emacs' IPython-shell and breakpoint set via pdb.set_trace() should work.
Checked with python-mode.el, M-x ipython RET etc.
Developing New Code
Debugging inside IPython
Use Jupyter/IPython cell execution to speed up experiment iterations
Use %%debug for step through
Cell Example:
%%debug
...: for n in range(4):
...: n>2
Debugging Existing Code
IPython inside debugging
Debugging a broken unit test: pytest ... --pdbcls=IPython.terminal.debugger:TerminalPdb --pdb
Debugging outside of test case: breakpoint(), python -m ipdb, etc.
IPython.embed() for full IPython functionality where needed while in the debugger
Thoughts on Python
I agree with the OP that many things MATLAB does nicely Python still does not have and really should since just about everything in the language favors development speed over production speed. Maybe someday I will contribute more than trivial bug fixes to CPython.
https://github.com/ipython/ipython/commit/f042f3fea7560afcb518a1940daa46a72fbcfa68
See also Is it possible to run commands in IPython with debugging?
If put import ipdb; ipdb.set_trace() at cell outside function, it will occur error.
Using %pdb or %debug, you can only see the filnal error result. You cannot see the code doing step by step.
I use following skill:
%%writefile temp.py
.....cell code.....
save the code of cell to file temp.py.
and then
%run -i -d temp.py, it will run the cell code by pdb .
-i: run the file in IPython’s namespace instead of an empty one.
-d: run your program under the control of pdb, the Python debugger.