We Keep Coding

Python debugger with line edition in a program that uses stdin

To add an ad hoc debugger breakpoint in a Python script, I can insert the line
import pdb; pdb.set_trace()
Pdb reads from standard input, so this doesn't work if the script itself also reads from standard input. As a workaround, on a Unix-like system, I can tell pdb to read from the terminal:
import pdb; pdb.Pdb(stdin=open('/dev/tty', 'r'), stdout=open('/dev/tty', 'w')).set_trace()
This works, but unlike with a plain pdb.set_trace, I don't get the benefit of command line edition provided by the readline library (arrow keys, etc.).
How can I enter pdb without interfering with the script's stdin and stdout, and still get command line edition?
Ideally the same code should work in both Python 2 and Python 3. Compatibility with non-Unix systems would be a bonus.
Toy program as a test case:
#!/usr/bin/env python
import sys
for line in sys.stdin:
#import pdb; pdb.set_trace()
import pdb; pdb.Pdb(stdin=open('/dev/tty', 'r'), stdout=open('/dev/tty', 'w')).set_trace()
sys.stdout.write(line)
Usage: { echo one; echo two; } | python cat.py

I hope I have not missed anything important, but it seems like you cannot really do that in an entirely trivial way, because readline would only get used if pdb.Pdb (resp. cmd.Cmd it sublcasses) has use_rawinput set to non-zero, which however would result in ignoring your stdin and mixing inputs for debugger and script itself. That said, the best I've come up with so far is:
#!/usr/bin/env python3
import os
import sys
import pdb
pdb_inst = pdb.Pdb()
stdin_called = os.fdopen(os.dup(0))
console_new = open('/dev/tty')
os.dup2(console_new.fileno(), 0)
console_new.close()
sys.stdin = os.fdopen(0)
for line in stdin_called:
pdb_inst.set_trace()
sys.stdout.write(line)
It is relatively invasive to your original script, even though it could be at least placed outside of it and imported and called or used as a wrapper.
I've redirected (duplicated) the incoming STDIN to a file descriptor and opened that as stdin_called. Then (based on your example) I've opened /dev/tty for reading, replaced process' file descriptor 0 (for STDIN; it should rather use value returned by sys.stdin.fileno()) with this one I've just opened and also reassigned a corresponding file-like object to sys.stdin. This way the programs loop and pdb are using their own input streams while pdb gets to interact with what appears to be just a "normal" console STDIN it is happy to enable readline on.
It isn't pretty, but should be doing what you were after and it hopefully provides useful hints. It uses (if available) readline (line editing, history, completion) when in pdb:
$ { echo one; echo two; } | python3 cat.py
> /tmp/so/cat.py(16)<module>()
-> sys.stdout.write(line)
(Pdb) c
one
> /tmp/so/cat.py(15)<module>()
-> pdb_inst.set_trace()
(Pdb) con[TAB][TAB]
condition cont continue
(Pdb) cont
two
Note starting with version 3.7 you could use breakpoint() instead of import pdb; pdb.Pdb().set_trace() for convenience and you could also check result of dup2 call to make sure the file descriptor got created/replaced as expected.
EDIT: As mentioned earlier and noted in a comment by OP, this is both ugly and invasive to the script. It's not making it any prettier, but we can employ few tricks to reduce impact on its surrounding. One such option I've hacked together:
import sys
# Add this: BEGIN
import os
import pdb
import inspect
pdb_inst = pdb.Pdb()
class WrapSys:
def __init__(self):
self.__stdin = os.fdopen(os.dup(0))
self.__console = open('/dev/tty')
os.dup2(self.__console.fileno(), 0)
self.__console.close()
self.__console = os.fdopen(0)
self.__sys = sys
def __getattr__(self, name):
if name == 'stdin':
if any((f.filename.endswith("pdb.py") for f in inspect.stack())):
return self.__console
else:
return self.__stdin
else:
return getattr(self.__sys, name)
sys = WrapSys()
# Add this: END
for line in sys.stdin:
pdb_inst.set_trace() # Inject breakpoint
sys.stdout.write(line)
I have not dug all the way through, but as is, pdb/cmd really seems to not only need sys.stdin but also for it to use fd 0 in order for readline to kick in. Above example takes things up a notch and within our script hijacks what sys stands for in order to preset different meaning for sys.stdin when code from pdb.py is on a stack. One obvious caveat. If anything else other then pdb also expects and depends on sys.stdin fd to be 0, it still would be out of luck (or reading its input from a different stream if it just went for it).

how to disable pdb.set_trace() without stopping python program and edit the code

I suspect that I have issue in one of my loops, so I setup a break points with pdb.set_trace()
import pdb
for i in range(100):
print("a")
pdb.set_trace()
print("b")
after check variable in this loop for a few times, I decide continue this programming without further breaks. So I try to get the break number with b command, no breaks listed. I guess this line of code don't setup a break point. but How Do I get ride of this "break points" without stopping the program and change the code?

to my knowledge, you could not bypass set_trace, but you could neutralize it, once debugger stopped, type:
pdb.set_trace = lambda: 1
then continue, it wont break again.

Setting a breakpoint (requires Python 3.7):
breakpoint()
Disabling breakpoints set with the breakpoint() function:
import os
os.environ["PYTHONBREAKPOINT"] = "0"
Long story:
In the 3.7 version of Python, the breakpoint() built-in function for setting breakpoints was introduced. By default, it will call pdb.set_trace(). Also, since the 3.7 version of Python the PYTHONBREAKPOINT environment variable is available. It is considered when the breakpoint() function is used.
So, in order to disable these breakpoints (set with the breakpoint() function), one can just set the PYTHONBREAKPOINT environment variable like this:
import os
os.environ["PYTHONBREAKPOINT"] = "0"
It may be useful to mention here sys.breakpointhook() which was also added in the 3.7 version of Python and allows to customize breakpoints behavior.

Unfortunately pdb is missing a bunch of functionality (even basic stuff like display lists), and you've found another example of that here. The good news is that pdb++ is a great drop-in replacement for pdb, and one of the things it solves is exactly the problem of disabling set_trace. So you can simply do:
pip install pdbpp
and then at the (Pdb++) prompt, type
pdb.disable()
Easy! And you will get lots of other useful goodies on top of that.

It is possible to start a Python script without PDB control but then hit a stray set_trace() left there. To prevent breaking into debugger every time set_trace() is encountered, a similar trick as above (changing the symbol's reference to point to a harmless function) can be applied.
However, the namespace of the debuggee has to be modified, not one of the debugger itself. Simply overwriting pdb.set_trace = lambda:1 or set_trace = lambda:1 did not work for me.
The following trick worked from the pdb prompt:
pdb> globals()['set_trace'] = lambda:1
This line first calls globals() to get access to a dict of the program under debugging, and then modifies the reference of set_trace there.

One way around this is to not write the breakpoints in the script itself, but rather set breakpoints when you start python with python -m pdb my_script.py
You then get into a prompt first, before execution of the script starts, and you can write for example
(Pdb) b 321
to set a breakpoint on line 321 (you can also specify file and specify conditions b 321, i == 50)
Then
(Pdb) c
(for continue) to start the execution of the actual script. The breakpoints you set in this way, you can clear when you're done with them with:
(Pdb) cl
(for clear)

Python doctest for shell scripts that test argument parsing without polluting docstring with os.popen()

Is there a way to write a python doctest string to test a script intended to be launched from the command line (terminal) that doesn't pollute the documentation examples with os.popen calls?
#!/usr/bin/env python
# filename: add
"""
Example:
>>> import os
>>> os.popen('add -n 1 2').read().strip()
'3'
"""
if __name__ == '__main__':
from argparse import ArgumentParser
p = ArgumentParser(description=__doc__.strip())
p.add_argument('-n',type = int, nargs = 2, default = 0,help = 'Numbers to add.')
p.add_argument('--test',action = 'store_true',help = 'Test script.')
a = p.parse_args()
if a.test:
import doctest
doctest.testmod()
if a.n and len(a.n)==2:
print a.n[0]+a.n[1]
Running doctest.testmod() without using popen just causes a test failure because the script is run within a python shell instead of a bash (or DOS) shell.
The advanced python course at LLNL suggests putting scripts in files that are separate from .py modules. But then the doctest strings only test the module, without the arg parsing. And my os.popen() approach pollutes the Examples documentation. Is there a better way?

Just found something looking like the answer you want:
shell-doctest.

doctest is meant to run python code, so you have to do a conversion somewhere. If you are determined to test the commandline interface directly via doctest, one possibility is to do a regexp substitution to __doc__ before you pass it to argparse, to take out the os.popen wrapper:
clean = re.sub(r"^>>> os\.popen\('(.*)'\).*", r"% \1", __doc__)
p = ArgumentParser(description=clean, ...)
(Of course there are all sorts of nicer ways to do that, depending on what you consider "nice").
That'll clean it up for the end user. If you also want it to look cleaner in the source, you can go the other way: Put commandline examples in the docstring and don't use doctest.testmodule(). Run your docstring through doctest.script_from_examples and post-process it to insert the os calls. (Then you'll have to embed it into something so you can test it with run_docstring_examples.) doctest doesn't care if the input is valid python, so you can do the following:
>>> print doctest.script_from_examples("""
Here is a commandline example I want converted:
>>> add -n 3 4
7
""")
# Here is a commandline example I want converted:
add -n 3 4
# Expected:
## 7
This will still expose the python prompt >>> in the help. If this bothers you, you may just have to process the string in both directions.

You can also load the docstring yourself and execute the command, like in this test.
import sys
module = sys.modules[__name__]
docstring = module.__doc__
# search in docstring for certain regex, and check that the following line(s) matches a pattern.

Tell if Python is in interactive mode

In a Python script, is there any way to tell if the interpreter is in interactive mode? This would be useful so that, for instance, when you run an interactive Python session and import a module, slightly different code is executed (for example, logging is turned off).
I've looked at tell whether python is in -i mode and tried the code there, however, that function only returns true if Python has been invoked with the -i flag and not when the command used to invoke interactive mode is python with no arguments.
What I mean is something like this:
if __name__=="__main__":
#do stuff
elif __pythonIsInteractive__:
#do other stuff
else:
exit()

__main__.__file__ doesn't exist in the interactive interpreter:
import __main__ as main
print hasattr(main, '__file__')
This also goes for code run via python -c, but not python -m.

I compared all the methods I found and made a table of results. The best one seems to be this:
hasattr(sys, 'ps1')
If anyone has other scenarios that might differ, comment and I'll add it

sys.ps1 and sys.ps2 are only defined in interactive mode.

Use sys.flags:
if sys.flags.interactive:
#interactive
else:
#not interactive

From TFM: If no interface option is given, -i is implied, sys.argv[0] is an empty string ("") and the current directory will be added to the start of sys.path.
If the user invoked the interpreter with python and no arguments, as you mentioned, you could test this with if sys.argv[0] == ''. This also returns true if started with python -i, but according to the docs, they're functionally the same.

The following works both with and without the -i switch:
#!/usr/bin/python
import sys
# Set the interpreter bool
try:
if sys.ps1: interpreter = True
except AttributeError:
interpreter = False
if sys.flags.interactive: interpreter = True
# Use the interpreter bool
if interpreter: print 'We are in the Interpreter'
else: print 'We are running from the command line'

Here's something that would work. Put the following code snippet in a file, and assign the path to that file to the PYTHONSTARTUP environment variable.
__pythonIsInteractive__ = None
And then you can use
if __name__=="__main__":
#do stuff
elif '__pythonIsInteractive__' in globals():
#do other stuff
else:
exit()
http://docs.python.org/tutorial/interpreter.html#the-interactive-startup-file

Jump into a Python Interactive Session mid-program?

Hey I was wondering... I am using the pydev with eclipse and I'm really enjoying the powerful debugging features, but I was wondering:
Is it possible to set a breakpoint in eclipse and jump into the interactive python interpreter during execution?
I think that would be pretty handy ;)
edit: I want to emphasize that my goal is not to jump into a debugger. pydev/eclipse have a great debugger, and I can just look at the traceback and set break points.
What I want is to execute a script and jump into an interactive python interpreter during execution so I can do things like...
poke around
check the values of things
manipulate variables
figure out some code before I add it to the app
I know you can do this all with a debugger, but I can do it faster in the interactive interpreter because I can try something, see that it didn't work, and try something else without having get the app back to the point of executing that code again.

So roughly a year on from the OP's question, PyDev has this capability built in. I am not sure when this feature was introduced, but all I know is I've spent the last ~2hrs Googling... configuring iPython and whatever (which was looking like it would have done the job), but only to realise Eclipse/PyDev has what I want ootb.
As soon as you hit a breakpoint in debug mode, the console is right there ready and waiting!
I only didn't notice this as there is no prompt or blinking cursor; I had wrongly assumed it was a standard, output-only, console... but it's not. It even has code-completion.
Great stuff, see http://pydev.org/manual_adv_debug_console.html for more details.

This is from an old project, and I didn't write it, but it does something similar to what you want using ipython.
'''Start an IPython shell (for debugging) with current environment.
Runs Call db() to start a shell, e.g.
def foo(bar):
for x in bar:
if baz(x):
import ipydb; ipydb.db() # <-- start IPython here, with current value of x (ipydb is the name of this module).
.
'''
import inspect,IPython
def db():
'''Start IPython shell with callers environment.'''
# find callers
__up_frame = inspect.currentframe().f_back
eval('IPython.Shell.IPShellEmbed([])()', # Empty list arg is
# ipythons argv later args to dict take precedence, so
# f_globals() shadows globals(). Need globals() for IPython
# module.
dict(globals().items() + __up_frame.f_globals.items()),
__up_frame.f_locals)
edit by Jim Robert (question owner): If you place the above code into a file called my_debug.py for the sake of this example. Then place that file in your python path, and you can insert the following lines anywhere in your code to jump into a debugger (as long as you execute from a shell):
import my_debug
my_debug.db()

I've long been using this code in my sitecustomize.py to start a debugger on an exception. This can also be triggered by Ctrl+C. It works beautifully in the shell, don't know about eclipse.
import sys
def info(exception_type, value, tb):
if hasattr(sys, 'ps1') or not sys.stderr.isatty() or not sys.stdin.isatty() or not sys.stdout.isatty() or type==SyntaxError:
# we are in interactive mode or we don't have a tty-like
# device, so we call the default hook
sys.__excepthook__(exception_type, value, tb)
else:
import traceback
import pdb
if exception_type != KeyboardInterrupt:
try:
import growlnotify
growlnotify.growlNotify("Script crashed", sticky = False)
except ImportError:
pass
# we are NOT in interactive mode, print the exception...
traceback.print_exception(exception_type, value, tb)
print
# ...then start the debugger in post-mortem mode.
pdb.pm()
sys.excepthook = info
Here's the source and more discussion on StackOverflow.

You can jump into an interactive session using code.InteractiveConsole as described here; however I don't know how to trigger this from Eclipse.
A solution might be to intercept Ctrl+C to jump into this interactive console (using the signal module: signal.signal(signal.SIGINT, my_handler)), but it would probably change the execution context and you probably don't want this.

If you are already running in debug mode you can set an additional breakpoint if the program execution is currently paused (e.g. because you are already at a breakpoint). I just tried it out now with the latest Pydev - it works just fine.
If you are running normally (i.e. not in debug mode) all breakpoints will be ignored. No changes to breakpoints will alter the way a non-debug run works.