I have a Python script, and I want to execute it up to a certain point, then stop, and keep the interpreter open, so I can see the variables it defines, etc.
I know I could generate an exception, or I could invoke the debugger by running pdb.set_trace(), then stop the debugger, which is what I currently use.
...but is there a command that will just stop the script, as if it had simply reached its end? This would be equivalent to commenting the entire rest of the script (but I would not like doing that), or putting an early return statement in a function.
It seems as if something like this has to exist but I have not found it so far.
Edit: Some more details of my usecase
I'm normally using the regular Python consoles in Spyder. IPython seems like a good thing but ( at least for the version I'm currently on, 2.2.5) some of the normal console's features don't work well in IPython (introspection, auto-completion).
More often than not, my code generates matplotlib figures. In debug mode, those cannot be updated (to my knowledge), which is why I need to get completely out of the script, but not the interpreter).
Another limit of the debugger is that I can't execute loops in it: you can copy/paste the code for a loop into the regular console and have it execute, but that won't work in the debugger (at least in my Spyder version).
If you invoke your program with python -i <script>, the interpreter will remain active after the script ends. raise SystemExit would be the easiest way to force it to end at an arbitrary point.
If you have ipython (highly, highly recommended), you can go to any point in your program and add the following lines
import IPython
IPython.embed()
Once your program reaches that point, the embed command will open up a new IPython shell within that context.
I really like to do that for things where I don't want to go the full pdb route.
If you are using the Python Shell, just press CTRL + C to throw a KeyboardInterrupt. You can then check out the state of the program at the time the exception was throw.
x = 0
while True:
x += 1
Running the script...
Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> ================================ RESTART ================================
Traceback (most recent call last):
File "C:/Python27/test.py", line 2, in
while True:
KeyboardInterrupt
>>> x
15822387
I have a data processing pipeline setup that I want to debug.
The pipeline consists of a bash script that calls a python script.
I usually use iPython's embed() function for debugging. However, when calling the python script from the bash file, the embed() function is called but immediately exited, without me being able to interfere. When running the same python program directly from the command line I don't observe this kind of behavior. Is this intended behavior or am I doing something wrong?
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
Type "copyright", "credits" or "license" for more information.
IPython 2.4.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]:
Do you really want to exit ([y]/n)?
'follow up code prints here'
I can replicate the problem like this:
# test.py
import IPython
import sys
print(sys.stdin.read())
IPython.embed()
# session
❯ echo 'foo' | python test.py
foo
Python 3.6.8 (default, Oct 7 2019, 12:59:55)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.10.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: Do you really want to exit ([y]/n)?
❯ # I didn't quit on purpose, it happened automatically
STDIN is not a TTY, so I'm thinking that IPython is worried that the inbound text (via the pipe) won't be a user typing. It doesn't want foo (from my example above) to spew into the IPython shell and do something unexpected.
You can work around this by getting your terminal id via the tty command, and redirecting stdin to the calling terminal after it has finished reading from the pipe, something like this:
with open('/dev/pts/16') as user_tty:
sys.stdin=user_tty
IPython.embed()
For more on ttys, see this post. Note also that if you put the wrong tty in there, input from some other terminal will control IPython.
I'm not sure if it's possible for IPython to know what the calling tty would have been, had it not been overwritten by bash to be the output-side of the pipe.
Edit: Here's my workaround put more simply: How do I debug a script that uses stdin with ipython?
I ran some experiments to see the behaviour. I noticed that IPython shows the console if any of the ancestor process is terminal.
Following are the files in /tmp directory:
x.py
import IPython
IPython.embed()
call.sh
/usr/bin/python /tmp/x.py
call2.sh
/tmp/call.sh
Experiment 1
Running python x.py does open the IPython shell and waits.
Experiment 2
Running bash call.sh also opens the IPython shell and waits.
Experiment 3
Running bash call2.sh also opens the IPython shell and waits.
As you can see, it does not matter how deep is your IPython.embed call is. It always starts the interactive console and waits.
Lets try if it also works when we fork a new process.
fork.sh
/usr/bin/python /tmp/x.py &
Experiment 4
In this case, IPython shell started but immediately exited. Notice the & at the end. It starts a different process. IPython was not able to access the terminal in this case and hence exited gracefully.
I have a python program which first does some tasks, and then in certain conditions goes into an interactive mode, where the user has access to an interactive python console with the current program's scope. Right now I do this using the code module by calling code.InteractiveConsole(globals()).interact('') (see http://docs.python.org/2/library/code.html).
My problem is that the resulting interactive console lacks some functionalities that I usually get with the standard python console (i.e. the one you get by typing 'python' in a terminal), such as remembering the previous command, etc. Is there a way to get that same interactive console in the middle of my python program, or even better yet ipython's interactive console?
Just use IPython.embed() where you're currently using code.InteractiveConsole(globals()).interact('').
Make sure you're importing IPython before you do that, though:
import IPython
# lots of code
# even more code
IPython.embed()
You can use the builtin breakpoint() function (Available in Python 3.7+) to launch the interactive IPython shell with IPython.embed(). This is nice as it is shorter to type (and does not need an import).
By default breakpoint() launches a python debugger. To make it launch the ipython shell, you have to set the environmental variable PYTHONBREAKPOINT to be IPython.embed.
On Linux:
Run or add to your ~/.bashrc or ~/.profile:
export PYTHONBREAKPOINT="IPython.embed"
For more info about breakpoint(), see PEP 553
this is the first time I have used Python.
I downloaded the file ActivePython-2.7.1.4-win32-x86
and installed it on my computer; I'm using Win7.
So when I tried to run a python program, it appears and disappears very quickly. I don't have enough time to see anything on the screen. I just downloaded the file and double-cliked on it.
How do I launch this file? I know that it is a long file for a first Python tutorial.
Add the line
input()
to the end of the program, with the correct indentation. The issue is that after the data is printed to the console the program finishes, so the console goes away. input tells the program to wait for input, so the console won't be closed when it finishes printing.
I hope you're not using that program to learn Python; it's pretty complicated!
go to Start > All programs > Accessories and click on Command Prompt. then drag the python file from the explorer view into this command line and press Enter...
now you can watch the output of the script execution !
run it from a command prompt:
> python myscript.py
You can also start only the python interpreter from the command prompt (or by running python.exe) and then try some commands:
> python
Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
>>> a = 2
>>> b = 7
>>> print a+b
9
>>>
Or run it from a batch file:
myprog.py
pause
Has the advantage that you can specify a different version of Python too.
Just a bit more on this.
You have a script myscript.py in a folder C:\myscripts. This is how to set up Windows 7 so that you can type > myscript into a CMD window and the script will run.
1) Set your PATH variable to include the Python Interpreter.
Control Panel > System and Security > System > Advanced Settings > Environment Variables. You can set either the System Variables or the User Variables. Scroll down till you find PATH, select it, click Edit.The Path appears selected in a new dialog. I always copy it into Notepad to edit it though all you need do is add ;C:\Python27 to the end of the list. Save this.
2) Set your PATH variable to include C:\myscripts
3) Set your PATHEXT variable to include ;.PY. (This is the bit that saves you from typing myscript.py)
This may now just work. Try opening a command window and typing myscript
But it may not. Windows can still mess you about. I had installed and then uninstalled a Python package and when I typed myscript Windows opened a box asking me which program to use. I browsed for C:\python27\python.exe and clicked that. Windows opened another command window ran the script and closed it before I could see what my script had done! To fix this when Windows opens its dialog select your Python and click the "Always do this" checkbox at the bottom. Then it doesn't open and close another window and things work as they should. Or they did for me.
Added: Above does not say how to pass arguments to your script. For this see answer Windows fails to pass arguments to python script
In Java/C# you can easily step through code to trace what might be going wrong, and IDE's make this process very user friendly.
Can you trace through python code in a similar fashion?
Yes! There's a Python debugger called pdb just for doing that!
You can launch a Python program through pdb by using pdb myscript.py or python -m pdb myscript.py.
There are a few commands you can then issue, which are documented on the pdb page.
Some useful ones to remember are:
b: set a breakpoint
c: continue debugging until you hit a breakpoint
s: step through the code
n: to go to next line of code
l: list source code for the current file (default: 11 lines including the line being executed)
u: navigate up a stack frame
d: navigate down a stack frame
p: to print the value of an expression in the current context
If you don't want to use a command line debugger, some IDEs like Pydev, Wing IDE or PyCharm have a GUI debugger. Wing and PyCharm are commercial products, but Wing has a free "Personal" edition, and PyCharm has a free community edition.
By using Python Interactive Debugger 'pdb'
First step is to make the Python interpreter to enter into the debugging mode.
A. From the Command Line
Most straight forward way, running from command line, of python interpreter
$ python -m pdb scriptName.py
> .../pdb_script.py(7)<module>()
-> """
(Pdb)
B. Within the Interpreter
While developing early versions of modules and to experiment it more iteratively.
$ python
Python 2.7 (r27:82508, Jul 3 2010, 21:12:11)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb_script
>>> import pdb
>>> pdb.run('pdb_script.MyObj(5).go()')
> <string>(1)<module>()
(Pdb)
C. From Within Your Program
For a big project and long-running module, can start the debugging from inside the program using
import pdb and set_trace()
like this :
#!/usr/bin/env python
# encoding: utf-8
#
import pdb
class MyObj(object):
count = 5
def __init__(self):
self.count= 9
def go(self):
for i in range(self.count):
pdb.set_trace()
print i
return
if __name__ == '__main__':
MyObj(5).go()
Step-by-Step debugging to go into more internal
Execute the next statement… with “n” (next)
Repeating the last debugging command… with ENTER
Quitting it all… with “q” (quit)
Printing the value of variables… with “p” (print)
a) p a
Turning off the (Pdb) prompt… with “c” (continue)
Seeing where you are… with “l” (list)
Stepping into subroutines… with “s” (step into)
Continuing… but just to the end of the current subroutine… with “r” (return)
Assign a new value
a) !b = "B"
Set a breakpoint
a) break linenumber
b) break functionname
c) break filename:linenumber
Temporary breakpoint
a) tbreak linenumber
Conditional breakpoint
a) break linenumber, condition
Note:**All these commands should be execute from **pdb
For in-depth knowledge, refer:-
https://pymotw.com/2/pdb/
https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/
There is a module called 'pdb' in python. At the top of your python script you do
import pdb
pdb.set_trace()
and you will enter into debugging mode. You can use 's' to step, 'n' to follow next line similar to what you would do with 'gdb' debugger.
Starting in Python 3.7, you can use the breakpoint() built-in function to enter the debugger:
foo()
breakpoint() # drop into the debugger at this point
bar()
By default, breakpoint() will import pdb and call pdb.set_trace(). However, you can control debugging behavior via sys.breakpointhook() and use of the environment variable PYTHONBREAKPOINT.
See PEP 553 for more information.
ipdb (IPython debugger)
ipdb adds IPython functionality to pdb, offering the following HUGE improvements:
tab completion
show more context lines
syntax highlight
Much like pdg, ipdb is still far from perfect and completely rudimentary if compared to GDB, but it is already a huge improvement over pdb.
Usage is analogous to pdb, just install it with:
python3 -m pip install --user ipdb
and then add to the line you want to step debug from:
__import__('ipdb').set_trace(context=21)
You likely want to add a shortcut for that from your editor, e.g. for Vim snipmate I have:
snippet ipd
__import__('ipdb').set_trace(context=21)
so I can type just ipd<tab> and it expands to the breakpoint. Then removing it is easy with dd since everything is contained in a single line.
context=21 increases the number of context lines as explained at: How can I make ipdb show more lines of context while debugging?
Alternatively, you can also debug programs from the start with:
ipdb3 main.py
but you generally don't want to do that because:
you would have to go through all function and class definitions as Python reads those lines
I don't know how to set the context size there without hacking ipdb. Patch to allow it: https://github.com/gotcha/ipdb/pull/155
Or alternatively, as in raw pdb 3.2+ you can set some breakpoints from the command line:
ipdb3 -c 'b 12' -c 'b myfunc' ~/test/a.py
although -c c is broken for some reason: https://github.com/gotcha/ipdb/issues/156
python -m module debugging has been asked at: How to debug a Python module run with python -m from the command line? and since Python 3.7 can be done with:
python -m pdb -m my_module
Serious missing features of both pdb and ipdb compared to GDB:
persistent command history across sessions: Save command history in pdb
ipdb specific annoyances:
multithreading does not work well if you don't hack some settings...
ipdb, multiple threads and autoreloading programs causing ProgrammingError
https://github.com/gotcha/ipdb/issues/51
Tested in Ubuntu 16.04, ipdb==0.11, Python 3.5.2.
VSCode
If you want to use an IDE, this is a good alternative to PyCharm.
Install VSCode
Install the Python extension, if it's not already installed
Create a file mymodule.py with Python code
To set a breakpoint, hover over a line number and click the red dot, or press F9
Hit F5 to start debugging and select Python File
It will stop at the breakpoint and you can do your usual debugging stuff like inspecting the values of variables, either at the tab VARIABLES (usually on the left) or by clicking on Debug Console (usually at the bottom next to your Terminal):
This screenshot shows VSCodium.
More information
Python debugging in VS Code
Getting Started with Python in VS Code
Debugging in Visual Studio Code
There exist breakpoint() method nowadays, which replaces import pdb; pdb.set_trace().
It also has several new features, such as possible environment variables.
Python Tutor is an online single-step debugger meant for novices. You can put in code on the edit page then click "Visualize Execution" to start it running.
Among other things, it supports:
hiding variables, e.g. to hide a variable named x, put this at the end:
#pythontutor_hide: x
saving/sharing
a few other languages like Java, JS, Ruby, C, C++
However it also doesn't support a lot of things, for example:
Reading/writing files - use io.StringIO and io.BytesIO instead: demo
Code that is too large, runs too long, or defines too many variables or objects
Command-line arguments
Lots of standard library modules like argparse, csv, enum, html, os, sys, weakref...
Python 3.7+
Let's take look at what breakpoint() can do for you in 3.7+.
I have installed ipdb and pdbpp, which are both enhanced debuggers, via
pip install pdbpp
pip install ipdb
My test script, really doesn't do much, just calls breakpoint().
#test_188_breakpoint.py
myvars=dict(foo="bar")
print("before breakpoint()")
breakpoint() # 👈
print(f"after breakpoint myvars={myvars}")
breakpoint() is linked to the PYTHONBREAKPOINT environment variable.
CASE 1: disabling breakpoint()
You can set the variable via bash as usual
export PYTHONBREAKPOINT=0
This turns off breakpoint() where it does nothing (as long as you haven't modified sys.breakpointhook() which is outside of the scope of this answer).
This is what a run of the program looks like:
(venv38) myuser#explore$ export PYTHONBREAKPOINT=0
(venv38) myuser#explore$ python test_188_breakpoint.py
before breakpoint()
after breakpoint myvars={'foo': 'bar'}
(venv38) myuser#explore$
Didn't stop, because I disabled breakpoint. Something that pdb.set_trace() can't do 😀😀😀!
CASE 2: using the default pdb behavior:
Now, let's unset PYTHONBREAKPOINT which puts us back to normal, enabled-breakpoint behavior (it's only disabled when 0 not when empty).
(venv38) myuser#explore$ unset PYTHONBREAKPOINT
(venv38) myuser#explore$ python test_188_breakpoint.py
before breakpoint()
[0] > /Users/myuser/kds2/wk/explore/test_188_breakpoint.py(6)<module>()
-> print(f"after breakpoint myvars={myvars}")
(Pdb++) print("pdbpp replaces pdb because it was installed")
pdbpp replaces pdb because it was installed
(Pdb++) c
after breakpoint myvars={'foo': 'bar'}
It stopped, but I actually got pdbpp because it replaces pdb entirely while installed. If I unistalled pdbpp, I'd be back to normal pdb.
Note: a standard pdb.set_trace() would still get me pdbpp
CASE 3: calling a custom debugger
But let's call ipdb instead. This time, instead of setting the environment variable, we can use bash to set it only for this one command.
(venv38) myuser#explore$ PYTHONBREAKPOINT=ipdb.set_trace py test_188_breakpoint.py
before breakpoint()
> /Users/myuser/kds2/wk/explore/test_188_breakpoint.py(6)<module>()
5 breakpoint()
----> 6 print(f"after breakpoint myvars={myvars}")
7
ipdb> print("and now I invoked ipdb instead")
and now I invoked ipdb instead
ipdb> c
after breakpoint myvars={'foo': 'bar'}
Essentially, what it does, when looking at $PYTHONBREAKPOINT:
from ipdb import set_trace # function imported on the right-most `.`
set_trace()
Again, much cleverer than a plain old pdb.set_trace() 😀😀😀
in practice? I'd probably settle on a debugger.
Say I want ipdb always, I would:
export it via .profile or similar.
disable on a command by command basis, without modifying the normal value
Example (pytest and debuggers often make for unhappy couples):
(venv38) myuser#explore$ export PYTHONBREAKPOINT=ipdb.set_trace
(venv38) myuser#explore$ echo $PYTHONBREAKPOINT
ipdb.set_trace
(venv38) myuser#explore$ PYTHONBREAKPOINT=0 pytest test_188_breakpoint.py
=================================== test session starts ====================================
platform darwin -- Python 3.8.6, pytest-5.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/myuser/kds2/wk/explore
plugins: celery-4.4.7, cov-2.10.0
collected 0 items
================================== no tests ran in 0.03s ===================================
(venv38) myuser#explore$ echo $PYTHONBREAKPOINT
ipdb.set_trace
p.s.
I'm using bash under macos, any posix shell will behave substantially the same. Windows, either powershell or DOS, may have different capabilities, especially around PYTHONBREAKPOINT=<some value> <some command> to set a environment variable only for one command.
If you come from Java/C# background I guess your best bet would be to use Eclipse with Pydev. This gives you a fully functional IDE with debugger built in. I use it with django as well.
https://wiki.python.org/moin/PythonDebuggingTools
pudb is a good drop-in replacement for pdb
PyCharm is an IDE for Python that includes a debugger. Watch this YouTube video for an introduction on using it to step through code:
PyCharm Tutorial - Debug python code using PyCharm (the debugging starts at 6:34)
Note: PyCharm is a commercial product, but the company does provide a free license to students and teachers, as well as a "lightweight" Community version that is free and open-source.
If you want an IDE with integrated debugger, try PyScripter.
Programmatically stepping and tracing through python code is possible too (and its easy!). Look at the sys.settrace() documentation for more details. Also here is a tutorial to get you started.
Visual Studio with PTVS could be an option for you: http://www.hanselman.com/blog/OneOfMicrosoftsBestKeptSecretsPythonToolsForVisualStudioPTVS.aspx