We Keep Coding

Debugging fails when reading input from file [duplicate]

I am developing FUSE filesystem with python. The problem is that after mounting a filesystem I have no access to stdin/stdout/stderr from my fuse script. I don't see anything, even tracebacks. I am trying to launch pdb like this:
import pdb
pdb.Pdb(None, open('pdb.in', 'r'), open('pdb.out', 'w')).set_trace()
All works fine but very inconvenient. I want to make pdb.in and pdb.out as fifo files but don't know how to connect it correctly. Ideally I want to type commands and see output in one terminal, but will be happy even with two terminals (in one put commands and see output in another). Questions:
1) Is it better/other way to run pdb without stdin/stdout?
2) How can I redirect stdin to pdb.in fifo (All what I type must go to pdb.in)? How can I redirect pdb.out to stdout (I had strange errors with "cat pdb.out" but maybe I don't understand something)

Ok. Exactly what I want, has been done in http://pypi.python.org/pypi/rpdb/0.1.1 .

Before starting the python app
mkfifo pdb.in
mkfifo pdb.out
Then when pdb is called you can interact with it using these two cat commands, one running in the background
cat pdb.out & cat > pdb.in
Note the readline support does not work (i.e. up arrow)

I just ran into a similar issue in a much simpler use-case:
debug a simple Python program running from the command line that had a file piped into sys.stdin, meaning, no way to use the console for pdb.
I ended up solving it by using wdb.
Quick rundown for my use-case. In the shell, install both the wdb server and the wdb client:
pip install wdb.server wdb
Now launch the wdb server with:
wdb.server.py
Now you can navigate to localhost:1984 with your browser and see an interface listing all Python programs running. The wdb project page above has instructions on what you can do if you want to debug any of these running programs.
As for a program under your control, you can you can debug it from the start with:
wdb myscript.py --script=args < and/stdin/redirection
Or, in your code, you can do:
import wdb; wdb.set_trace()
This will pop up an interface in your browser (if local) showing the traced program.
Or you can navigate to the wdb.server.py port to see all ongoing debugging sessions on top of the list of running Python programs, which you can then use to access the specific debugging session you want.
Notice that the commands for navigating the code during the trace are different from the standard pdb ones, for example, to step into a function you use .s instead of s and to step over use .n instead of n. See the wdb README in the link above for details.

Python pdb on python script run as package

I have a python program that I usually run as a part of a package:
python -m mymod.client
in order to deal with relative imports inside "mymod/client.py." How do I run this with pdb - the python debugger. The following does not work:
python -m pdb mymod.client
It yields the error:
Error: mymod.client does not exist
EDIT #1 (to address possible duplicity of question)
My question isn't really about running two modules simultaneously python, rather it is about how to use pdb on a python script that has relative imports inside it and which one usually deals with by running the script with "python -m."
Restated, my question could then be, how do I use pdb on such a script while not having to change the script itself just to have it run with pdb (ie: preserving the relative imports inside the script as much as possible). Shouldn't this be possible, or am I forced to refactor in some way if I want to use pdb? If so what would be the minimal changes to the structure of the script that I'd have to introduce to allow me to leverage pdb.
In summary, I don't care how I run the script, just so long as I can get it working with pdb without changing it's internal structure (relative imports, etc) too much.

I think I have a solution.
Run it like this:
python -m pdb path/mymod/client.py arg1 arg2
that will run it as a script, but will not treat it as a package.
At the top of client.py, the first line should be:
import mymod
That will get the package itself loaded.
I am still playing with this, but it seems to work so far.

This is not possible. Though unstated in documentation, Python will not parse two modules via the -m command line option.

ipython -pylab debugging: Can I stop the execution at a specific line and drop into the shell?

When writing python code (mostly numpy + matplotlib), I usually just type the code in vim and run the program to test it:
python2 foo.py
Occasionally, when this is not sufficient and I need to inspect the problem more thoroughly, I just launch the program in ipython:
ipython -pylab foo.py, and then inspect the variables, test some commands and so on. I like ipython, because of the tab completion and the availability of bash commands.
This worked well enough for me, but now my programs grew bigger and include many subroutines (in multiple files). The ipython approach doesn't work any more, because it always runs the complete code till the end of foo.py (when it drops into the pylab shell). What I'd like to do instead is, stop execution at a given line in a subroutine (could be in another file) and inspect variables there. I.e. set a break point at which the pylab shell kicks in.
Is there an easy way to adapt my ipython way of working? E.g. stop at a line in bar.py
ipython -pylab --stop-at bar.py:423 foo.py
or, stop at a subroutine name in bar.py
ipython -pylab --stop-at bar.py:subroutine-name foo.py

You can import the pdb module into the code and then add a pdb.set_trace() call where you would like to have the code stop. Ipython will drop into the interactive debugger, and you are free to step though your code as you wish.

You can drop into a IPython debug session by inserting the following code at the desired point:
import sys, IPython
IPython.Shell.IPShell(argv=[])
IPython.Debugger.Pdb(IPython.ipapi.get().options.colors).set_trace(sys._getframe())
Not exactly what you seem to be looking for, but it works quite nicely for me. It also makes it really easy to have complex conditional breakpoints. There are several other methods of starting an IPython debug session from within your source file floating around on the web, but this - in my experience anyway - is the most reliable in terms of loading the correct colours, tab completion working properly etc.
Once the debug session has started you can set further breakpoints using the break command:
ipdb> break test.py:11
Breakpoint 1 at /tmp/test.py:11
ipdb> b my_function
Breakpoint 2 at /tmp/test.py:5
To make it easy to insert, you can set a macro/key combination in your editor. I'm also a Vim user and I have the following keymap in my vimrc:
nmap <C-P><C-D> oimport sys, IPython<CR>IPython.Shell.IPShell(argv=[])<CR>IPython.Debugger.Pdb(IPython.ipapi.get().options.colors).set_trace(sys._getframe())<ESC>:w<CR>
From normal mode, pressing Ctrl-P then Ctrl-D inserts the debug code after the current line with the correct indentation and then saves the file.

How to step through Python code to help debug issues?

In Java/C# you can easily step through code to trace what might be going wrong, and IDE's make this process very user friendly.
Can you trace through python code in a similar fashion?

Yes! There's a Python debugger called pdb just for doing that!
You can launch a Python program through pdb by using pdb myscript.py or python -m pdb myscript.py.
There are a few commands you can then issue, which are documented on the pdb page.
Some useful ones to remember are:
b: set a breakpoint
c: continue debugging until you hit a breakpoint
s: step through the code
n: to go to next line of code
l: list source code for the current file (default: 11 lines including the line being executed)
u: navigate up a stack frame
d: navigate down a stack frame
p: to print the value of an expression in the current context
If you don't want to use a command line debugger, some IDEs like Pydev, Wing IDE or PyCharm have a GUI debugger. Wing and PyCharm are commercial products, but Wing has a free "Personal" edition, and PyCharm has a free community edition.

By using Python Interactive Debugger 'pdb'
First step is to make the Python interpreter to enter into the debugging mode.
A. From the Command Line
Most straight forward way, running from command line, of python interpreter
$ python -m pdb scriptName.py
> .../pdb_script.py(7)<module>()
-> """
(Pdb)
B. Within the Interpreter
While developing early versions of modules and to experiment it more iteratively.
$ python
Python 2.7 (r27:82508, Jul 3 2010, 21:12:11)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb_script
>>> import pdb
>>> pdb.run('pdb_script.MyObj(5).go()')
> <string>(1)<module>()
(Pdb)
C. From Within Your Program
For a big project and long-running module, can start the debugging from inside the program using
import pdb and set_trace()
like this :
#!/usr/bin/env python
# encoding: utf-8
#
import pdb
class MyObj(object):
count = 5
def __init__(self):
self.count= 9
def go(self):
for i in range(self.count):
pdb.set_trace()
print i
return
if __name__ == '__main__':
MyObj(5).go()
Step-by-Step debugging to go into more internal
Execute the next statement… with “n” (next)
Repeating the last debugging command… with ENTER
Quitting it all… with “q” (quit)
Printing the value of variables… with “p” (print)
a) p a
Turning off the (Pdb) prompt… with “c” (continue)
Seeing where you are… with “l” (list)
Stepping into subroutines… with “s” (step into)
Continuing… but just to the end of the current subroutine… with “r” (return)
Assign a new value
a) !b = "B"
Set a breakpoint
a) break linenumber
b) break functionname
c) break filename:linenumber
Temporary breakpoint
a) tbreak linenumber
Conditional breakpoint
a) break linenumber, condition
Note:**All these commands should be execute from **pdb
For in-depth knowledge, refer:-
https://pymotw.com/2/pdb/
https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/

There is a module called 'pdb' in python. At the top of your python script you do
import pdb
pdb.set_trace()
and you will enter into debugging mode. You can use 's' to step, 'n' to follow next line similar to what you would do with 'gdb' debugger.

Starting in Python 3.7, you can use the breakpoint() built-in function to enter the debugger:
foo()
breakpoint() # drop into the debugger at this point
bar()
By default, breakpoint() will import pdb and call pdb.set_trace(). However, you can control debugging behavior via sys.breakpointhook() and use of the environment variable PYTHONBREAKPOINT.
See PEP 553 for more information.

ipdb (IPython debugger)
ipdb adds IPython functionality to pdb, offering the following HUGE improvements:
tab completion
show more context lines
syntax highlight
Much like pdg, ipdb is still far from perfect and completely rudimentary if compared to GDB, but it is already a huge improvement over pdb.
Usage is analogous to pdb, just install it with:
python3 -m pip install --user ipdb
and then add to the line you want to step debug from:
__import__('ipdb').set_trace(context=21)
You likely want to add a shortcut for that from your editor, e.g. for Vim snipmate I have:
snippet ipd
__import__('ipdb').set_trace(context=21)
so I can type just ipd<tab> and it expands to the breakpoint. Then removing it is easy with dd since everything is contained in a single line.
context=21 increases the number of context lines as explained at: How can I make ipdb show more lines of context while debugging?
Alternatively, you can also debug programs from the start with:
ipdb3 main.py
but you generally don't want to do that because:
you would have to go through all function and class definitions as Python reads those lines
I don't know how to set the context size there without hacking ipdb. Patch to allow it: https://github.com/gotcha/ipdb/pull/155
Or alternatively, as in raw pdb 3.2+ you can set some breakpoints from the command line:
ipdb3 -c 'b 12' -c 'b myfunc' ~/test/a.py
although -c c is broken for some reason: https://github.com/gotcha/ipdb/issues/156
python -m module debugging has been asked at: How to debug a Python module run with python -m from the command line? and since Python 3.7 can be done with:
python -m pdb -m my_module
Serious missing features of both pdb and ipdb compared to GDB:
persistent command history across sessions: Save command history in pdb
ipdb specific annoyances:
multithreading does not work well if you don't hack some settings...
ipdb, multiple threads and autoreloading programs causing ProgrammingError
https://github.com/gotcha/ipdb/issues/51
Tested in Ubuntu 16.04, ipdb==0.11, Python 3.5.2.

VSCode
If you want to use an IDE, this is a good alternative to PyCharm.
Install VSCode
Install the Python extension, if it's not already installed
Create a file mymodule.py with Python code
To set a breakpoint, hover over a line number and click the red dot, or press F9
Hit F5 to start debugging and select Python File
It will stop at the breakpoint and you can do your usual debugging stuff like inspecting the values of variables, either at the tab VARIABLES (usually on the left) or by clicking on Debug Console (usually at the bottom next to your Terminal):
This screenshot shows VSCodium.
More information
Python debugging in VS Code
Getting Started with Python in VS Code
Debugging in Visual Studio Code

There exist breakpoint() method nowadays, which replaces import pdb; pdb.set_trace().
It also has several new features, such as possible environment variables.

Python Tutor is an online single-step debugger meant for novices. You can put in code on the edit page then click "Visualize Execution" to start it running.
Among other things, it supports:
hiding variables, e.g. to hide a variable named x, put this at the end:
#pythontutor_hide: x
saving/sharing
a few other languages like Java, JS, Ruby, C, C++
However it also doesn't support a lot of things, for example:
Reading/writing files - use io.StringIO and io.BytesIO instead: demo
Code that is too large, runs too long, or defines too many variables or objects
Command-line arguments
Lots of standard library modules like argparse, csv, enum, html, os, sys, weakref...
Python 3.7+

Let's take look at what breakpoint() can do for you in 3.7+.
I have installed ipdb and pdbpp, which are both enhanced debuggers, via
pip install pdbpp
pip install ipdb
My test script, really doesn't do much, just calls breakpoint().
#test_188_breakpoint.py
myvars=dict(foo="bar")
print("before breakpoint()")
breakpoint() # 👈
print(f"after breakpoint myvars={myvars}")
breakpoint() is linked to the PYTHONBREAKPOINT environment variable.
CASE 1: disabling breakpoint()
You can set the variable via bash as usual
export PYTHONBREAKPOINT=0
This turns off breakpoint() where it does nothing (as long as you haven't modified sys.breakpointhook() which is outside of the scope of this answer).
This is what a run of the program looks like:
(venv38) myuser#explore$ export PYTHONBREAKPOINT=0
(venv38) myuser#explore$ python test_188_breakpoint.py
before breakpoint()
after breakpoint myvars={'foo': 'bar'}
(venv38) myuser#explore$
Didn't stop, because I disabled breakpoint. Something that pdb.set_trace() can't do 😀😀😀!
CASE 2: using the default pdb behavior:
Now, let's unset PYTHONBREAKPOINT which puts us back to normal, enabled-breakpoint behavior (it's only disabled when 0 not when empty).
(venv38) myuser#explore$ unset PYTHONBREAKPOINT
(venv38) myuser#explore$ python test_188_breakpoint.py
before breakpoint()
[0] > /Users/myuser/kds2/wk/explore/test_188_breakpoint.py(6)<module>()
-> print(f"after breakpoint myvars={myvars}")
(Pdb++) print("pdbpp replaces pdb because it was installed")
pdbpp replaces pdb because it was installed
(Pdb++) c
after breakpoint myvars={'foo': 'bar'}
It stopped, but I actually got pdbpp because it replaces pdb entirely while installed. If I unistalled pdbpp, I'd be back to normal pdb.
Note: a standard pdb.set_trace() would still get me pdbpp
CASE 3: calling a custom debugger
But let's call ipdb instead. This time, instead of setting the environment variable, we can use bash to set it only for this one command.
(venv38) myuser#explore$ PYTHONBREAKPOINT=ipdb.set_trace py test_188_breakpoint.py
before breakpoint()
> /Users/myuser/kds2/wk/explore/test_188_breakpoint.py(6)<module>()
5 breakpoint()
----> 6 print(f"after breakpoint myvars={myvars}")
7
ipdb> print("and now I invoked ipdb instead")
and now I invoked ipdb instead
ipdb> c
after breakpoint myvars={'foo': 'bar'}
Essentially, what it does, when looking at $PYTHONBREAKPOINT:
from ipdb import set_trace # function imported on the right-most `.`
set_trace()
Again, much cleverer than a plain old pdb.set_trace() 😀😀😀
in practice? I'd probably settle on a debugger.
Say I want ipdb always, I would:
export it via .profile or similar.
disable on a command by command basis, without modifying the normal value
Example (pytest and debuggers often make for unhappy couples):
(venv38) myuser#explore$ export PYTHONBREAKPOINT=ipdb.set_trace
(venv38) myuser#explore$ echo $PYTHONBREAKPOINT
ipdb.set_trace
(venv38) myuser#explore$ PYTHONBREAKPOINT=0 pytest test_188_breakpoint.py
=================================== test session starts ====================================
platform darwin -- Python 3.8.6, pytest-5.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/myuser/kds2/wk/explore
plugins: celery-4.4.7, cov-2.10.0
collected 0 items
================================== no tests ran in 0.03s ===================================
(venv38) myuser#explore$ echo $PYTHONBREAKPOINT
ipdb.set_trace
p.s.
I'm using bash under macos, any posix shell will behave substantially the same. Windows, either powershell or DOS, may have different capabilities, especially around PYTHONBREAKPOINT=<some value> <some command> to set a environment variable only for one command.

If you come from Java/C# background I guess your best bet would be to use Eclipse with Pydev. This gives you a fully functional IDE with debugger built in. I use it with django as well.

https://wiki.python.org/moin/PythonDebuggingTools
pudb is a good drop-in replacement for pdb

PyCharm is an IDE for Python that includes a debugger. Watch this YouTube video for an introduction on using it to step through code:
PyCharm Tutorial - Debug python code using PyCharm (the debugging starts at 6:34)
Note: PyCharm is a commercial product, but the company does provide a free license to students and teachers, as well as a "lightweight" Community version that is free and open-source.

If you want an IDE with integrated debugger, try PyScripter.

Programmatically stepping and tracing through python code is possible too (and its easy!). Look at the sys.settrace() documentation for more details. Also here is a tutorial to get you started.

Visual Studio with PTVS could be an option for you: http://www.hanselman.com/blog/OneOfMicrosoftsBestKeptSecretsPythonToolsForVisualStudioPTVS.aspx

vim and python scripts debugging

Are there any ways to debug python scripts not leaving vim in *nix systems (executing the script, setting up breakpoints, showing variables in watch-list, etc)?

Use pdb:
import pdb
def main():
list = [1,2,3]
pdb.set_trace()
list = [2,3,4]
if __name__ == '__main__':
main()
Now run using :!python % and you'll hit your breakpoint and be able to debug interactively like in gdb.

As of Python 3.7, you can use breakpoint() builtin without importing anything.
Built-in breakpoint() calls sys.breakpointhook(). By default, the latter imports pdb and then calls pdb.set_trace()
Inheriting code from Pierre-Antoine's answer, the code would look like this:
def main():
list = [1,2,3]
breakpoint()
list = [2,3,4]
if __name__ == '__main__':
main()
Source: https://docs.python.org/3/whatsnew/3.7.html#pep-553-built-in-breakpoint

Try pyclewn. It allows to use vim as front end for pdb. You can create/delete break points, control flow of debugging process, look at values of your variables. All from vim!

Also try https://pypi.python.org/pypi/pudb - its like pdb but more advanced. Contains code highlighting, stack, showing avaliable values, etc. Not only-vim solution but for me works perfectly.
Three Steps:
Install:
pip install pudb
Paste set_trace in code
from pudb import set_trace; set_trace()
Run your code

As 2020 the Debugger Adapter Protocol is taken care by vimspector.
Supporting Cpp, Python, Java, Js, Go ...
See my other answer

The vimpdb plugin integrates the Python debugger pdb into the VIM editor.
I do recommend it.
Hope it helps.

Vim and pdb-clone is the combination I use. I use Home - pyclewn which provides a replacement for pdb called pdb-clone that is quite faster than vanilla pdb. It integrates well with vim via a plugin, and the thing I appreciate most is that it takes care of breakpoints outside the code, not setting traces within, thus not messing up my line numbers. It does not have a watch window for python yet. You might have a look at vim-debug too, which I could not get to work with my existing highlighting setup.

See the "Debugging" section in this blog post. It shows how to setup F7 to set breakpoints and Shift+F7 to remove breakpoints. It also uses pdb, as mentioned before. With a little modification, you can replace the use of pdb with ipdb (pdb using ipython), which is a lot nicer to use.

It sounds like you want to use VIM as a Python IDE.
A quick Google search found this and this example, with many more.
EDIT: Well, Ok, it seems likely you've searched more than I have.
I hope someone else has some ideas.

From what I know, there is one more option: You could use Eclipse + PyDev for project managing and Vim as an editor for Eclipse. That way You could use the best of both worlds.
Also, I haven't tried it, but You could try this script.