I am trying to find a way that I can have a program step through Python code line by line and do something with the results of each line. In effect a debugger that could be controlled programmatically rather than manually. pdb would be exactly what I am looking for if it returned its output after each step as a string and I could then call pdb again to pickup where I left off. However, instead it outputs to stdout and I have to manually input "step" via the keyboard.
Things I have tried:
I am able to redirect pdb's stdout. I could redirect it to a second
Python program which would then process it. However, I cannot
figure out how to have the second Python program tell pdb to
step.
Related to the previous one, if I could get pdb to step all the way
through to the end (perhaps I could figure out something to spoof a
keyboard repeatedly entering "step"?) and redirect the output to a
file, I could then write another program that acted like it was
stepping through the program when it was actually just reading the
file line by line.
I could use exec to manually run lines of Python code. However,
since I would be looking at one line at a time, I would need to
manually detect and handle things like conditionals, loops, and
function calls which quickly gets very complicated.
I read some posts that say that pdb is implemented using
sys.settrace. If nothing else works I should be able to recreate
the behavior I need using this.
Is there any established/straight forward way to implement the behavior that I am looking for?
sys.settrace() is the fundamental building block for stepping through Python code. pdb is implemented entirely in Python, so you can just look at the module to see how it does things. It also has various public functions/methods for stepping under program control, read the library reference for your version of Python for details.
I read some posts that say that pdb is implemented using sys.settrace.
If nothing else works I should be able to recreate the behavior I need
using this.
Don't view this as a last resort. I think it's the best approach for what you want to accomplish.
Related
Being in the process of switching from Matlab to Python/IPython/Spyder I ran into the following issue.
In a Matlab script (and I actually did not notice this until I switched to Python) there is no warning if you are using a variable which you did not define in that script.
What I kept doing all the times in Matlab was to have, say, 2 (or more) scripts which I would then execute one after the other. The second script would typically use variables defined in the first, but would not give me any warning.
In Spyder I noticed that the situation is different. In my hypothetical second script, all the variables which are not defined in the second script itself give me a warning (undefined name '...'), which is not nice too see...
Another typical example, would be the following: I have a main script but there is something in this script I wanna look at better. So, without touching the script, I would create another file where I would copy paste a few lines from the script to play a little with them.
In this new file I would be using variables which are in the console already but that are techically unknown to the file itself, so would give me a warning.
So I guess I am simply asking if there a way to suppress this kind of warnings.
Or there might be a deeper question of whether there s a more pythonic way of working, but there I am really not sure about...
I am new to Python, have some experience in MatLab and r. My question is: Is it possible to run part of the code in .py block by block (line by line)?
In r or Matlab, I can first have some data and variables loaded in the memory first. Then experimentally I can run a line or two to try out the syntax... this is particularly useful for new learners I believe. I know there is something called the iPython which can execute Python code line by line however this is not what I am after. Thanks.
Since ipython has already been discounted, I'm not sure this answer will be better. But I will tell you the two things that I do.
I drop into the debugger at the point where I want to "try out" something, so the code will run up to that point, and then drop me into the debugger. You do this simply by inserting this code at that point:
import pdb; pdb.set_trace()
Once you've done what needs to be done, you can either press q to quit, or c to continue running the process.
I use the -i option to python. This enters interactive mode at the end of your python code. This is useful if you want to set up a bunch of data structures, and try out some code on it, instead of typing all of it into a python shell first. (that might be why you rejected ipython?)
I think what you need is a debugger.
You can use the pydev plugin for Eclipse which has a debugger.
Another option is pdb as already suggested but it's not very easy to use.
Lets say you have a large file containing LETTER,NUMBER comma-delimited tokens. You want to write a program that reads from standard input and prints out NUMBER+1 for each line. Very trivial program, I understand. However, here is the constraint -- you can only read from this standard in pipe one-time AND you have to start out with programming an empty file.
So for example:
cat FILE.csv | python empty_program.py
This should pop up an interactive session which allows you to write what ever code you want. Since empty_program.py has not called stdin.readline(), the stdin buffer is appropriately in-tact.
Is something like this possible?
One example of something that can sort of do this is the Excel VBA debugger/IDE. It allows you to pause execution -- add new lines to the programs source code and continue exeuction.
cat FILE.csv | python empty_program.py
Well, python will try to read "empty_program.py", and fail to find anything in it, assuming there is file, and then exit. If the file doesn't exist, you get an error. I tested it [you should have been able to do that as well, doesn't take that much effort - probably a lot less than it took to go to SO and write the question].
So, my next thought was to use an interactive python process, but since you are feeding things through stdin, it won't work - I didn't have a good csv file, so I did "cat somefile.c|python", and that falls over at "int main()" with "invalid syntax". I'm surprised it got as far as that, but I guess that's because #include's are seen as comments.
Most interactive programming languages read from stdin, so you can't really do what you are describing with any of them.
I'm far from sure why you'd want to either. If your first program can produce the relevant program code, why would you not just put it in a file and let python read that file... Rather than jump through hoops? Note that an IDE is not the same as a command line program. I'm pretty sure that if you work hard enough at something, you can write a C program that acesses the Eclipse IDE with Python plugins. But that's really doing things the hard way. Why anyone would WANT to spend that much effort to achieve so little, I don't see.
Sorry, but I don't really see the point of what you are trying to do - I'm sure you have some good idea in there, but I'm sure the implementation details need to be worked on.
I'm experimenting with debugging my python from the raw pdb program rather than running pdb though emacs (which tracks the current line with a marker in the text display of the code). It's slightly annoying that the list command l in pdb only displays a few lines of code, I would rather have it fill my terminal with all the code of the current function up to the current line.
I know I can do this manually by looking at the line numbers are typing l 50,100 (where 100 is the current line) but this is time consuming and I'd like to set this up to work automatically.
I wonder if there is a way to define a pdb command to do this? I'm guessing it would need to (1) access the number of the current line, N; then (2) execute "l N-50, N". I've searched around a lot but can't find anyone who has done this before. Perhaps there is a way to access the pdb module's own internals to get the line number?
(Or a roundabout way would be to write something that calls list once, parses the output to extract the current line, then executes a new list command, I wonder if anyone has done this already? Is this how IDEs manage to get the current line information from pdb or are they using its internals I wonder?)
You could probably extend the PDB class, but the extension interface isn't documented very well.
I'd instead recommend using pdb++ and its sticky feature which pretty much does what you want, if I understood your use case correctly.
I'm debugging an script that I'm writing and the result of
executing a statement from pdb does not make sense so my
natural reaction is to try to trace it with pdb.
To paraphrase:
Yo dawg, I like python, so can you put my pdb in my pdb so I can debug while I debug?
It sounds like you're looking for something listed fairly prominently in the docs, which is the set of methods that let you programmatically invoke the debugger on expressions, code in strings, or functions:
http://docs.python.org/library/pdb.html#pdb.run
http://docs.python.org/library/pdb.html#pdb.runeval
http://docs.python.org/library/pdb.html#pdb.runcall
I use these when I'm already at the pdb prompt (generally having gotten there by encountering a well-placed pdb.set_trace() statement) and want to test out, for example, variations on some method calls that aren't called in my source but which I can call right in the current context, manually.
If that's not what you were looking for, do you simply want the "step" command instead of the "next" command at the prompt? (It's unclear what you really want here. An example might help.)