This question already has answers here:
Python REPL tab completion on MacOS
(2 answers)
Closed 7 years ago.
I recently switched from Matlab to Numpy and love it. However, one really great thing I liked about Matlab was the ability to complete commands. There are two ways that it does this:
1) tab completion. If I have a function called foobar(...), I can do 'fo' and it will automatically fill in 'foobar'
2) "up-button" completion (I'm not sure what to call this). If I recently entered a command such as
'x = linspace(0, 1, 100); A = eye(50);'
and then I wish to quickly type in this same command so that I can re-evaluate it or change it slightly, then I simply type 'x =' then press up and it will cycle through all previous commands you typed that started with 'x ='. This was an awesome awesome feature in Matlab (and if you have heard of Julia, it has done it even better by allowing you to automatically re-enter entire blocks of code, such as when you are defining functions at the interactive prompt)
Both of these features appear to not be present in the ordinary python interactive shell. I believe tab autocomplete has been discussed before and can probably be enabled using the .pythonrc startup script and some modules; however I have not found anything about "up-button" completion. Python does have rudimentary up-button functionality that simply scrolls through all previous commands, but you can't type in the beginning of the command and have that narrow down the range of commands that are scrolled through, and that makes a huge difference.
Anyone know any way to get this functionality on the ordinary python interactive shell, without going to any fancy things like IPython notebooks that require separate installation?
Tab completion is not a standard feature of the python 2.x interpreter. It is possible that a particular distribution (intending, Linux distribution) ships with initialization files that enable tab completion. On the other hand, python 3.x has autocompletion enabled by default.
To enable tab completion in 2.x, you need to instruct the interpreter about loading some startup code, using an environment variable
export PYTHONSTARTUP=$HOME/.whatever
The code that you want to put into the startup file varies, but for enabling tab completion the docs have
try:
import readline
except ImportError:
print "Module readline not available."
else:
import rlcompleter
readline.parse_and_bind("tab: complete")
Coming eventually to your ast question, what you named “up-button” command completion, matlab/julia style, IPython has it and I'm not aware of a module that implements it, even if it seems to me that I read something on such a beast on comp.lang.python some month ago.
In your question you reference IPython's notebook... It may be necessary to remind that you don't need the notebook interface to use IPython, it can be used to its full potential even in a text console.
Use iPython or some other Python shell. There are plenty. You may even program your own that will do whatever you want.
tab completion. If I have a function called foobar(...), I can do 'fo' and it will automatically fill in 'foobar'
Really? Which version of Python are you using and how did you get it? It works for me on a regular python shell for both windows and Linux with both python 2.7 and python 3.4. It sounds like your version of Python might not have been built with readline support, which I think is required for this sort of thing.
This is what I tried:
>>> sup
after tab becomes:
>>> super(
"up-button" completion (I'm not sure what to call this). If I recently entered a command such as 'x = linspace(0, 1, 100); A = eye(50);' and then I wish to quickly type in this same command so that I can re-evaluate it or change it slightly, then I simply type 'x =' then press up and it will cycle through all previous commands you typed that started with 'x ='.
It is called a "History search", and it also works for me in the default Python shell in both windows and Linux. Again, I think this requires readline.
>>> a = 'test'
>>> a
Then I press up, and I get:
>>> a = 'test'
You can also press Ctrl+r, then start typing. This will search the history for the last command that includes what you typed. So, for example:
>>> a = 'test'
>>> b = 5
>>> c = a
Then ctrl+r:
>>>
forward-i-search1`b: b = 5
Then hit Enter to execute that command.
>>>
>>> b = 5
>>>
If the match isn't what you want, you can type more, or hit Ctrl+r again and again to cycle through the matches.
Edit:
It looks like this is a known problem with the built-in Mac Os X version of Python. It doesn't come with readline due to readline being GPL. Instead it includes libedit, which is not fully compatible. There are instructions on how to get it working on Mac Os X here
I'm using the QUuid class in my project and for testing and debugging purposes it would be very nice to see the QUuid objects in human readable form instead of their low-level form.
For some reason, the people at Qt have not included a dump method for this type so I attempted to create one on my own, following this documentation and this guide.
I'm not familiar with Python so unfortunately, I could not get something running. Could someone help me create such a function that does nothing more than display the output of QUuid::toString() in the value column of Qt Creator?
Edit:
Mitko's solution worked perfectly. I expanded it a bit so the details can still be read if so desired:
from dumper import *
import gdb
def qdump__QUuid(d, value):
this_ = d.makeExpression(value)
finalValue = gdb.parse_and_eval("%s.toString()" % (this_))
d.putStringValue(finalValue)
d.putNumChild(4)
if d.isExpanded():
with Children(d):
d.putSubItem("data1", value["data1"])
d.putSubItem("data2", value["data2"])
d.putSubItem("data3", value["data3"])
d.putSubItem("data4", value["data4"])
The following python script should do the job:
from dumper import *
import gdb
def qdump__QUuid(d, value):
this = d.makeExpression(value)
stringValue = gdb.parse_and_eval("%s.toString()" % this)
d.putStringValue(stringValue)
d.putNumChild(0)
The easiest way to use it with Qt Creator is to just paste these lines at the end of your <Qt-Creator-Install-Dir>/share/qtcreator/debugger/personaltypes.py file. In this case you can skip the first line, as it's already in the file.
As the personaltypes.py file is overwritten when you update Qt Creator you might want to put the script above in its own file. In that case you'll need to configure Qt Creator to use your file. You can do this by going to Tools > Options... > Debugger > GDB > Extra Debugging Helpers > Browse and selecting your file.
Note:
This script will only work inside Qt Creator, since we use its specific dumper (e.g. putStringValue).
We call QUuid::toString() which creates a QString object. I'm not sure exactly how gdb and python handle this, and if there is a need to clean this up in order to avoid leaking memory. It's probably not a big deal for debugging, but something to be aware of.
I'm trying to add bespoke suggestions to the interactive python tab auto complete. I found this toy example on the interweb
import readline, rlcompleter
addrs = ['angela#domain.com', 'michael#domain.com', 'david#test.com']
class mycompleter(rlcompleter.Completer):
def completer(self, text, state):
options = [x for x in addrs if x.startswith(text)]
try:
return options[state]
except IndexError:
return None
readline.set_completer(mycompleter().completer)
readline.parse_and_bind("tab: complete")
This works very nicely in python if I save it in a module and them import it. It also works in IPython if I paste it into an active session using the %paste magic.
However, I can't get it to work in an IPython Notebook, either by loading a module or by running it in a cell. I've found the ipython docs about their extension to the readline module but this hasn't helped. I've tried inheriting from IPCompleter objects, and using rlcompete methods etc, but this doesn't seem to have helped.
Any suggestions about how to add things to the autocomplete suggestions in a way that works in plain python and IPython Notebook
Thanks
Niall
UPDATE:
Ultimately, I'm looking of a way to add functionality to a module so that it can dynamically update the session autocomplete list (ideally for args for a specific set of functions so that it doesn't pollute the suggestions).
There is a way to do it, which is not the recommended one but works.
def my_matches(test):
# might want to be smarter here
return ['angela#domain.com', 'michael#domain.com', 'david#test.com']
ip = get_ipython()
ip.Completer.matchers.append(my_matches)
# it works
The old ways require setting hooks, but I haven't used it and is pretty old
and could be refactored
Via
import win32com.client
wordapp = win32com.client.gencache.EnsureDispatch('Word.Application')
I can get a Word Application object documented e.g. here. However, ipython's autocompletion is not aware of that API, is there any way to add that?
Quick solution
Perhaps the simplest way to achieve code completion in IPython (tested with 6.2.1, see the answer below for a snippet that works with 7.1) and Jupyter is to run the following snippet:
from IPython.utils.generics import complete_object
import win32com.client
#complete_object.when_type(win32com.client.DispatchBaseClass)
def complete_dispatch_base_class(obj, prev_completions):
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
return list(ole_props) + prev_completions
except AttributeError:
pass
Short story long
With some more details being outlined in this guide, win32com ships with a script, makepy.py for generating Python types corresponding to the type library of a given COM object.
In the case of Word 2016, we would proceed as follows:
C:\Users\username\AppData\Local\Continuum\Anaconda3\pkgs\pywin32-221-py36h9c10281_0\Lib\site-packages\win32com\client>python makepy.py -i "Microsoft Word 16.0 Object Library"
Microsoft Word 16.0 Object Library
{00020905-0000-0000-C000-000000000046}, lcid=0, major=8, minor=7
>>> # Use these commands in Python code to auto generate .py support
>>> from win32com.client import gencache
>>> gencache.EnsureModule('{00020905-0000-0000-C000-000000000046}', 0, 8, 7)
The location of makepy.py will of course depend on your Python distribution. The script combrowse.py, available in the same directory, can be used to find the names of available type libraries.
With that in place, win32com.client will automatically make use of the generated types, rather than the raw IPyDispatch, and at this point, auto-completion is available in e.g. IPython or Jupyter, given that the COM object of interest actually publishes its available properties and methods (which is not a requirement).
Now, in your case, by invoking EnsureDispatch instead of Dispatch, the makepy part of the process is performed automatically, so you really should be able to obtain code completion in IPython for the published methods:
Note, though, that while this does give code completion for methods, the same will not be true for properties. It is possible to inspect those using the _prop_map_get_ attribute. For example, wordapp.Selection.Range.Font._prop_map_get_ gives all properties available on fonts.
If using IPython is not a strong requirement, note also that the PythonWin shell (located around \pkgs\pywin32\Lib\site-packages\pythonwin\Pythonwin.exe) has built-in code completion support for both properties and methods.
This, by itself, suggests that the same is achievable in IPython.
Concretely, the logic for auto-completion, which in turn relies on _prop_map_get_, can be found in scintilla.view.CScintillaView._AutoComplete. On the other hand, code completion in IPython 6.2.1 is handled by core.completer.IPCompleter. The API for adding custom code completers is provided by IPython.utils.generics.complete_object, as illustrated in the first solution above. One gotcha is that with complete_object being based on simplegeneric, only one completer may be provided for any given type. Luckily, all types generated by makepy will inherit from win32com.client.DispatchBaseClass.
If this turns out to ever be an issue, one can also circumvent complete_object entirely and simply manually patch IPython by adding the following five lines to core.completer.Completion.attr_matches:
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
words += list(ole_props)
except AttributeError:
pass
Conversely, IPython bases its code-completion on __dir__, so one could also patch gencache, which is where the code generation ultimately happens, to include something to like
def __dir__(self):
return list(set(self._prop_map_get_).union(set(self._prop_map_put_)))
to each generated DispatchBaseClass.
fuglede's answer is great, just want to update it for the newest versions of IPython (7.1+).
Since IPython.utils.generics has changes from using simplegeneric to using functools, the #complete_object.when_type method should be changed to #complete_object.register. So his initial code should changed to:
from IPython.utils.generics import complete_object
import win32com.client
#complete_object.register(win32com.client.DispatchBaseClass)
def complete_dispatch_base_class(obj, prev_completions):
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
return list(ole_props) + prev_completions
except AttributeError:
pass
From what I have read, there are two ways to debug code in Python:
With a traditional debugger such as pdb or ipdb. This supports commands such as c for continue, n for step-over, s for step-into etc.), but you don't have direct access to an IPython shell which can be extremely useful for object inspection.
Using IPython by embedding an IPython shell in your code. You can do from IPython import embed, and then use embed() in your code. When your program/script hits an embed() statement, you are dropped into an IPython shell. This allows the full inspection of objects and testing of Python code using all the IPython goodies. However, when using embed() you can't step-by-step through the code anymore with handy keyboard shortcuts.
Is there any way to combine the best of both worlds? I.e.
Be able to step-by-step through your code with handy pdb/ipdb keyboard shortcuts.
At any such step (e.g. on a given statement), have access to a full-fledged IPython shell.
IPython debugging as in MATLAB:
An example of this type of "enhanced debugging" can be found in MATLAB, where the user always has full access to the MATLAB engine/shell, and she can still step-by-step through her code, define conditional breakpoints, etc. From what I have discussed with other users, this is the debugging feature that people miss the most when moving from MATLAB to IPython.
IPython debugging in Emacs and other editors:
I don't want to make the question too specific, but I work mostly in Emacs, so I wonder if there is any way to bring this functionality into it. Ideally, Emacs (or the editor) would allow the programmer to set breakpoints anywhere on the code and communicate with the interpreter or debugger to have it stop in the location of your choice, and bring to a full IPython interpreter on that location.
What about ipdb.set_trace() ? In your code :
import ipdb; ipdb.set_trace()
update: now in Python 3.7, we can write breakpoint(). It works the same, but it also obeys to the PYTHONBREAKPOINT environment variable. This feature comes from this PEP.
This allows for full inspection of your code, and you have access to commands such as c (continue), n (execute next line), s (step into the method at point) and so on.
See the ipdb repo and a list of commands. IPython is now called (edit: part of) Jupyter.
ps: note that an ipdb command takes precedence over python code. So in order to write list(foo) you'd need print(list(foo)), or !list(foo) .
Also, if you like the ipython prompt (its emacs and vim modes, history, completions,…) it's easy to get the same for your project since it's based on the python prompt toolkit.
You can use IPython's %pdb magic. Just call %pdb in IPython and when an error occurs, you're automatically dropped to ipdb. While you don't have the stepping immediately, you're in ipdb afterwards.
This makes debugging individual functions easy, as you can just load a file with %load and then run a function. You could force an error with an assert at the right position.
%pdb is a line magic. Call it as %pdb on, %pdb 1, %pdb off or %pdb 0. If called without argument it works as a toggle.
(Update on May 28, 2016) Using RealGUD in Emacs
For anyone in Emacs, this thread shows how to accomplish everything described in the OP (and more) using
a new important debugger in Emacs called RealGUD which can operate with any debugger (including ipdb).
The Emacs package isend-mode.
The combination of these two packages is extremely powerful and allows one to recreate exactly the behavior described in the OP and do even more.
More info on the wiki article of RealGUD for ipdb.
Original answer:
After having tried many different methods for debugging Python, including everything mentioned in this thread, one of my preferred ways of debugging Python with IPython is with embedded shells.
Defining a custom embedded IPython shell:
Add the following on a script to your PYTHONPATH, so that the method ipsh() becomes available.
import inspect
# First import the embed function
from IPython.terminal.embed import InteractiveShellEmbed
from IPython.config.loader import Config
# Configure the prompt so that I know I am in a nested (embedded) shell
cfg = Config()
prompt_config = cfg.PromptManager
prompt_config.in_template = 'N.In <\\#>: '
prompt_config.in2_template = ' .\\D.: '
prompt_config.out_template = 'N.Out<\\#>: '
# Messages displayed when I drop into and exit the shell.
banner_msg = ("\n**Nested Interpreter:\n"
"Hit Ctrl-D to exit interpreter and continue program.\n"
"Note that if you use %kill_embedded, you can fully deactivate\n"
"This embedded instance so it will never turn on again")
exit_msg = '**Leaving Nested interpreter'
# Wrap it in a function that gives me more context:
def ipsh():
ipshell = InteractiveShellEmbed(config=cfg, banner1=banner_msg, exit_msg=exit_msg)
frame = inspect.currentframe().f_back
msg = 'Stopped at {0.f_code.co_filename} at line {0.f_lineno}'.format(frame)
# Go back one level!
# This is needed because the call to ipshell is inside the function ipsh()
ipshell(msg,stack_depth=2)
Then, whenever I want to debug something in my code, I place ipsh() right at the location where I need to do object inspection, etc. For example, say I want to debug my_function below
Using it:
def my_function(b):
a = b
ipsh() # <- This will embed a full-fledged IPython interpreter
a = 4
and then I invoke my_function(2) in one of the following ways:
Either by running a Python program that invokes this function from a Unix shell
Or by invoking it directly from IPython
Regardless of how I invoke it, the interpreter stops at the line that says ipsh(). Once you are done, you can do Ctrl-D and Python will resume execution (with any variable updates that you made). Note that, if you run the code from a regular IPython the IPython shell (case 2 above), the new IPython shell will be nested inside the one from which you invoked it, which is perfectly fine, but it's good to be aware of. Eitherway, once the interpreter stops on the location of ipsh, I can inspect the value of a (which be 2), see what functions and objects are defined, etc.
The problem:
The solution above can be used to have Python stop anywhere you want in your code, and then drop you into a fully-fledged IPython interpreter. Unfortunately it does not let you add or remove breakpoints once you invoke the script, which is highly frustrating. In my opinion, this is the only thing that is preventing IPython from becoming a great debugging tool for Python.
The best you can do for now:
A workaround is to place ipsh() a priori at the different locations where you want the Python interpreter to launch an IPython shell (i.e. a breakpoint). You can then "jump" between different pre-defined, hard-coded "breakpoints" with Ctrl-D, which would exit the current embedded IPython shell and stop again whenever the interpreter hits the next call to ipsh().
If you go this route, one way to exit "debugging mode" and ignore all subsequent breakpoints, is to use ipshell.dummy_mode = True which will make Python ignore any subsequent instantiations of the ipshell object that we created above.
You can start IPython session from pudb and go back to the debugging session as you like.
BTW, ipdb is using IPython behind the scenes and you can actually use IPython functionality such as TAB completion and magic commands (the one starts with %). If you are OK with ipdb you can start it from IPython using commands such as %run and %debug. ipdb session is actually better than plain IPython one in the sense you can go up and down in the stack trace etc. What is missing in ipdb for "object inspection"?
Also, python.el bundled with Emacs >= 24.3 has nice ipdb support.
Looks like the approach in #gaborous's answer is deprecated.
The new approach seems to be:
from IPython.core import debugger
debug = debugger.Pdb().set_trace
def buggy_method():
debug()
Prefixing an "!" symbol to commands you type in pdb seems to have the same effect as doing something in an IPython shell. This works for accessing help for a certain function, or even variable names. Maybe this will help you to some extent. For example,
ipdb> help(numpy.transpose)
*** No help on (numpy.transpose)
But !help(numpy.transpose) will give you the expected help page on numpy.transpose. Similarly for variable names, say you have a variable l, typing "l" in pdb lists the code, but !l prints the value of l.
You can start IPython from within ipdb.
Induce the ipdb debugger1:
import idpb; ipdb.set_trace()
Enter IPython from within in the ipdb> console2:
from IPython import embed; embed()
Return to the ipdb> console from within IPython:
exit
If you're lucky enough to be using Emacs, things can be made even more convenient.
This requires using M-x shell. Using yasnippet and bm, define the following snippet. This will replace the text ipdb in the editor with the set-trace line. After inserting the snippet, the line will be highlighted so that it is easily noticeable and navigable. Use M-x bm-next to navigate.
# -*- mode: snippet -*-
# name: ipdb
# key: ipdb
# expand-env: ((yas-after-exit-snippet-hook #'bm-toggle))
# --
import ipdb; ipdb.set_trace()
1 All on one line for easy deletion. Since imports only happen once, this form ensures ipdb will be imported when you need it with no extra overhead.
2 You can save yourself some typing by importing IPython within your .pdbrc file:
try:
from IPython import embed
except:
pass
This allows you to simply call embed() from within ipdb (of course, only when IPython is installed).
Did you try this tip?
Or better still, use ipython, and call:
from IPython.Debugger import Tracer; debug_here = Tracer()
then you can just use
debug_here()
whenever you want to set a breakpoint
the right, easy, cool, exact answer for the question is to use %run macro with -d flag.
In [4]: run -d myscript.py
NOTE: Enter 'c' at the ipdb> prompt to continue execution.
> /cygdrive/c/Users/mycodefolder/myscript.py(4)<module>()
2
3
----> 4 a=1
5 b=2
One option is to use an IDE like Spyder which should allow you to interact with your code while debugging (using an IPython console, in fact). In fact, Spyder is very MATLAB-like, which I presume was intentional. That includes variable inspectors, variable editing, built-in access to documentation, etc.
If you type exit() in embed() console the code continue and go to the next embed() line.
The Pyzo IDE has similar capabilities as the OP asked for. You don't have to start in debug mode. Similarly to MATLAB, the commands are executed in the shell. When you set up a break-point in some source code line, the IDE stops the execution there and you can debug and issue regular IPython commands as well.
It does seem however that step-into doesn't (yet?) work well (i.e. stopping in one line and then stepping into another function) unless you set up another break-point.
Still, coming from MATLAB, this seems the best solution I've found.
From python 3.2, you have the interact command, which gives you access to the full python/ipython command space.
Running from inside Emacs' IPython-shell and breakpoint set via pdb.set_trace() should work.
Checked with python-mode.el, M-x ipython RET etc.
Developing New Code
Debugging inside IPython
Use Jupyter/IPython cell execution to speed up experiment iterations
Use %%debug for step through
Cell Example:
%%debug
...: for n in range(4):
...: n>2
Debugging Existing Code
IPython inside debugging
Debugging a broken unit test: pytest ... --pdbcls=IPython.terminal.debugger:TerminalPdb --pdb
Debugging outside of test case: breakpoint(), python -m ipdb, etc.
IPython.embed() for full IPython functionality where needed while in the debugger
Thoughts on Python
I agree with the OP that many things MATLAB does nicely Python still does not have and really should since just about everything in the language favors development speed over production speed. Maybe someday I will contribute more than trivial bug fixes to CPython.
https://github.com/ipython/ipython/commit/f042f3fea7560afcb518a1940daa46a72fbcfa68
See also Is it possible to run commands in IPython with debugging?
If put import ipdb; ipdb.set_trace() at cell outside function, it will occur error.
Using %pdb or %debug, you can only see the filnal error result. You cannot see the code doing step by step.
I use following skill:
%%writefile temp.py
.....cell code.....
save the code of cell to file temp.py.
and then
%run -i -d temp.py, it will run the cell code by pdb .
-i: run the file in IPython’s namespace instead of an empty one.
-d: run your program under the control of pdb, the Python debugger.