We Keep Coding

Jupyter: Write a custom magic that modifies the contents of the cell it's in

In a Jupyter notebook there are some built-in magics that change the contents of a notebook cell. For example, the %load magic replaces the contents of the current cell with the contents of a file on the file system.
How can I write a custom magic command that does something similar?
What I have so far prints something to stdout
def tutorial_asset(line):
print('hello world')
def load_ipython_extension(ipython):
ipython.register_magic_function(tutorial_asset, 'line')
And I can load it with %load_ext tutorial_asset. But from there I'm lost.
[Edit]:
I've found a way to get to the interactive shell instance:
#magics_class
class MyMagics(Magics):
#line_magic
def tutorial_asset(self, parameters):
self.shell
The self.shell object seems to give complete access to the set of cells in the notebook, but the only way I can find to modify the cells is to do self.shell.set_next_input('print("hello world")'). This isn't sufficient because, in a Jupyter notebook, that input cell is skipped, and it doesn't overwrite the input cell, it instead creates a new input cell after it.
This would be fine, but if I run the notebook a second time, it creates another input cell with the same file loaded, which is annoying. Can I have it load only once, say, by checking if the contents are already in the next cell?

EDIT: After a little further digging, I found that the current build of notebook cannot do both.
Well, this is a little tricky... Looking at the IPython code, it looks like you need to use set_next_input if you want to replace the cell, and run_cell if you actually want to run some code. However, I can't get both to work at once - it looks like set_next_input always wins.
Digging into the code, the web front-end supports optional clearing of the output on set_next_input. However, the kernel doesn't yet support setting this flag (and so output will always be cleared as the default action). To do better will require a patch to ipykernel.
The best I therefore have is the following code, using jupyter notebook version 4.2.1:
from __future__ import print_function
from IPython import get_ipython
from IPython.core.magic import Magics, magics_class, line_magic
#magics_class
class MyMagics(Magics):
#line_magic
def lmagic(self, line):
"Replace current line with new output"
raw_code = 'print("Hello world!")'
# Comment out this line if you actually want to run the code.
self.shell.set_next_input('# %lmagic\n{}'.format(raw_code), replace=True)
# Uncomment this line if you want to run the code instead.
# self.shell.run_cell(raw_code, store_history=False)
ip = get_ipython()
ip.register_magics(MyMagics)
This gives you a magic command lmagic that will either replace the current cell or run the raw_code depending on which bit of the code you have commented out.

One can use cell_magic for that:
from IPython import get_ipython
from IPython.core.magic import Magics, cell_magic, magics_class
from IPython.core import magic_arguments
#magics_class
class MyMagic(Magics):
#cell_magic
def my_magic(self, line, cell):
new_cell = cell + "\n" + "print('World')"
self.shell.run_cell(new_cell)
ipy = get_ipython()
ipy.register_magics(MyMagic)
%%my_magic
print("Hello")
Hello
World

What is the right way to debug in iPython notebook?

As I know, %debug magic can do debug within one cell.
However, I have function calls across multiple cells.
For example,
In[1]: def fun1(a)
def fun2(b)
# I want to set a breakpoint for the following line #
return do_some_thing_about(b)
return fun2(a)
In[2]: import multiprocessing as mp
pool=mp.Pool(processes=2)
results=pool.map(fun1, 1.0)
pool.close()
pool.join
What I tried:
I tried to set %debug in the first line of cell-1. But it enter into debug mode immediately, even before executing cell-2.
I tried to add %debug in the line right before the code return do_some_thing_about(b). But then the code runs forever, never stops.
What is the right way to set a break point within the ipython notebook?

You can use ipdb inside jupyter with:
from IPython.core.debugger import Tracer; Tracer()()
Edit: the functions above are deprecated since IPython 5.1. This is the new approach:
from IPython.core.debugger import set_trace
Add set_trace() where you need a breakpoint. Type help for ipdb commands when the input field appears.

Use ipdb
Install it via
pip install ipdb
Usage:
In[1]: def fun1(a):
def fun2(a):
import ipdb; ipdb.set_trace() # debugging starts here
return do_some_thing_about(b)
return fun2(a)
In[2]: fun1(1)
For executing line by line use n and for step into a function use s and to exit from debugging prompt use c.
For complete list of available commands: https://appletree.or.kr/quick_reference_cards/Python/Python%20Debugger%20Cheatsheet.pdf

In Python 3.7 you can use breakpoint() function. Just enter
breakpoint()
wherever you would like runtime to stop and from there you can use the same pdb commands (r, c, n, ...) or evaluate your variables.

Your return function is in line of def function(main function), you must give one tab to it.
And Use
%%debug
instead of
%debug
to debug the whole cell not only line. Hope, maybe this will help you.

You can always add this in any cell:
import pdb; pdb.set_trace()
and the debugger will stop on that line. For example:
In[1]: def fun1(a):
def fun2(a):
import pdb; pdb.set_trace() # debugging starts here
return fun2(a)
In[2]: fun1(1)

After you get an error, in the next cell just run %debug and that's it.

The %pdb magic command is good to use as well. Just say %pdb on and subsequently the pdb debugger will run on all exceptions, no matter how deep in the call stack. Very handy.
If you have a particular line that you want to debug, just raise an exception there (often you already are!) or use the %debug magic command that other folks have been suggesting.

Just type import pdb in jupyter notebook, and then use this cheatsheet to debug. It's very convenient.
c --> continue, s --> step, b 12 --> set break point at line 12 and so on.
Some useful links:
Python Official Document on pdb, Python pdb debugger examples for better understanding how to use the debugger commands.
Some useful screenshots:

I just discovered PixieDebugger. Even thought I have not yet had the time to test it, it really seems the most similar way to debug the way we're used in ipython with ipdb
It also has an "evaluate" tab

A native debugger is being made available as an extension to JupyterLab. Released a few weeks ago, this can be installed by getting the relevant extension, as well as xeus-python kernel (which notably comes without the magics well-known to ipykernel users):
jupyter labextension install #jupyterlab/debugger
conda install xeus-python -c conda-forge
This enables a visual debugging experience well-known from other IDEs.
Source: A visual debugger for Jupyter

In VsCode
File -> Preferances -> Settings ->
Open Settings (JSON) [small page icon on the upper right]
Paste this line of code to the end of it
"jupyter.experimental.debugging": true
Now you should see debug option at the top navigation bar

Add suggestions to python session autocomplete that appear in IPython notebook too

I'm trying to add bespoke suggestions to the interactive python tab auto complete. I found this toy example on the interweb
import readline, rlcompleter
addrs = ['angela#domain.com', 'michael#domain.com', 'david#test.com']
class mycompleter(rlcompleter.Completer):
def completer(self, text, state):
options = [x for x in addrs if x.startswith(text)]
try:
return options[state]
except IndexError:
return None
readline.set_completer(mycompleter().completer)
readline.parse_and_bind("tab: complete")
This works very nicely in python if I save it in a module and them import it. It also works in IPython if I paste it into an active session using the %paste magic.
However, I can't get it to work in an IPython Notebook, either by loading a module or by running it in a cell. I've found the ipython docs about their extension to the readline module but this hasn't helped. I've tried inheriting from IPCompleter objects, and using rlcompete methods etc, but this doesn't seem to have helped.
Any suggestions about how to add things to the autocomplete suggestions in a way that works in plain python and IPython Notebook
Thanks
Niall
UPDATE:
Ultimately, I'm looking of a way to add functionality to a module so that it can dynamically update the session autocomplete list (ideally for args for a specific set of functions so that it doesn't pollute the suggestions).

There is a way to do it, which is not the recommended one but works.
def my_matches(test):
# might want to be smarter here
return ['angela#domain.com', 'michael#domain.com', 'david#test.com']
ip = get_ipython()
ip.Completer.matchers.append(my_matches)
# it works
The old ways require setting hooks, but I haven't used it and is pretty old
and could be refactored

python: run interactive python shell from program

I often have the case that I'll be writing a script, and I'm up to a part of the script where I want to play around with some of the variables interactively. Getting to that part requires running a large part of the script I've already written.
In this case it isn't trivial to run this program from inside the shell. I would have to recreate the conditions of that function somehow.
What I want to do is call a function, like runshell(), which will run the python shell at that point in the program, keeping all variables in scope, allowing me to poke around in it.
How would I go about doing that?

import code
code.interact(local=locals())
But using the Python debugger is probably more what you want:
import pdb
pdb.set_trace()

By far the most convenient method that I have found is:
import IPython
IPython.embed()
You get all your global and local variables and all the creature comforts of IPython: tab completion, auto indenting, etc.
You have to install the IPython module to use it of course:
pip install ipython

For practicality I'd like to add that you can put the debugger trace in a one liner:
import pdb; pdb.set_trace()
Which is a nice line to add to an editor that supports snippets, like TextMate or Vim+SnipMate. I have it set up to expand "break" into the above one liner.

You can use the python debugger (pdb) set_trace function.
For example, if you invoke a script like this:
def whatever():
x = 3
import pdb
pdb.set_trace()
if __name__ == '__main__':
whatever()
You get the scope at the point when set_trace is called:
$ python ~/test/test.py
--Return--
> /home/jterrace/test/test.py(52)whatever()->None
-> pdb.set_trace()
(Pdb) x
3
(Pdb)

Not exactly a perfect source but I've written a few manhole's before, here is one I wrote for an abandoned pet project http://code.google.com/p/devdave/source/browse/pymethius/trunk/webmud/handlers/konsole.py
And here is one from the Twisted Library http://twistedmatrix.com/trac/browser/tags/releases/twisted-8.1.0/twisted/manhole/telnet.py the console logic is in Shell.doCommand

Embedded interactive shell in IPython

Before switching to IPython v0.11 (using Python 2.6.1), it was possible to embed an interactive IPython shell using for example this, e.g.
from IPython.Shell import IPShellEmbed
ipshell = IPShellEmbed()
ipshell() # this call anywhere in your program will start IPython
"The embedded shell has been refactored into a truly standalone subclass of InteractiveShell called InteractiveShellEmbed. All embedding logic has been taken out of the base class and put into the embedded subclass" (see here and here).
The way I understand it you should now be able to simply start a console by
import IPython
IPython.embed()
However, this raises
TraitError: The 'exit_msg' trait of an InteractiveShellEmbed instance must be a string, but a value of u'' was specified.
If we pass a string for exit_msg by
IPython.embed(exit_msg='Whatever')
Then it raises a different error
AttributeError: 'InteractiveShellEmbed' object has no attribute 'set_completer'
Did anybody else encounter this problem? Otherwise this might be a bug since it is a developer version after all.

These days (3.0+) all you need to do is:
from IPython import embed; embed()
If you mean embedding another IPython shell in IPython (recursively), there was a long time that this was not supported, but that problem was patched last year.

There are specific instructions on the github wiki:
from IPython.frontend.terminal.ipapp import TerminalIPythonApp
app = TerminalIPythonApp.instance()
app.initialize(argv=[]) # argv=[] instructs IPython to ignore sys.argv
app.start()