I have a pet project which I started as a Jupyter notebook. So far, I put all the Python code in the notebook.
At the start everything was fine. But over time the code I wrote in the notebook became more and more complex. Now it is close to unmanagenable: When I find an error, I need
to navigate to the code part with the error (usually at the beginning of the notebook),
fix the error there,
go to (usually) the bottom of the notebook to trigger the execution of the code I changed.
I want to separate the code in two parts:
One that will be stored as Python files and which I will edit using an editor (and/or an IDE).
The code in the Jupyter notebook that calls code parts 1 and presents its output (i. e. use the Jupyter notebook as a user interface for the Python code from step 1).
Let's assume that the notebook runs on my local machine (Windows 7; Jupyter runs in Anaconda) and the Python files are also stored locally.
What are good ways to use code from IPython files such that I can modify this code frequently and fast?
By "frequently and fast" I mean "with as little steps as possible that are necessary to propagate changes from Python files to the notebook". The ideal solution would be something where I change one of the Python files, run one command, and then the changes are available in the Jupyter notebook. Or, to use an older analogy, I want it to be like PHP -- you change the code often and immediately see the results of your changes.
Update 1: I tried to use the solution with %load TestClass.py in a cell.
The problem is that the cell contents is not updated, if the file changes.
Example:
Let's say I put the text
class TestClass:
def __init__(self):
print("TestClass constructor")
into TestClass.py. Then I create a cell in Jupyter notebook with %load TestClass.py. When I execute that cell, the code from TestClass.py is imported and the line %load TestClass.py gets commented out.
Now I change TestClass.py to
class TestClass:
def __init__(self):
print("TestClass constructor")
print("change")
When I execute the cell, its contents has not changed.
Sounds like the autoload extension of IPython is exactly what you need. Simply plug
%load_ext autoreload
%autoreload 2
in one of the first cells of your Jupyter notebook and imported python modules are automatically reloaded on change. You also can do changes to installed python packages, provided you have installed them editable.
I've been working on a similar sort of problem. If you use Jupyterlab instead of the older Jupyter Notebooks, you can open multiple nb's at the same time, edit the one holding your functions, and then restart the kernel in the other to update when you import. Unfortunately you lose all your variables and need to run the entire notebook again but it allows the changes in the first nb to propogate into the second.
This isn't a perfect answer but has allowed me to keep working on my projects.
You're writing:
and the line %load TestClass.py gets commented out.
I understand, that you comment it out. Why? Re-run the import and the class gets updated.
BTW !run TestClass.py should do the same - at least in Jupyter NB.
I usually unload the modules I want to refresh then reimport. Objects bound to old code don't get refreshed and every now and then I have to restart the kernel but it's a lot faster than restarting the kernel each time. I would only suggest this for someone who already understands or is willing to understand the impact of removing modules from sys.modules. But it saves a lot of time for me.
See UnloadModules here.
My typical usage in Jupyter:
from coppertop import Unload
Unload("fred.joe")
from fred.joe.sally import arthur
arthur.doSomethingInteresting()
Related
I'm new to coding and to PyCharm. I've learned with Jupyter and trying to get my head around PyCharm debugging (using single/multiple breakpoints).
What I liked about Jupyter is that when you put your code into different blocks, run them to output lists which you'll then be working with (the lists - as long as you haven't done anything to permanently affect them, but in that case, I would do new_list = og_list.copy() and play with the new list, to avoid re-running all of the code above) to create/test your next lines of codes, I can't figure out a way to proceed in the same way into PyCharm/Debugging.
In PyCharm, the way that I am using it, it seems that every time that I am changing a line of code to re-test/debug, I need to run the full code once again. I know I must be doing it wrong. I've played a few videos now and read a few posts online but I still can't seem to get it. Anyone here available to help?
Are you aware that you could write jupyter notebooks in PyCharm? - The cell functionality of jupyter notebook is only given in jupyter notebook and ipython as far as I know.
You may also open a iPython-shell in PyCharm.
Also, remember that you can easily convert a jupyter notebook to a .py-File, so nothings prevents you from writing code in jupyter notebook.
I am currently writing some Python code in a Jupyter notebook and I wonder if there is any possibility to track the execution of statements directly in Jupyter? I know there is this module called "trace": https://docs.python.org/3/library/trace.html#trace.Trace.results But it seems to me that with "trace" it is only possible to show the flow of execution in the console or to export it to a new file. However, I would like to display it directly in Jupyter Notebooks. Is this possible - with or without the mentioned module - and if yes how?
Thanks for your help.
For something along the lines of debugging code, you may be interested in the cell magic %%debug, discussed here to open a debugger in the notebook and use pdb commands. Or PixieDebugger if you need something fancier.
You may also be interested in the variable inspector extension for JupyterLab. There is (old) an animated example of using it here. You can easily try it out by clicking on launch binder badge there and starting a notebook. Then right-click in the open notebook and select Open Variable Inspector from the list. The animation will show you how to drag the tabs to arrange them side by side on your screen. Now as you run code, you'll see the variables updated.
I am running quite a time consuming python code in PyCharm, where data is initially loaded, then a model is created etc.
I would like to know if it is possible to set a breakpoint at a particular point in the code where the state of the environment will be saved. From there on I want to make edits to the rest of the code, however ideally re-run from the breakpoint I set.
By doing this I would avoid the loading of data, and model training which would be a repeated process.
Is this possible in Python or is there such a functionality in PyCharm?
Thanks
AFAIK and according to this question form 2014, PyCharm does not support hot-swapping code during debug mode. When you set a breakpoint and do some changes in code they will not be applied.
You can however setup a Python propmt in debug console, which will have access to all your variables at the time of current code execution. You can modify their values and check why your code is failing. To do so just click on this icon
yes, you can do check the python code at breakpoints using pycharm debugger
Refer this for how to use it .how to use a debugger in pycharm
You cannot use code modified code in between and then start from that breakpoints using debugger.
little hack to do this, one need to run the python code in debugger mode ie python -i main.py and also need to make sure that there in no interdependence of previous code to the new modified one.
One option I have found is to convert your code to ipython notebook. Put all the heavy-weight/one-time run code in the first few cells and unstable/to be debugged code in the later cells.
Now, you run the first few cells one time, so the output from these cells is stored and readily available always for later cells. Whenever you change code in later cells, you only have to run those cells (unless you modified something in the earlier cells in which case you need to rerun those cells as well).
Pycharm/most python IDEs support running and debugging ipython notebook cells (individually or in groups).
I just started using jupyter for a python project. I constantly find myself adding an extra cell just to perform some basic try&error debugging. This way I omit the whole code of the cell is being executed but it still doesn't feel like the right way to do it.
Does Jupyter provide something like a static kernel terminal, for example always visible at the bottom of the screen, where I can simply paste code and execute runtime variables?
By the way: I did search but didn't find anything looking for static console, and terminal. Maybe I'm just looking in the wrong direction.
Thanks!
The jupyter console command will provide you with an interpreter environment that you can experiment with code running within the Jupyter environment outside of a notebook.
It's not exactly what you're looking for but may provide a better environment for testing and developing code that you can then paste into the appropriate notebook.
Sorry for the awkwardly phrased title. But here's the situation. I am using Jupyter to do some Python development. I am editing an import file, but when I run the program that is doing the importing, it does not use the newest save, it just uses (apparently) the version that is last in its stack.
I stopped the kernel and ran the program again, to no avail. I started a new notebook, copy and pasted the code, and that time it did freshly import the imports.
Is there a way to streamline this procedure so as to avoid having to start a new notebook every time I edit a source file?
First file:
# the_import.py
def doStuff():
...
Second file:
# main program
import the_import
def doOtherStuff():
...
# When I run this program, it does not use the newest save
# of the_import.py unless a new Jupyter notebook is started.
When you kill the IPython notebook server and restart it, you will end up with a new kernel instance. You should you start your workflow after restarting and opening a notebook again by running all cells. In the top menu, before you do anything else, first select "Cell->Run all"