I'm trying to understand the best workflow for impotring script files into a jupyter notebook.
I have a notebook that does somethig like:
%load_ext autoreload
%autoreload 2
import functions as F
Inside functions.py, I further do imports such as
import numpy as np
import mymodule
It seems then that, for example, numpy will get reloaded every time I execute a cell, which makes things a bit slow. How could I automatically reload functions.py without reloading the imports there that I never change?
I don't quite understand your question. The main functionality of %autoreload is to automatically reload modules, what it does, according to you. You can read about it here, I find it pretty well explained.
However, if you need to access import internals, you should take a look at importlib and especially importlib.reload():
import importlib
importlib.reload(my_module)
or
from importlib import reload
reload(my_module)
It is available starting from Python 3.1.
This can be achieved by specifying the configuration option(%autoreload 1) to auto-reload a selected module.
However, the module must be imported as %aimport my_module.
Related
I cannot properly access submodules in pycharm. Let's say I have
import numpy as np
And then I tried accessing the testing submodule as np.testing.assert_all_close(). The code runs, but pycharm will not recognise the testing submodule, or do any kind of autocompletion. If I import like this however, it works:
import numpy as np
import numpy.testing
Then it works as np.testing.... This did not happen before, maybe it is an interpreter issue? I am using poetry as environment:
https://plugins.jetbrains.com/plugin/14307-poetry
If I open an iPython console using this same environment, there is no issue, autocomplete works perfectly.
I have two files:
MyModule.py
MyNotebook.ipynb
I am using Jupyter Notebook, latest, and Python, latest. I have two code cells in my Notebook.
Cell #1
import some stuff
Run some code
(Keep everything in the environment takes about five minutes to run this cell).
Cell #2
import MyModule
Execute code from MyModule
I would like to make code changes in MyModule.py and rerun Cell #2 but without restarting the kernel (Cell #1 did a fair amount of work which I don't want to rerun each time).
If I simply run the second cell, those changes made to MyModule.py do not propagate through. I did some digging, and I tried using importlib.reload. The actual code for Cell #2 :
from Nash.IOEngineNash import *
import importlib
importlib.reload(Nash.IOEngineNash)
Unfortunately, this isn't quite working. How can I push those changes in MyModule.py (or Nash/IOEngineNash.py in actual fact) into my Notebook without restarting the kernel and running from scratch?
I faced a similar issue , while importing a custom script in jupyter notebook
Try importing the module as an alias then reloading it
import Nash as nash
from importlib import reload
reload(nash)
To make sure that all references to the old version of the module are updated, you might want to re-import it after reloading, e.g.
import mymodule
reload(mymodule)
import mymodule
Issues may arise due to implicit dependencies, because importlib.reload only reloads the (top-level) module that you ask it to reload, not its submodules or external modules.
If you want to reload a module recursively, you might have a look at this gist, which allows you to simply add a line like this at the top of a notebook cell:
%reload mymodule
import mymodule
This recursively reloads the mymodule and all of its submodules.
I do a lot of interactive work in iPython. Currently, I'm working with Jupyter QtConsole. Suppose I start with this:
from myFuncs import func1
Then I go out to myFuncs.py and add a new function, func2. If I try this:
from myFuncs import func2
It doesn't see it. Presumably myFuncs is somehow cached. I have read about reload, but it seems to only work with entire modules, not cherry picked functions. autoreload also seems ineffective here. Is there a way around, short of restarting the kernel?
Incidentally, ipython within Spyder is fine with files changing while interacting. It is also unusably slow, so maybe related?
As #jss367 mentioned here, you can achieve this with importlib and sys modules:
import importlib
import sys
importlib.reload(sys.modules['myFuncs'])
from myFuncs import func2
In order to import a project specific module somewhere located on your disk, one can easily append this directory to sys.path:
import sys
sys.path.append(some_module_path)
import some_module
However, the latter import now violates PEP E402 ("module level import not at top of file"). At least spyder tells me so. Is spyder here too picky?
In spyder there is the principal idea of a "project", where I assumed environments can be adjusted specific for this project. However, I have no clue, how to modify e.g. the sys.path depending on a spyder project.
How can I modify sys.path in a spyder project? Or is there a general python way of solving this issue?
You could put the sys.path extension in a separate module, e.g. _paths.py.
Contents of _paths.py:
import sys
sys.path.append(some_module_path)
sys.path.append(some_other_module_path)
# ...and so on...
And then in your main application:
import sys
import _paths
import some_module
some_module.some_func()
This solution puts your "project configuration" nicely in a single place (which makes it easy to maintain in the future), and complies with at least PEP8 (including E402) and pylint rules.
As alternative solution to the answer with a separate module if found this as working solution for me.
try:
sys.path.append(Path(__file__).parent.parent)
except IndexError:
pass
If I just use the sys.path.append(...), I get the warning, but using the try-catch block does not produce a warning.
I know this doesn't answer the question, but it may be helpful information.
You can import that module by directly specifying its path, without using sys.path.append
In Python 3 this is as simple as
import imp
some_module = imp.load_source('some_module', '/path/to/some_module.py')
More information here: How to import a module given the full path?
My program reads a python script for configuration. So far I'm loading the script called lab.py like this:
self.lab_file = "/not/interesting/path/lab.py"
sys.path.insert(0, os.path.dirname(self.lab_file))
import lab as _config
But when I'm unit-testing it, I've a strange behavior:
when I launch only one unit test calling this code, it succeed
when I launch several unit tests, each of one calling this code independantly, some tests failed
Tracing the problem with logging, It seems the lab script is imported only the first time. This behavior seems coherent in respect of python but I was assuming than unit tests are isolated between each other. I am wrong ? If test are not independant in respect of the import, how can I write test to force the loading of my script each time ?
Try using reload.
For example:
import lab as _config
reload(_config)
In python 2 reload is a builtin function.
In python 3.2+ reload is in the imp module, but deprecated in 3.4+.
In python 3.4+ reload is in the importlib module.
I would suggest deleting the module from sys.modules
import sys
if 'lab' in sys.modules:
del sys.modules['lab']
import lab as _config
just deleting the import will not work because import checks in sys.modules if the module is already imported.
if you import then reload it works because it first loads the module from sys.modules into the local namespace and then reloads the module from file.
Maybe it helps if you run nose with this flag:
--with-isolation
From the nosedoc
Enable plugin IsolationPlugin: Activate the isolation plugin to isolate changes to external modules to a single test module or package. The isolation plugin resets the contents of sys.modules after each test module or package runs to its state before the test. PLEASE NOTE that this plugin should not be used with the coverage plugin, or in any other case where module reloading may produce undesirable side-effects. [NOSE_WITH_ISOLATION]