We Keep Coding

Jupyter autoreload workflow

I'm trying to understand the best workflow for impotring script files into a jupyter notebook.
I have a notebook that does somethig like:
%load_ext autoreload
%autoreload 2
import functions as F
Inside functions.py, I further do imports such as
import numpy as np
import mymodule
It seems then that, for example, numpy will get reloaded every time I execute a cell, which makes things a bit slow. How could I automatically reload functions.py without reloading the imports there that I never change?

I don't quite understand your question. The main functionality of %autoreload is to automatically reload modules, what it does, according to you. You can read about it here, I find it pretty well explained.
However, if you need to access import internals, you should take a look at importlib and especially importlib.reload():
import importlib
importlib.reload(my_module)
or
from importlib import reload
reload(my_module)
It is available starting from Python 3.1.

This can be achieved by specifying the configuration option(%autoreload 1) to auto-reload a selected module.
However, the module must be imported as %aimport my_module.

Import [Module] could not be resolved (PylancereportMissingImports), with module in the same folder/directory

The first few lines of the code of evaluation.py:
import os
import torch
from torch.nn import functional as F
from torch.utils.data import DataLoader
import numpy as np
from dataset import CLSDataset # warning is reported here
from tqdm import tqdm
The structure of the folder:
./
|-dataset.py
|-dictionary.py
|-evaluation.py
|-model.py
|-models/
|-[some files]
|-__pycache__
|-train.py
Notice that dataset.py is in the same folder as that of evaluation.py and https://github.com/microsoft/pylance-release/blob/main/TROUBLESHOOTING.md#unresolved-import-warnings says that The language server treats the workspace root (i.e. folder you have opened) as the main root of user module imports. But it still throws an warning of "Import dataset could not be resolved".
I tried to add the
{
"python.analysis.extraPaths": ["./"]
}
on the settings.json of both local and remote files, but it does not help.

In VSCode, go to the main window and do the following:
Do Ctrl+Shift+P (for Windows) and Command+Shift+P (for Mac)
Scroll and go to Python Select Interpreter
Now select whichever python version is installed in your system.
You're good to go!
Do upvote if it helps you!

Dataset is a relative import.
Add an __init__.py file your directory (a file with no content). Then try:
from .dataset import CLSDataset
That being said, Python imports are a tricky business. It looks like we’re only getting a glimpse of one part of workspace setup.
(I am surprised that your code runs as-is.)
Consider scaffolding your project with cookiecutter. Also try out the vs code python project tutorials. You don’t have to remake your entire project but these will give you a starting point for understanding where your current project goes awry.

I recently had the same error, just try restarting the IDE. You probably installed it thought the code was already open. Or you haven't installed the module at all.
Edit:
I'm sorry I understood it wrong, please send the code.
Edit2:
You need to add the .py at the end of the import if it is a .py file

Jupyter / IPython: After editing a module, changes are not effective even with kernel restart

I'm writing some code in a Jupyter notebook, and for some reason it is not recognizing when I've made changes to a module that the notebook imports.
I have the following cell at the top of my Jupyter notebook:
%load_ext autoreload
%autoreload 2
Based on this, my understanding is that this cell will force modules to reload upon execution of any code within the notebook. Meaning if I edit one of my imported modules, the edit should be reflected next time I run the code.
Even without this cell though, I should be able to restart the module and run each cell from scratch to the same effect. Unfortunately this is not the case. No matter how many times I kill/restart the module, or even completely shut down Jupyter notebooks, the notebook will not recognize the changes I've made to the module(s) in question.
I'm working in a virtual environment, and so far the only solution I've found is to run a script that completely rebuilds the environment (including re-downloading packages, etc). This is a pain, and a much bigger waste of time than should be necessary.
This behavior is relatively new. I've been using Jupyter notebooks on this environment for months without such problems, but I can't pinpoint what I may have changed to cause this new behavior.
Any insight would be greatly appreciated.
Edit - Let me give a quick illustration of such a problem:
Say, in a module called foo.py I have a class:
class Foo(object):
foo_bool = True
And in my notebook I have the cell:
from foo import Foo
Then later I add to foo.py:
class Foo(object):
foo_bool = True
class Bar(object):
bar_bool = False
And I now want to import both classes, so I change the notebook cell to:
from foo import Foo, Bar
Instead of properly importing both classes, I get something like:
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-2-1d0ce6099263> in <module>()
1 import some_module
2 import some_other_module
----> 3 from foo import Foo, Bar
4 from yet_another_module import Thing
5 from the_last_module import TheLastThing
ImportError: cannot import name Bar
Thanks again for the help!

I have the similar problem. I found an answer in How do I make ipython aware of changes made to a selfwritten module? works.
It will also help in your case. According to that answer, in your context, the following code should solve your problem. (It is in Python 3. Python 2 has the similar solution. Please refer to the above link).
import importlib
import foo
from foo import Foo, Bar
foo = importlib.reload(foo)
I have no idea why the magic commands, %load_ext autoreload and %autoreload 2, are not working though. It will be helpful if someone can shed light on it.

Try to install your package with the --editable flag,
pip install --editable <your-package>
# or
pip install --editable .
The issue you're experiencing could be due to you using pip install <your-package> or pip install . to install the module. This will always produce a copy of your source code and move it to the pip install directory. No matter how many times you import this module, it will always refer to the same "snapshot" of your code from when you installed it.
If instead you use pip install --editable <your-package> or pip install --editable . to install your package, the pip install of your source will refer to the actual directory you did the install from. When you then change code there, it should correctly be reflected in your notebook (assuming you're still using the magic commands; otherwise you'll need to use importlib or do a kernel restart to reimport the modules)
%load_ext autoreload
%autoreload 2

I think what is happening is that once you install a package by running setup.py (ref here), your python interpreter will not look at the source code anymore as it will have it readily compiled. You will the have available whatever the version of the code was when you run setup.
This is then more of a python issue and not a Jupyter issue, and you will need to re-run setup in order to update whatever you have available in your code. That fixes it for me at least

pycharm 2017.1 inconsistency on directory path for loading modules

I developed a python3 package in Pycharm but am running into some confusing behavior when I try to import modules in my test cases. The problem seems to be with the directory path to the internal package modules. It is a bit difficult to explain the issue, but here is the gist.
So the python package name is pyugend. Now when I try and import a module--inside the package--named Models into a test case, pycharm forces me to reference the path as pyugend.pyugend.Models. So I need to reference pyugend twice.
However, when I build, install, and import this package into a jupyter notebook or some script, then I run into errors about the pyugend package not finding the internal modules. The only way to fix these errors is to change the paths inside of the module to references like pyugend.Models.
So basically, to run tests inside of pycharm I have make sure all of the internal package imports use a directory path like from pyugend.pyugend.Models import ... But when I want to use the package outside of pycharm then I actually have to go into the package, convert all of the import pyugend.pyugend... references to just single import pyugend.Models ... references.
I have included a picture of the directory structure as well as a picture of the __init__.py.

You can add the linesys.path.append(os.path.dirname(os.path.abspath(__file__))) before the imports in __init.py__ and then change the imports to from pyugend.Models import Base_model and so on, to enable consistent behavior wherever you use the package.

Unit testing and import

My program reads a python script for configuration. So far I'm loading the script called lab.py like this:
self.lab_file = "/not/interesting/path/lab.py"
sys.path.insert(0, os.path.dirname(self.lab_file))
import lab as _config
But when I'm unit-testing it, I've a strange behavior:
when I launch only one unit test calling this code, it succeed
when I launch several unit tests, each of one calling this code independantly, some tests failed
Tracing the problem with logging, It seems the lab script is imported only the first time. This behavior seems coherent in respect of python but I was assuming than unit tests are isolated between each other. I am wrong ? If test are not independant in respect of the import, how can I write test to force the loading of my script each time ?

Try using reload.
For example:
import lab as _config
reload(_config)
In python 2 reload is a builtin function.
In python 3.2+ reload is in the imp module, but deprecated in 3.4+.
In python 3.4+ reload is in the importlib module.

I would suggest deleting the module from sys.modules
import sys
if 'lab' in sys.modules:
del sys.modules['lab']
import lab as _config
just deleting the import will not work because import checks in sys.modules if the module is already imported.
if you import then reload it works because it first loads the module from sys.modules into the local namespace and then reloads the module from file.

Maybe it helps if you run nose with this flag:
--with-isolation
From the nosedoc
Enable plugin IsolationPlugin: Activate the isolation plugin to isolate changes to external modules to a single test module or package. The isolation plugin resets the contents of sys.modules after each test module or package runs to its state before the test. PLEASE NOTE that this plugin should not be used with the coverage plugin, or in any other case where module reloading may produce undesirable side-effects. [NOSE_WITH_ISOLATION]