I located a really weird behavior of Jupyter and caused me to reinstall it but the problem persisted.
I have a Windows machine where I have installed anaconda (the latest version). There are two environments:
(base) Which contains nothing special maybe seaborn pandas etc. and is on python version 3.8.12.
(tf-gpu) Which contains tensorflow-gpu and is on python version 3.9.7.
Mind you, these two environments are created directly from a fresh installation of anaconda, and I didn't touch anything except installing tensorflow-gpu.
When I launch Jupyter through Anaconda, it launches normally on the home folder "C:\User\user" for both environments i.e. correct packages installed and correct version of python on both. I can also launch an instance of Powershell through Anaconda or Windows, (always in home dir), and launch Jupyter both environments behave as they should (packages, python version). So far so good...
The problem starts when I want to launch Jupyter from a directory other than home dir. I keep all my project files in a separate partition (D:), thus I navigate to that directory through a cmd/powershell (launched through Anaconda or Win) and type "jupyter notebook". The notebooks open on that director and:
When the base env is selected the notebook reports a correct name "base"(os.environ['CONDA_DEFAULT_ENV']) and python version 3.8.12 (sys.version_info). Everything behave as base env was setup.
When the tf-gpu env is selected the notebook reports a correct name "tf-gpu" but I get a python version 3.8.12, which is completely wrong! It should be 3.9.7. I can't import tensorflow since I get an error. Essentially I am in the base env without knowing it.
I then check in the same cli window (tf-gpu) that launched jupyter and python --version reports 3.9.7 and of course can import tf run it etc.
Jupyter just reports the environment name but in reality it uses the base environment even though it launched from said env. How can this change?
Why does Jupyter works normally when launched from the user directory?
If this is not solved easily, what is a workaround? Jupyter doesn't recognize symbolic links, and thus I can not navigate to another directory if it is not a subdirectory of home.
Finally, I also tried adding the env as kernel through python -m ipykernel install --user --name tf-gpu --display-name "Tensorflow GPU (tf-gpu)", but it doesn't seem to change a thing.
Update 1:
Of course I activate the tf-gpu through conda activate tf-gpu and check the python version and tensorflow (correct results), the problem is when launching jupyter notebook. The notebook reports the tf-gpu env but wrong python version and tf is missing.
Update 2:
After some searching around it seems that jupyter notebook has much trouble selecting a python version if multiple versions of python are installed in the system (in my case 2, 1 on each environment environments). Which is ridiculous if you think that that's why we are using environments for...
The reason you are having this problem I’m guessing is because your launched jupyter notebook from your conda base environment. You will see a (base) in your comment prompt.
To remedy this, When you use the command prompt to launch, do conda activate tf-gpu before you launch Jupyter notebook, now check if your versions are correct.
What I did and worked was run ipython3 notebook in the environment which launched the notebook server and everything was working properly correct environment name and correct python version and packages (tensorflow is working as it should). This must have updated some configuration files since I can now run jupyter notebook from a different folder and the gpu-tfu env works correctly.
Please remember that these problems were occurring only if I was launching jupyter outside the home directory (C:\User\user) on a different partition. Also keep in mind that this is a Windows installation and more complicated things may be happening with user paths, environments, etc than in linux.
Related
I have a python project (folder) that I'm working on in VS Code (in Windows) and it uses Jupyter notebook. The project uses a virtual env. It was working fine a few days ago. Today when I open up one of the .ipynb files in the project, I see:
Running cells with 'Python 3.9.12 64-bit' requires ipykernel package.
Run the following command to install 'ipykernel' into the Python
environment. Command:
'c:/Users/myName/AppData/Local/Programs/Python/Python39/python.exe -m
pip install ipykernel -U --user --force-reinstall'
I don't understand why VS Code is looking at the system installation of python. I have the following in settings.json in the .vscode subfolder of this project:
{
"python.pythonPath": "proj_env\\Scripts\\python.exe",
"python.terminal.activateEnvironment": true
}
And in the virtual environment for this project, I have confirmed I have ipykernel installed. It was working fine a few days ago. I see that ipykernel was updated 10/31/2022, so just to try it, I updated the ipykernel package in the virtual env from 6.15.2 to 6.17.0.
But clearly for some reason VS Code is looking to the system installation of python instead of the virtual env.
Pulling my hair out because I changed nothing in the past few days. In fact I just opened the project today for the first time since a few days ago when it was working fine.
When I press CTRL+SHIFT+P to select python interpreter, the virtual env python interpreter is already selected. But I still see the warning posted above re: ipykernel package needing to be installed.
Where am I going wrong?
Here's my attempt at an explanation of the solution. I'm sure someone understands this better and can explain it better (please do and I will pick your answer as the solution!):
VS Code uses one python interpreter for .py files and terminal and a different python interpreter for Jupyter notebook (.ipynb) files. The kernel I wanted wasn't there, and it was using the system installation of python, which is why I was getting this message (ipykernel not installed to system environment, and I don't want it there), while the virtual env python (which has ipykernel) was being used for .py and terminal in the same project, hence the confusion.
To 'install' the kernel for Jupyter, I activated the virtual env in VS Code powershell terminal. Then ran:
python -m ipykernel install --user --name=project_name
Then exit VS Code, restart, open project folder and in upper right corner of Jupyter notebook, use the kernel picker to select the kernel. The kernel I just created wasn't there, so I used CTRL+SHIFT+P in VS Code, then type Jupyter: Filter Kernels, and there I could see the kernel I created, and it was checked, then went back to kernel picker and was able to select the correct kernel.
So now the .ipynb code has access to all the packages installed in this virtual environment.
One annoyance is VS Code doesn't 'remember' that this is the kernel I want for this project, so I have to select the kernel each time I open this project. I'm sure there's a solution for this, but I'll have to look for that later.
In general I am new to python so keep it easy. I have started my coding in Spyder since it was easy to install libraries with Anaconda.
Now we want to switch to VS Code for version control. I have switched interpreter and python path to Anaconda folder, but if I open python files I have created in Spyder I can't use in VS code, issue always is with importing libraries.
Screenshot
The first rule with Anaconda is to not manually mess around with the paths. Anaconda has a different philosophy: Before you can use Python, you need to activate a conda environment.
c:\> conda activate
(Anaconda3) c:\> python
>>>
VSCode detects conda environments and you just have to select one for your project (Python: Select interpreter). Now open a new terminal and watch VSCode activating the environment for you (like shown above).
However, when you start your script for the first time, VSCode sometimes invokes the interpreter even before the conda activation has finialized. Then simply restart your script.
I am attempting to import cartopy into a notebook but am running into issues. If I do a "conda install cartopy" in my base environment, I get the frozen/flexible solve issue (https://github.com/conda/conda/issues/9367). When I open a new environment and do a "conda install cartopy", it seems like everything works. "conda list" shows version 0.18.0 in that environment. Then I open a Jupyter Notebook from within that environment and try to import it into the notebook but originally got the response "no module named cartopy". I tried fiddling with my environments and settings based on feedback on this page (In which conda environment is Jupyter executing?), but now the error is "no module named numpy"! Can someone please help me understand why the notebook isn't seeing these modules? Thank you.
Are you certain that numpy is installed in this new environment?
Given that you are using the Anaconda distribution of Python, you should be able to view/configure your environment and it's installed modules using the Anaconda Navigator. Here you can see a full list of all installed modules, and via the terminal/console, launch a Jupyter Notebook from not only within a specific environment, but within a specific directory!
On the left hand side, you can choose the environment that you want to use to start Jupyter Notebook, and on the right you can view all of the installed modules in that environment. Make sure that jupyter, numpy, and cartopy are all listed as installed packages.
To make sure that I am launching Jupyter from my desired environment, I always launch it directly from the terminal. I "Open Terminal" with my environment, "cd" into the directory that I want to use, and then use the jupyter notebook or jupyter lab command.
I have an Anaconda distribution installed on Mac OSX (10.14.2). With it, I installed VSCode which I'm to write python scripts. Recently, I've updated some environments in Anaconda and something happened with the configuration of the python path VSCode uses for its integrated terminal: when I run my code, an ImportError comes up for the joblib package that I now have installed in my conda environment (also happens for other packages), which doesn't happen when I run the same code in a regular terminal.
Using conda list in the integrated terminal gives me the correct list of packages for the environment I'm using, and conda env list indicates that indeed this specific environment should be active. However, I can see that that something is off, because in my normal terminal I get:
$ which python
python is /Users/Joris/anaconda3/envs/astro3/bin/python
python is /usr/bin/python
While in the VSCode terminal:
$ which python
python is /usr/bin/python
python is /Users/Joris/anaconda3/envs/astro3/bin/python
I've tried changing some of the VSCode settings, like manually pointing python.pythonPath and python.venvPath to my anaconda environment folders. (The setting python.terminal.activateEnvironment is set to true.) Also, reinstalling VSCode through the Anaconda distribution does not seem to help. So unfortunately, I can't seem to get VSCode to run python with my Anaconda environment which contains the packages I need.
Press (macOS): ⇧+⌘+P, (Linux/Windows: Ctrl+Shift+P) in VS Code.
Type: python select interpreter
Select /Users/Joris/anaconda3/envs/astro3/bin/python
Test if your lib is usable.
I'm using virtualenvwrapper-win to manage environments, I created one, pip-installed a bunch of packages but when I try to import them from jupyter notebook or lab it only looks in the main directory and not in the environments. I tried using the conda environments and had the same problem, so I suspect the problem is in the jupyter configuration, which I haven't meddled with much but might have messed up at some point.
sys.executable returns 'c:\\anaconda3\\python.exe', but I suspect it should be using the one in the virtual environment right?