I noticed that when I create a new enviornment with conda, I can import python modules in that environment that were NOT installed there.
Example with keras:
Although the module is NOT in that enviornment:
(py2) user#user-Precision-7920-Tower:~$ conda list keras
# packages in environment at /home/user/anaconda3/envs/py2:
#
# Name Version Build Channel
I can still import it, apparently from the system (user) install, outside conda!
(py2) user#user-Precision-7920-Tower:~$ python
Python 2.7.15 | packaged by conda-forge | (default, Mar 5 2020, 14:56:06)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import keras
Using TensorFlow backend.
>>> keras.__file__
'/home/user/.local/lib/python2.7/site-packages/keras/__init__.pyc'
In fact, python inside conda has access to non-conda paths!
>>> import sys
>>>
>>> sys.stdout.write("\n".join(sys.path))
/home/user/anaconda3/envs/py2/lib/python27.zip
/home/user/anaconda3/envs/py2/lib/python2.7
/home/user/anaconda3/envs/py2/lib/python2.7/plat-linux2
/home/user/anaconda3/envs/py2/lib/python2.7/lib-tk
/home/user/anaconda3/envs/py2/lib/python2.7/lib-old
/home/user/anaconda3/envs/py2/lib/python2.7/lib-dynload
/home/user/.local/lib/python2.7/site-packages <--
/home/user/anaconda3/envs/py2/lib/python2.7/site-packages>>>
Conda is supposed to keep things isolated. How did this path endd up in here, and how to avoid this from happening?
UPDATE:
My user-level python is 2.7, and I noticed this behavior always happen when I create a new conda environment with python 2.7, this just automatically adds the .local/lib/python2.7/site-packages to PYTHONPATH.
If I create new conda environments with python3.x , this does not happen.
Does this mean that one cannot create a separate isolated conda environment for the same python version as the user-level python?
In addition to what #VikashB mentioned, these can result from packages installed with pip install --user. As #TimRoberts alluded to in the comments, the site module, which populates the sys.path variable, searches paths like ~/.local/lib/python*/site-packages by default.
Temporary Options
One can disable the site module from loading such packages (see PEP 370), either by launching Python with an -s flag (python -s) or by setting the environment variable PYTHONNOUSERSITE:
export PYTHONNOUSERSITE=1
python
Longer-term options
Hiding from site module
If you need to keep these packages for some reason, one option is to move them to a non-default location so the site module doesn't find them. For example,
mkdir ~/.local/lib/py_backup
mv ~/.local/lib/python* ~/.local/lib/py_backup
This will effectively hide them, and they could still be used through PYTHONPATH if necessary.
Removal
If you don't need the packages, and only use Conda then consider just removing them
rm -r ~/.local/lib/python*
For reference, Conda users are discouraged from using the --user flag in the Conda documentation. Conda environments assume full isolation of environments, so leakage such as OP reports can lead to undefined behavior.
Experimental: envvar-pythonnousersite-true
In response to another question, I put together a simple Conda package that sets the PYTHONNOUSERSITE=1 variable at environment activation time. There are other ways to set environment variables, but this is a quick and minimal patch.
It can be installed with:
conda install merv::envvar-pythonnousersite-true
There are multiple possible reasons why this might happen.
Check if you have any activation scripts that manually add these paths.
Another option is to check if you have environment variables such as PYTHONPATH or PYTHONHOME set. If they are, then check where they are being set and remove them.
You can use conda info -a in an activated environment to show all the relevant variables/information.
Related
I just installed miniconda on mac in order to use python 3. But I have python 2 installed the old fashioned way that I want to preserve. The old configuration files for ipython and matplotlib are in my home directory at the default locations (~/.ipython/profile_default/ and ~/.matplotlib/matplotlibrc).
I thought I would use the conda environments to isolate my new python3 so I made a new "main" environment with
conda create --name=main
and installed ipython and matplotlib in my new environment
conda activate main
conda install ipython matplotlib
What is a good way for my new ipython and matplotlib to use their own configuration files? I want to keep the old config files in place so I can still use my old python 2 if I need to.
My tiny bit of progress was that in my new main conda environment I ran ipython profile create condaenv_main, which created a new ipython profile in ~/.ipython/profile_condaenv_main. Is there a way that I can have ipython load this profile automatically if I'm in this environment (maybe with some aliases or something)? I.e. I would like to be able to type
conda activate main
ipython
rather than
conda activate main
ipython --profile=condaenv_main
Right now the first way would load my old profile_default ipython which I don't want.
Is there an analogous way I can keep the conda main environment's matplotlib config also separate just for main?
Here's one solution:
For ipython I followed the idea here to define the alias
alias ipython="ipython --profile=condaenv_main"
when the environment is activated and unalias ipython when deactivated. The instructions are here.
For matplotlib I created a subdirectory of ~/.matplotlib called condaenv_main/ and put my new matplotlibrc file in there. Then I have conda set the environment variable MPLCONFIGDIR when environment is activated (and it is automatically removed when environment is deactivated). The command is (see here):
conda env config vars set MPLCONFIGDIR=$HOME/.matplotlib/condaenv_main
I have two environments within Anaconda, and I would like to remove one, but preserve the various modules installed in each.
One is base (root), and the other is Miniconda3, in which I've installed the plotly module. This was not intentional. I am not planning to need multiple environments in the foreseeable future.
Some Jupyter notebooks only run after switching to the Miniconda3 environment (otherwise the error of not finding the plotly module in the base (root) environment)
Another acceptable solution might be to set the Miniconda3 environment to default.
Another acceptable solution - install modules into the base environment.
You could try using conda-merge. According to the developer, it preserves dependencies between environments.
Install it from this command: conda install -c conda-forge conda-merge
Then use the following command to merge the environments:
conda-merge FILE1 FILE2 ... FILE-N > OUTPUT-FILE
I want to start using Anaconda the correct way, by making a new environment for each project instead of always using base. I completely reinstalled Anaconda. As far as making my first environment, do I need to specify every module that I want or are the basic ones that come with python (like OS and MATH) included?
The standard library should be included with all python installs (which also applies to conda environments when you specify python or a python version). Here you can find what is included in the standard library.
Creating a new environment in conda the following way:
conda create --name <name> python=3.x.y
or
conda create --name <name> python
will install python in your environment.
You can reference this nice cheatsheet for more conda commands.
I was suggested to conda create a new environment for installing tensorflow
First question, in general:
Why do environment exist in conda or in Python ? (Why) is it preferable to install a new library in a new environment ?
Here, in practice:
After install conda shell says $conda activate test will activate the test environment. Does it mean i can't access the lib in Spyder unless i activate test in conda shell ? Do i need to restart python shell to see the lib ? I can't access the lib (no module named tensorflow) and I assume it has to do with python not finding the path.
After install conda shell says $conda activate test will activate the
test environment. Does it mean i can't access the lib in Spyder unless
i activate test in conda shell ? Do i need to restart python shell to
see the lib ? I can't access the lib (no module named tensorflow) and
I assume it has to do with python not finding the path.
Have you installed TF within the environment?
I haven't used Spyder in a while, but what usually happens is that you can start a program (like Spyder or Jupyter) from an environment if you have installed the application within it and the environment is active. (Some editors/IDE like VS Code lets you choose the environment for a specific project, once it is able to discover all the environments.)
And, also usually, though perhaps not always, you will not need to restart the shell to import a library, after installing it. It's best to refer to the specific library's installation instructions for details like this.
Virtual Environment is used to manage Python packages for different projects. Using virtual environment allows you to avoid installing Python packages globally which could break system tools or other projects. You can install virtual environment using pip.
For example, say you have two projects, and each requires a different version of Tensorflow. This is a real problem for Python since it can’t differentiate between versions in the “site-packages” directory. So both say V1.1 and V2.1 would reside in the same directory with the same name.
This also allows easy clean up, once you are done with the project just delete the virtual environment.
Checkout more, https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
I want to upgrade python's default version i.e /usr/bin/python in Linux.
I have multiple python versions installed as
/usr/bin/python2.7
/usr/bin/python3.3
However, python command still returns python2.7
# python
Python 2.7
Type "help", "copyright", "credits" or "license" for more information.
>>>
Now, I have installed a module, which got installed in the default version 2.7.
That's why I can't use python3.3 script.py, as it returns error for missing module.
How to update this default version to 3.3?
Is there a way to install the module in /usr/bin/python3.3 as well?
Added: Module is pexpect-2.3.
Installing new python, installs by default in /usr/local/bin.
Adding this path to PATH before previous default python's path, solves the problem.
export PATH=/usr/local/bin:$PATH
# This export statement could be added to .bashrc for permanent effect.
This way old python is not messed and new one is installed.
Also, If there is already a python present in /usr/local/bin, changing symbolic link of /usr/local/bin/python to new /usr/local/bin/python3.3 solves the problem. (Python install generally only creates link when it installs in /usr/local/bin. You can do ls on /usr/local/bin/python to verify that it is link. Because python is installed as /usr/local/bin/python2.7 and then a link to this is created as below)
/usr/local/bin/python -> /usr/local/bin/python2.7
or
/usr/local/bin/python -> /usr/local/bin/python3.3
Ofcourse, path of this should be added to PATH as already mentioned above.
It's always better to never touch and mess with /usr/bin/python version, unless there is strong reason, because /usr/bin/python is generally not a link and is required by many of os modules.
The accepted answer is good though, however I have found another hack trick to this problem and I think it's pretty simple.
At the location of /usr/bin/ there are many python related files available. You can see the file python is actually a link and it points to the python2(which is pointed to python2.7).
So whenever you command python it calls the python2.7 not python3.5
The solution is to delete the original python link and make another link that points to python3.5 and make the newly link file name to python.
And you are done. :D