We Keep Coding

From which location does Python access the installed packages? [duplicate]

I noticed that when I create a new enviornment with conda, I can import python modules in that environment that were NOT installed there.
Example with keras:
Although the module is NOT in that enviornment:
(py2) user#user-Precision-7920-Tower:~$ conda list keras
# packages in environment at /home/user/anaconda3/envs/py2:
#
# Name Version Build Channel
I can still import it, apparently from the system (user) install, outside conda!
(py2) user#user-Precision-7920-Tower:~$ python
Python 2.7.15 | packaged by conda-forge | (default, Mar 5 2020, 14:56:06)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import keras
Using TensorFlow backend.
>>> keras.__file__
'/home/user/.local/lib/python2.7/site-packages/keras/__init__.pyc'
In fact, python inside conda has access to non-conda paths!
>>> import sys
>>>
>>> sys.stdout.write("\n".join(sys.path))
/home/user/anaconda3/envs/py2/lib/python27.zip
/home/user/anaconda3/envs/py2/lib/python2.7
/home/user/anaconda3/envs/py2/lib/python2.7/plat-linux2
/home/user/anaconda3/envs/py2/lib/python2.7/lib-tk
/home/user/anaconda3/envs/py2/lib/python2.7/lib-old
/home/user/anaconda3/envs/py2/lib/python2.7/lib-dynload
/home/user/.local/lib/python2.7/site-packages <--
/home/user/anaconda3/envs/py2/lib/python2.7/site-packages>>>
Conda is supposed to keep things isolated. How did this path endd up in here, and how to avoid this from happening?
UPDATE:
My user-level python is 2.7, and I noticed this behavior always happen when I create a new conda environment with python 2.7, this just automatically adds the .local/lib/python2.7/site-packages to PYTHONPATH.
If I create new conda environments with python3.x , this does not happen.
Does this mean that one cannot create a separate isolated conda environment for the same python version as the user-level python?

In addition to what #VikashB mentioned, these can result from packages installed with pip install --user. As #TimRoberts alluded to in the comments, the site module, which populates the sys.path variable, searches paths like ~/.local/lib/python*/site-packages by default.
Temporary Options
One can disable the site module from loading such packages (see PEP 370), either by launching Python with an -s flag (python -s) or by setting the environment variable PYTHONNOUSERSITE:
export PYTHONNOUSERSITE=1
python
Longer-term options
Hiding from site module
If you need to keep these packages for some reason, one option is to move them to a non-default location so the site module doesn't find them. For example,
mkdir ~/.local/lib/py_backup
mv ~/.local/lib/python* ~/.local/lib/py_backup
This will effectively hide them, and they could still be used through PYTHONPATH if necessary.
Removal
If you don't need the packages, and only use Conda then consider just removing them
rm -r ~/.local/lib/python*
For reference, Conda users are discouraged from using the --user flag in the Conda documentation. Conda environments assume full isolation of environments, so leakage such as OP reports can lead to undefined behavior.
Experimental: envvar-pythonnousersite-true
In response to another question, I put together a simple Conda package that sets the PYTHONNOUSERSITE=1 variable at environment activation time. There are other ways to set environment variables, but this is a quick and minimal patch.
It can be installed with:
conda install merv::envvar-pythonnousersite-true

There are multiple possible reasons why this might happen.
Check if you have any activation scripts that manually add these paths.
Another option is to check if you have environment variables such as PYTHONPATH or PYTHONHOME set. If they are, then check where they are being set and remove them.
You can use conda info -a in an activated environment to show all the relevant variables/information.

conda command will prompt error: "Bad Interpreter: No such file or directory"

I'm using arch linux and I've installed Anaconda as per the instruction on the Anaconda site. When I'm attempting to run conda info --envs I get the following error:
bash: /home/lukasz/anaconda3/bin/conda:
/opt/anaconda1anaconda2anaconda3/bin/python: bad interpreter: No such
file or directory
I've tried looking for the directory /opt/anaconda1anaconda2anaconda3/bin/python: but it simply doesn't exist.
Furthermore, when I run python from the terminal it runs as normal with the following displayed at the top
Python 3.5.2 |Anaconda custom (64-bit)| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
for completeness my .bashrc file resembles:
#
# ~/.bashrc
#
# If not running interactively, don't do anything
[[ $- != *i* ]] && return
alias ls='ls --color=auto'
PS1='[\u#\h \W]\$ '
# added by Anaconda3 4.0.0 installer
export PATH="/home/lukasz/anaconda3/bin:$PATH"
# python startup for up keys
export PYTHONSTARTUP=$HOME/.pythonstartup
I've tried following this and making the the appropriate changes but nothing, I've also attempted to do this but there really isn't a solution posted.
I would like to try to fix this without having to remove Anaconda and reinstalling it.

Something must have gone wrong during the installation, I suppose.
The bad interpreter means that a script is looking for an interpreter that doesn't exist - as you rightfully pointed out.
The problem is likely to be in the shebang #! statement of your conda script.
From Wikipedia: Under Unix-like operating systems, when a script with a shebang is run as a program, the program loader parses the rest
of the script's initial line as an interpreter directive; the
specified interpreter program is run instead, passing to it as an
argument the path that was initially used when attempting to run the
script.
If you run
cat ~/anaconda3/bin/conda
You will probably get the following:
#!/opt/anaconda1anaconda2anaconda3/bin/python
if __name__ == '__main__':
import sys
import conda.cli
sys.exit(conda.cli.main())
Changing the first line to point a correct interpreter, i.e., changing it to:
#!/home/lukasz/anaconda3/bin/python
Should make the conda command work.
If you are sure that you installed everything properly, then I'd suggest maybe reaching out for support from the anaconda community.

I encountered the same error while trying
conda
The error you should interpret as follows:
bash: "path_to_file_with_error": "path_to_file_it_points_to":
bad interpreter: No such file or directory
How to fix
Type in terminal
nano "path_to_file_with_error"
Change first line of the file to correct path of the python (in my case it was in miniconda/bin)

As the response above, this issue can be solved by changing the
#!/opt/anaconda1anaconda2anaconda3/bin/python
to
#!/opt/anaconda3/bin/python
However, as soon as you do the next installation, e.g. "conda install [...]" this will be changed again to anaconda1anaconda2anaconda3, for whatever reasons.
You might also realize some installation warnings and errors which are very likely to be related to this problem. If you want to get rid of this problem, you have to solve this warnings and errors. My strongest assumption is that there are missing administrator rights causing this problem, when you attempt to installs some conda packages the first time.

When you change the path to the interpreter conda will not be activated, so by following any of the previous answers you will end up in dead end.
you get the following
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
So to solve this renaming path issue you need to:
you can use any text editor:
nano ~/anaconda3/etc/profile.d/conda.sh
change the CONDA_EXE and CONDA_PYTHON_EXE paths to the correct path
example:
export
CONDA_EXE='/home/yourusername/anaconda3/bin/conda'
export _CE_M=''
export _CE_CONDA=''
export CONDA_PYTHON_EXE='/home/yourusername/anaconda3/bin/python'
then last step do:
source ~/anaconda3/etc/profile.d/conda.sh
To test your conda
Try:
conda activate
conda deactivate
to make this change permanent to all terminals
please add this line to ~/.bashrc
source ~/anaconda3/etc/profile.d/conda.sh

If somebody runs into this problem and none of the above solutions works, it could be that on some update process the conda executable (which is a python script) was replaced with an identical-looking script with one key difference, it contains windows line endings.
This results in that executing the script via bash e.g.:
<some_path>/conda/bin/conda
will result in the error, but executing directly via python works
<some_path>/conda/bin/python <some_path>/conda/bin/conda
Can be fixed by dos2unix
dos2unix <some_path>/conda/bin/conda
or just move the file away and move it back.

For centos/Rocky Linux 8 OS:
conda is a 32 bit application but centos OS is 64 bit,
you need to install package below:
yum install glibc.i686
reference:
https://forums.centos.org/viewtopic.php?t=14169

os x, python, homebrew -> something weird going on

hoping someone can help me out. I followed a couple of guides about setting up a new machine for python development. (one) (two)
I followed pretty much everything but I am really confused about the changes to my .bash_profile
When I comment out export PATH=/usr/local/bin:$PATH in .bash_profile and type which python into terminal, I see what I expect.
which python
/usr/local/bin/python
python
Python 2.7.11 (default, Jan 22 2016, 08:29:18)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
What is happening in bash_profile, how come it doesn't matter if I comment everything out, save it, and run the commands?
More importantly, how do I get to my default install of python and make sure it is still there and unaffected? Am I correct in understanding that the above version is the python installed by Homebrew?

.bash_profile is read only when the Bash shell is loaded, so changing this file after the shell has been loaded doesn't affect it. To see the changes after modifying and saving the file, you need to restart the shell or run source ~/.bash_profile.
Modifying the PATH environment variable doesn't affect the Python itself, it only modifies where it's searched first to be executed, when you will run python .... E.g., if you have python installed in /a/python and /b/python, then:
if your PATH is set to /a/python:/b/python:$PATH and you run python ....py actually the /a/python ....py is executed;
if your PATH is set to /b/python:/a/python:$PATH and you run python ....py actually the /b/python ....py is executed.
If you didn't modified the PATH (e.g. but executing something like export PATH=/usr/local/bin:$PATH in the shell or adding it to .bash_profile), then executing which python should show the default python path.
In the case you described, it shows you the python, that you set it to use by modifying the PATH. To see the default one, comment the export... string, save the .bash_profile file and either run source ~/.bash_profile or restart the shell, and only then see the which python output.

How to switch to python2 in a particular terminal window temporarily?

I have a big third-party python2.7 application, a bunch of python scripts, which is added into PATH and it requires python2.7 instead of python 3.5 which I have by default.
$ python
Python 3.5.1 (default, Mar 3 2016, 09:29:07)
[GCC 5.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
$ python2
Python 2.7.11 (default, Mar 3 2016, 11:00:04)
[GCC 5.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
I run that application as $ some_app arg1 arg2. How can make it use python2 as a temporary solution, preferably only in that single terminal window?
I don't substitute a call of that application with "python". I call it like $ some_app arg1 arg2. I don't want to change the version of python permanently or modify the source code of the application.

My favorite way: use virtualenv or pyenv
The most used tool for keeping multiple Python environments is virtualenv. I like to use a sugary wrapper around it called virtualenvwrapper.
Usually I install it using:
sudo pip install virtualenvwrapper
Then I add a line like the following one to my shell profile:
source /usr/local/bin/virtualenvwrapper.sh
Now you can make a virtual environment for each version of Python. Gosh, you can make one for each Python project!
mkvirtualenv py3
mkvirtualenv py2 --python=python2
Then you can switch temporarily to python2 by typing:
workon py2
There is another tool called pyenv that lets you easily switch between multiple versions of Python. Quoting the project readme, "It's simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well. This project was forked from rbenv and ruby-build, and modified for Python". Looks like it is getting popular.
Both are elegant, easy to use and a perfect fit for this use case (switch to a particular python version in a terminal window temporarily).
The Unix way: set $PATH
It is a best practice to write the shebang for Python scripts like:
#!/usr/bin/env python
If the 3rd-party app you are running follows this convention, you can change the PATH. For example, suppose the default python interpreter is not the one you want:
$ env python --version
Python 3.5.1
Make a local copy of the python you want to make the default temporarily (you just have to do this one time):
$ mkdir ~/local
$ mkdir ~/local/python2
$ ln -s `which python2` ~/local/python2/python
Then every time you want to make python2 the default, do:
$ export PATH=~/local/python2/:$PATH
Now the default is the version you want:
$ env python --version
Python 2.7.10
The positive karma way: if the 3rd-party app has a hardcoded path in the shebang
Perhaps the app author did not follow the best practices and hardcoded the Python path in the shebang. For example, suppose the first line of the program is:
#!/usr/bin/python
In this case, none of the previous tricks would work. Personally, I would change it to the canonical form and send a pull request to the author of the 3rd-party app.
The bossy way: change the default Python version system-wide
While this does not really answer your question - because it affects every window and avery other user in the system - I'm mentioning it for the sake of completeness.
For example, if you are using Ubuntu you can change the default python interpreter using:
$ update-alternatives --config python
Other Linux distributions probably have something like this.
Quick and dirty
The shebang just tells the shell to call the script using the given interpreter. You can override the shebang when calling the app as interpreter app:
$ python2 `which app_name`
Or:
$ python2 /path/to/app_name
You can also just edit the program and change the shebang:
#!/usr/bin/python2
Note that this would change permanently the Python interpreter for that program, it is not temporary nor restricted to a particular terminal. Worst, if the program is in the global PATH, all the other users of the system will be affected. If you want to restore the old behavior, you must undo your changes.
That said, sometimes you just don't care about doing it right, you just want to make it work.

You can make use of terminal alias. The alias will not be temporary but it will not affect your system, so you can have a short alias like:
alias pyt "/usr/bin/python2.7"
in your .bashrc file inside home directory

As the question is tagged with Linux, I assume your scripts starts with that line
#! /usr/bin/env python
This is great to allow the system to choose the current Python installation, but here your requirement is to specifically use version 2.7
So IMHO you just have to change the hashbang line to select the required version:
#! /path/to/python-2.7

If you are using Anaconda as your Python distribution, it has virtual enviroments built in. If not, use virtualenv and virtualenvwrapper as Paul recommended.
In addition I recommend autoenv. With that you can automatically change your venv depending on the folder you're in. There is one for bash and one for zsh.

setting Python path

I installed both Python 2.7 and 3.4 on my computer, but now Python seems to confuse about the paths. It always shows syntax error when I run the scripts that worked before. I uninstalled both of the versions by removing them in application, and reinstall 2.7, but the problem remains the same. What should I do now?
Now I type Python in terminal:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
I tried this http://hints.binaryage.com/how-to-install-python-3-2-3-on-mac-os-x/ before I uninstall both versions but it didn't work.

If you want to understand which path variables you have set, you can type:
echo $PATH
at the command line in terminal and it'll tell you the 'path' (where it's looking when you type a command like 'python'). Another test you can do uses the 'which' command to see which file it'll pick when you type a command e.g.
which python
You can specify what you want the PATH variable to look like by editing one of several files. These are always stored in your home directory (shortcut on command line is ~ E.G. ~/myfile.txt) but there are a few file names that might be being used depending on your setup. Commonly the file names are .profile or .bash_profile, but there are others. NB: the '.' at the start of the file name means they're hidden from normal view. You can view all files in your home directory using the command:
ls -al ~
On my Mac, the file used is .bash_profile
Anyway, that'll help you see where the problem lies. As the previous answer states, often when you install both, one is given a specific name e.g. python27 or python3 or whatever.
However, the best approach is probably to get to know and then use, VirtualEnv: https://virtualenv.pypa.io/en/latest/
This lets you create a new, virtual python environment for every project. You set them up, telling them which python version to use and which libraries/packages to include, and then there are no clashes between projects. There is a stack overflow for how to tell virtualenv which python to use here: Use different Python version with virtualenv
And that brings me back to where I was. Understanding where your python files are and what they are called (e.g. python, python27, python3 etc) is necessary for this process. So hopefully the stuff at the top of my answer will help you to discover your pythons and then you can make use of VirtualEnv.