I installed IPython and Python 3 by using Homebrew on a clean macOS Catalina (virtual machine).
$ brew install ipython
As the ipython package is dependent on the python3 package, Homebrew installs ipython and python3 together.
$ brew info ipython
ipython: stable 7.13.0 (bottled), HEAD
Interactive computing in Python
https://ipython.org/
/usr/local/Cellar/ipython/7.13.0 (2,905 files, 21.8MB) *
Poured from bottle on 2020-04-15 at 18:48:22
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/ipython.rb
==> Dependencies
Required: python ✔, zeromq ✔
==> Options
--HEAD
Install HEAD version
==> Analytics
install: 11,543 (30 days), 33,591 (90 days), 98,995 (365 days)
install-on-request: 5,404 (30 days), 15,768 (90 days), 49,364 (365 days)
build-error: 0 (30 days)
I expected that both of these two commands read PYTHONPATH from my shell environment, because ipython works so when it is installed by pip3 install ipython.
However ipython and python3 installed by using Homebrew have different sys.path settings.
$ which ipython
/usr/local/bin/ipython
$ which python3
/usr/local/bin/python3
$ ipython
Python 3.7.7 (default, Mar 10 2020, 15:43:33)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import sys; sys.path
Out[1]:
['/usr/local/Cellar/ipython/7.13.0/libexec/bin',
'',
'/usr/local/Cellar/ipython/7.13.0/libexec/lib/python3.7/site-packages',
'/usr/local/Cellar/ipython/7.13.0/libexec/vendor/lib/python3.7/site-packages',
'/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python37.zip',
'/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7',
'/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload',
'/usr/local/lib/python3.7/site-packages',
'/usr/local/Cellar/ipython/7.13.0/libexec/lib/python3.7/site-packages/IPython/extensions',
'/Users/oxon/.ipython']
$ python3
Python 3.7.7 (default, Mar 10 2020, 15:43:33)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; sys.path
['', '/Users/oxon/root-6.20.02/obj/lib', '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python37.zip', '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/site-packages']
Q1. Why are they different?
Q2. Is this an expected behavior?
Q3. Why IPython reads PYTHONPATH when it is installed by pip3 install ipython?
It comes down to the following line in the Homebrew recipe for ipython:
bin.env_script_all_files(libexec/"bin", :PYTHONPATH => ENV["PYTHONPATH"])
It creates the /usr/local/bin/ipython script that sets PYTHONPATH to a fixed value that is needed for IPython to work properly (since its modules are outside the default Python module directory) before calling the ipython executable:
#!/bin/bash
PYTHONPATH="/usr/local/Cellar/ipython/7.14.0/libexec/lib/python3.8/site-packages:/usr/local/Cellar/ipython/7.14.0/libexec/vendor/lib/python3.8/site-packages" exec "/usr/local/Cellar/ipython/7.14.0/libexec/bin/ipython" "$#"
It simply overrides your set value of PYTHONPATH. You may modify the script to read:
PYTHONPATH="$PYTHONPATH:/usr/local/Cellar/...
This will make it append to your PYTHONPATH instead of completely overriding it, but the file will be overwritten when the Homebrew package is updated or reinstalled. Therefore, it is advisable to instead put in the IPython startup directory ~/.ipython/profile_default/startup a script, say root.py that looks like:
import sys
sys.path.append('/Users/oxon/root-6.20.02/obj/lib')
When you install ipython using pip, its modules go into the default Python module directory, be it the system one or the user one or the one in the virtual environment, and it is not necessary to mess with PYTHONPATH for ipython to function correctly.
I'd say the answer to Q2 is therefore that it's probably a bug and you may submit an issue with Homebrew on GitHub, although I wouldn't hold my breath given how they treated the same issue with Jupyter.
Q1. Why are they (i.e., sys.path) different?
The way sys.path is populated is typically:
1) the current working directory, followed by 2) directories listed in your PYTHONPATH environment variable, followed by 3) installation-dependent default paths, which are controlled by the site module.
From https://docs.python.org/3/using/cmdline.html : PYTHONPATH: Augment the default search path for module files.
See also e.g.,
https://leemendelowitz.github.io/blog/how-does-python-find-packages.html
https://realpython.com/lessons/module-search-path/
https://learn.microsoft.com/en-us/visualstudio/python/search-paths?view=vs-2019
Therefore, each python installation may have its own sys.path.
In addition, sys.path does not need to be equal to PYTHONPATH
Check in each of the two (or three?) cases also the value and post back:
import os
print( 'PYTHONPATH = ', os.environ['PYTHONPATH'] )
For instance, I have spyder3 and python3.8 under Ubuntu 20.04. When started at the same CLI each has its own sys.path, both "derived" from the same PYTHONPATH.
Q2. Is this an expected behavior?
Yes.
Q3. Why IPython reads PYTHONPATH when it is installed by pip3 install ipython?
Please post the value of PYTHONPATH ($ echo $PYTHONPATH) prior to executing either of the pythons at the CLI. It is not evident that they are or are not reading its value.
Please post back if this helps finding the cause.
Trying to breathe some life back into a django package that has fallen into a state of disrepair. They use tox for testing so I've setup pyenv on my MacBook. I've installed 3 versions of python as you can see below, and everything looks like it should work, but if it was I wouldn't be asking why it is not.
I've replaced my home directory with ~ to make it a bit easier to read.
pyenv was installed with brew install pyenv and the various versions of python were installed with pyenv install #.#.#
The shims exist:
$ echo $PATH
~/.pyenv/shims:~/.platformsh/bin:/usr/local/sbin:...
$ which python3.6
~/.pyenv/shims/python3.6
$ which python3.4
~/.pyenv/shims/python3.4
$ which python3.5
~/.pyenv/shims/python3.5
But executing them does not work as expected:
$ pyenv local 3.4.9 3.5.6 3.6.8
$ python3.4
Python 3.4.9 (default, Feb 12 2019, 10:33:47)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
$ python3.5
pyenv: python3.5: command not found
The `python3.5' command exists in these Python versions:
3.5.6
$ python3.6
pyenv: python3.6: command not found
The `python3.6' command exists in these Python versions:
3.6.8
And tox fails like this:
py34-1.11: commands succeeded
ERROR: py36-1.11: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError("Failed to get version_info for python3.6: pyenv: python3.6: command not found\n\nThe `python3.6' command exists in these Python versions:\n 3.6.8\n\n", None)
ERROR: py36-2.0: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError("Failed to get version_info for python3.6: pyenv: python3.6: command not found\n\nThe `python3.6' command exists in these Python versions:\n 3.6.8\n\n", None)
py36-latest: commands succeeded
docs: commands succeeded
But in the .tox folder you will find these VirtualEnvs that can be activated manually.
$ ls .tox
dist docs flake8 log py34-1.11 py36-1.11 py36-2.0 py36-latest
Because at some point it was working....
I do understand the mechanics of why it isn't working, what I don't understand is why pyenv is not setting up the environment correctly (Or maybe this is exactly how it is supposed to behave). Everything I read seems to indicate that python3.6 should launch a python3.6.8 interpreter
$ bash -x python3.6
+ set -e
+ '[' -n '' ']'
+ program=python3.6
+ [[ python3.6 = \p\y\t\h\o\n* ]]
+ export PYENV_ROOT=~/.pyenv
+ PYENV_ROOT=~/.pyenv
+ exec /usr/local/Cellar/pyenv/1.2.9/libexec/pyenv exec python3.6
pyenv: python3.6: command not found
The `python3.6' command exists in these Python versions:
3.6.8
pyenv by default picks the python "locally", that is it looks for the PYTHON_VERSION environment variable or a .python-version file.
Personally I find this setup a little cumbersome (needing to have these files littered around all projects, especially in projects which need multiple versions). Fortunately, you can make these "shims" function anywhere with a default version of python by using pyenv global #.#.#
In your case, to make the python3.6 shim execute 3.6.8 without needing to set up the .python-version files, you'd run pyenv global 3.6.8 -- you can run this multiple times for different python versions as well: pyenv global 3.6.8 3.5.6 ...
The reason that you're likely not having these resolve inside tox is tox clears the environment when executing, so the PYTHON_VERSION environment variable will not carry through. You can turn that off by setting passenv= in your tox.ini. For example:
[testenv]
passenv = PYTHON_VERSION
In my ubuntu 16.04 system I have the following pyenv versions
$ pyenv versions
* system (set by /home/myuser/.pyenv/version)
3.6.0
3.6.0/envs/general
general
I intend that the version named system should coincide with the real system one, but when I check which python version will be executed, I get:
$ python --version
Python 3.6.0
$ which python # expecting /usr/bin/python
/home/myuser/.pyenv/shims/python
instead of the expected /usr/bin/python symlink which points to python2.7
$ ll /usr/bin/python
lrwxrwxrwx 1 root root 9 apr 18 10:11 /usr/bin/python -> python2.7*
Why is that? Is there something that I understand wrong?
How do I reset the pyenv system to make it coherent with the real system? I want to be sure of using the real system python when executing normal commands in my home dir.
I'm having a strange problem with a virtualenv I've created for python 3. I went through the usual steps:
$ virtualenv --python=/opt/local/bin/python3.3 .py3
$ source .py3/bin/activate
The problem I'm having is that when I call python --version, it is still reporting 2.7.5, even though the paths all appear to be set up correctly. My virtualenv was created in /Users/barry.flinn/projects/.py3/bin, and I get thes results when I run which python:
$ which python
/Users/barry.flinn/projects/.py3/bin/python
The bin folder has the following python executables:
lrwxr-xr-x 1 barry.flinn obfuscated\Domain Users 9 Sep 23 19:39 python -> python3.3
lrwxr-xr-x 1 barry.flinn obfuscated\Domain Users 9 Sep 23 19:39 python3 -> python3.3
-rwxr-xr-x 1 barry.flinn obfuscated\Domain Users 9100 Sep 23 19:39 python3.3
Clearly, invoking python should give me python 3.3, and yet it still seems to revert to the system python, which is 2.7.5. My $PATH when the virtualenv is active is:
/Users/barry.flinn/projects/.py3/bin:/usr/local/mysql/bin:/opt/local/lib/postgresql92/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/go/bin:/usr/local/munki
Since all of this seems correct, I'm stumped as to what is going on here.
Update:
This reports correctly:
$ env python --version
Python 3.3.2
Which, to me, is slightly more baffling.
If you have a shell alias defined for the python interpreter, it will override the python chosen for your virtualenv. For example:
user#x790:~/temp$ alias python=/usr/bin/python
user#x790:~/temp$ python --version
Python 2.7.4
user#x790:~/temp$ virtualenv --python=/usr/bin/python3 foo
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in foo/bin/python3
Also creating executable in foo/bin/python
Installing distribute.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done.
Installing pip................done.
user#x790:~/temp$ . foo/bin/activate
(foo)user#x790:~/temp$ python --version
Python 2.7.4
Note that the python version reported was 2.7.4, even though python3 was selected for the virtualenv. Using the env command circumvents the alias:
(foo)user#x790:~/temp$ env python --version
Python 3.3.1
Lastly, you can get around this issue by either temporarily disabling the python alias for a single command or permanently undefining it:
(foo)user#x790:~/temp$ \python --version
Python 3.3.1
(foo)user#x790:~/temp$ unalias python
(foo)user#x790:~/temp$ python --version
Python 3.3.1
I'm trying to get python3 working and all setup while keeping the python2 libraries in tact (system dependencies and all that).
I've installed Python3.3 (compiled, etc..) and its in /usr/bin/Python3 side by side with /usr/bin/Python2.x
Entering 'python' will take me to a 2.6 IDE and 'python3' or 'python3.3' does nothing. I'm just a little confused, I'm not sure why this isn't working at the moment.
I can actually call it with /usr/bin/python3/bin/python3 and it works. This doesn't look too pretty though, what should I do here?
$ pwd
/usr/bin/python3/bin
$ ls
2to3 idle3.3 python3 python3.3m pyvenv
2to3-3.3 pydoc3 python3.3 python3.3m-config pyvenv-3.3
idle3 pydoc3.3 python3.3-config python3-config
$ cat /proc/version
Linux version 2.6.34.7-56.40.amzn1.i686 (mockbuild#build-31003.build) (gcc version 4.4.4 20100525 (Red Hat 4.4.4-5) (GCC) ) #1 SMP Fri Oct 22 18:48:33 UTC 2010
The first thing that comes to mind is that python3 didn't automatically export its path. I think that adding
export PATH="/usr/bin/python3/bin:$PATH"
to .bash_profile or .bashrc and restarting terminal could do the trick.