After installing Homebrew's Python onto a system with an established Apple Python, the last entries listed by sys.path using Homebrew's Python are
/Library/Python/2.7/site-packages
/usr/local/lib/python2.7/site-package
This is the reverse of the order I expect. Shouldn't Homebrew's packages be searched first? (In fact, shouldn't it be the only place searched?) That's what's implied in the documentation. Where is it set and how can I (or should I) change it?
Or is this the way Brewed Python is supposed to work? Is it designed to prevent duplicate packages and assume that any package in the system site-packages is meant to stay there unless explicitly uninstalled and then subsequently installed (into Brew's); with the exception of pip and setuptools which are duplicated (and put first in Brewed Python's path).
That is the intended behaviour. The rationale behind it is that you can keep using your old installed modules despite the fact that you are now using a new homebrewed Python.
Now this has some drawbacks, for example some libraries like numpy, won't work across different Python versions, so if you had installed numpy it will be imported from the old system's site-packages and won't work.
There are at least two ways to change sys.path:
Use a .pth file:
Python will pick that from some of the builtin locations (ex: ~/Library/Python/2.7/lib/python/site-packages/homebrew.pth). This appends to sys.path which is not ideal but has the advantage that it won't be picked by Python 3. It is currently the recommended method. You can achieve this with:
echo "$(brew --prefix)/lib/python2.7/site-packages" > ~/Library/Python/2.7/lib/python/site-packages/homebrew.pth
Set PYTHONPATH:
This gets prepended to sys.path, it has the drawback that is global to all python versions so it is not recommended if you are going to use different python versions. You can do it by adding to your .bash_profile:
export PYTHONPATH=`brew --prefix`/lib/python2.7/site-packages:$PYTHONPATH
I personally used option 2 with homebrew-python (I now use and recommend Anaconda). My reasons were that I didn't care about system's Python or Python 3 at the time.
Related
Recently I've installed Python 3 after using 2 for quite a while, so I have many libraries for Python that I've installed using pip. I've already started using the symlink python3, but what else do I need to do to change over?
For example, while I have made the alias python=python3, I don't know about the modules I've installed with pip. I was using a version of pip for Python 2, but does that mean modules were written for Python 2 and not 3? Do I now need to start using pip3 (or make a corresponding alias pip=pip3 or pip=pip3.5)? If modules were installed using the command pip, does that mean I have to reinstall them for Python 3 using pip3? If I do need to reinstall them for 3, should I start removing them from my Python 2 env as cleanup?
And another question that seems like it may entail a lot of work: How should I change all my virtual environments for Python projects? They were all Python 2 envs, and now I'm not sure whether it's necessary, useful, or possible to reinstall or upgrade the modules within them to the corresponding ones for Python 3.
Sorry if this seems like a duplicate question - I've searched and only found resources regarding "how to upgrade" rather than "what to do after upgrading". Thanks for your help!
It is strongly recommended not to symlink python to Python 3, at least on Debian-based Linux distros. Debian utility scripts use both Python 3 and 2.7 to work (which means you already had a copy of Python 3 somewhere...), and as a consequence Debian-based distributions ship both versions. Symlinking can cause unexpected breakages. This may or may not be true on other systems, however, but with such ambiguity it is probably useful not to try.
Instead, live with symlinking python3 to Python 3. Leave python alone.
pip is usually symlinked to the corresponding pip program for 2.7. Use pip3 to specify you want to use the pip program for Python 3. If you use pip, you'll find modules for Python 3 being installed to Python 2.7 site directories and nothing will work as you expect it to. There should be no reason why you can't symlink pip to pip3, but I'd still not recommend it - it's better to tread cautiously in this regard.
You don't need to change your current virtual environments. Just create new ones with virtualenv, pointing to the correct Python program to use. This is the whole point of virtual environments: to sandbox different Python versions, so that neither of them adversely interact with each other.
If you have projects you really want to ship to Python 3, I'd recommend using a version control system to back up your current files, and then reproducing them in a new virtual environment configured for Python 3. This is quick, simple and painless.
I have, in /usr/local/lib/python2.7/site-packages multiple versions of the same package.
E.g. I have django-angular-0.7.13-py2.6.egg, django-angular-0.7.13-py2.6.egg-info, and django-angular-0.7.13-py2.7.egg.
Is it safe to delete the two files that are, ostensibly, the wrong version?
When I got into the python interpreter, import the package/module, it tells me it's run from <module 'django_angular' from '/usr/local/lib/python2.7/site-packages/django-angular-0.7.13-py2.6.egg/django_angular/__init__.py'>...
I'm concern I'll irreparably damage my python packages, as is so easy when you mess with these things.
All I could find on the topic is this article, but that's windows specific, and doesn't address having multiple versions of the same thing.
You shouldn't go and delete the files manually, although it is probably safe. It is quite unlikely that your system depends on Django and Python always uses only one version of the installed packages anyway.
However, I would strongly recommend keeping your development environment separated from your system packages. If you haven't already, take a look at pip and virtualenv. Here is one tutorial for them.
With pip you can also uninstall your system libraries if you really want to do that. However, pip only sees one version at a time, so if you want to uninstall the above packages you have to run pip uninstall multiple times. But after all the versions are gone you can install the version you really want.
Also, a better option for displaying the installed versions is to use yolk. With that you don't have to browse the site-packages manually.
I have spent days struggling to get a scientific Python environment running on Mac OS Lion. I tried the SciPack Superpack route, and also various manual installations via pip and easy_install, but still got errors trying to import or use various modules. Based on the advice in this Stackoverflow thread I set up a fresh installation using MacPorts.
However, when I run the macports Python, it is ignoring packages in the macports installation and instead trying to load incompatible packages from the old installation. I am absolutely sure that I am running the newly installed macports Python. I have checked the symlinks and have checked python_select and have launched Python by directly typing the path to the new installation. But when I try to import, say, statsmodels, it pulls in the old version from another directory.
Here are the contents of sys.path (edited for brevity):
['',
'/Library/Python/2.7/site-packages/mrjob-0.4.3_dev-py2.7.egg',
'/Library/Python/2.7/site-packages/statsmodels-0.6.0-py2.7-macosx-10.9-intel.egg',
...
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
'/Library/Python/2.7/site-packages/twilio-3.6.6-py2.7.egg',
'/Library/Python/2.7/site-packages/six-1.6.1-py2.7.egg',
'/Library/Python/2.7/site-packages/httplib2-0.9-py2.7.egg',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/readline',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyObjC',
'/Library/Python/2.7/site-packages']
The macports installation is in /opt/local, while the older installation is under /Library/Python. As you can see, the older packages are higher up in the list, which means that they have higher priority.
Environmental variable PYTHONPATH is empty. If I do put anything into PYTHONPATH, that appears in sys.path after the /Library entries but before the /opt entries. So it does not solve the problem.
If I invoke the new python with the -S option, sys.path becomes:
['',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload']
That succeeds in getting rid of the alien site packages entries, but it also nukes macports' site-packages entries so then I can't load anything.
I believe that the culprit is a file called /Library/Python/2.7/site-packages/easy-install.pth , with the following contents (again edited for brevity):
import sys; sys.__plen = len(sys.path)
./mrjob-0.4.3_dev-py2.7.egg
...
./statsmodels-0.6.0-py2.7-macosx-10.9-intel.egg
...
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
./twilio-3.6.6-py2.7.egg
./six-1.6.1-py2.7.egg
./httplib2-0.9-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
If I rename that file before launching macports python, python no longer is adding these alien packages to its sys.path. Now sys.path looks a lot like it does when I use "python -S" option except it has these additional entries at the end:
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyObjC',
'/Library/Python/2.7/site-packages'
Now I am able to load macports' own packages, say for example statsmodels. By checking statsmodels.__file__ I have confirmed that it is importing the local package and not the alien package.
However, I consider this a workaround/kludge rather than a real solution. A solution should make it behave as expected, which is: Macports' Python, launched from /opt/local/... installation, should prioritize packages installed via macports into the /opt/local/... directory tree and go looking elsewhere only if the module does not exist locally. So for example I would expect the /opt/local entries to come first in sys.path, with the /Library entries further down the list.
It seems like this should be the default behavior and I've seen a lot of comments here on Stackoverflow that just assert that this is how Macports Python behaves. So...how do I make it so?
There are at least two separate root causes for the problem you are seeing. One is that non-system OS X framework builds of Python 2.7, including MacPorts Python 2.7, deliberately include the system Python site-packages location, /Library/Python/2.7/site-packages, in sys.path, normally at the end of sys.path after that instance's own site-packages directory. This has been a debated feature in upstream Python and there is an open MacPorts issue to remove it (https://trac.macports.org/ticket/34763), the argument being that it is better to keep the system Python and the MacPorts Python totally separate.
The second root cause is the behavior of the original setuptools package and its easy_install command as supplied with OS X. As you have discovered, it does some fancy manipulation of sys.path by some magic tricks with .pth files, including easy-install.pth, to ensure that the packages that have been installed using easy_install show up first in sys.path and override other installed versions of those packages. Also as you have found, one way - and the easiest way - to remove that behavior is to delete the easy-install.pth file in /Library/Python/2.7/site-packages. That is assuming you don't want to use any of the packages installed there with the Apple-supplied system Python.
The long-term strategy to avoid this problem in the future is to make sure you do not use the Apple-supplied easy_install or easy_install-2.7 commands found in /usr/bin. They will install packages to /Library/Python for use by the system Python and will create or update easy-install.pth. In general, you should avoid use of easy_install all together. Its modern replacement is pip which provides better control and avoids tricks with .pth files. If MacPorts doesn't already provide a port for the Python package you want to install, usually of the form py27-xxxx, install and use the MacPorts py27-pip port instead of easy_install:
/opt/local/bin/pip-2.7 install xxxx
You can also use the usual MacPorts features like port select pip pip27.
If I install all packages using
python setup.py install --prefix=~/.local
how can I make Python read my packages from there and not from the system wide version? I tried editing PYTHONPATH to put ~/.local/lib/python2.x/site-packages/ first, but it does not help.
I thought that ~/.local is guaranteed to be read first in Python 2.6 and later versions. Is this true? Is something special required to make it true? That would solve the issue. Right now it seems that PYTHONPATH paths get incorporated into sys.path but are in the list after the system-wide site-packages directories, making the system wide version get used instead of the one in ~/.local.
Using pip is not an option unfortunately.
It is recommended that you use virtualenv with a proper activation script which will take care of settings up PYTHONPATH properly, without messing up system-wide Python installation in any way
http://www.virtualenv.org/en/latest/index.html
First: I'm running Macports. No problems with that, except:
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin
which is the value of sys.exec_prefix, for my macports python even though:
/opt/local/lib/python2.6/site-packages/
seems to be quite a logical place to put things, /opt/local being the macports --prefix, as it were. Why does easy_install put things in this odd Frameworks/Python.framework thing?
More importantly, can i use the methods here, to ensure that all my systemwide python, particularly the scripts which I really want in /opt/local/bin, things I use all over the place like (i|b)python for example are accessible?
So first I'd make the observation that the directory
/Library/Frameworks/Python.framework/Versions/2.6/...
is where you can find the OS X provided version of Python so I'd guess that what the MacPorts developers want to do is replicate the OS X Python directory structure but keep it as far away from the OS X version of Python as possible. Also I think replicating the structure allows you to install third-party extensions outside of the MacPorts modules. I've done that before so I know it's possible, its just you still need to make sure:
When installing the modules, the version that the installer encounters when doing a PATH search is the Python installation you want to install it to (ie. a module meant for MacPorts Python doesn't end up in OS X Python). The page you cited earlier spells out the procedure an installer goes through when trying to find a Python installation to install to so you have some hints there about what to look into if you have problems installing third party modules.
Once you have installed your modules you'll have to keep track of them yourself because MacPorts won't do that for you. On the bright side everything should be in the site-extensions folder of the Python installation, I think.
This question from the MacPorts FAQ kind of lays the groundwork for their philosophy of directory organization I think. Also this question too.