I have just spent 2 days trying to build Tensorflow from source, and finally succeeded when I realized that sudo pip (even with the -H flag) was not finding my anaconda pip, but instead finding a pip installed with apt. Running, then, sudo -H ~/anaconda3/bin/pip ... fixed my problem.
In order to avoid this kind of issue ever again (I had several issues in this process with the "wrong" python being used), is it possible for me to completely remove python from my system, keeping only Anaconda? Is it advisable?
No, you can't. Python is an essential part of ubuntu (and of just any linux distrib nowadays), so removing it will just break your system. Well, if you want a broken system then it's of course up to you ;)
The right solution is to use virtualenvs for your own workspaces instead, so you can have different isolated installs with different python and 3rd part packages versions. This also avoids breaking anything when two projects depends on different versions of a same package. For a local use you probably want to install virtualenvwrapper too (it's an addon over virtualenv that makes working with virtualenvs easier - switching between virtualenvs, running hooks on env switch etc).
Oh and yes: those virtualenvs are per-user so no more sudo and you don't risk messing up your system install (thx Attie for mentioning this).
And to be even more exhaustive: darthbith mentions in a comment that anaconda has it's own virtualization/package management system conda that might make virtualenv redundant and is anyway a best fit if you're using anaconda.
This is not just a Ubuntu issue but also a linux world wide issue. The system python is at the core of apt-get and yum package managers. Also the modern grub is based on python so removing it can make your machine unbootable.
In short, this will affect RHEL related distributions (CentOS/Fedora) and Debian related distributions (Debian/Ubuntu).
Related
Background
Currently, I have Python 2.7.17 and Python 3.8.1 installed on my Mac (v 10.14.6). Both Python versions were downloaded directly from Python.org using a "macOS 64-bit installer" .pkg file, in late 2019. Both were installed here:
python2: /Library/Frameworks/Python.framework/Versions/2.7/bin/python2
python3: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3
Currently, I use only pip (either pip2 or pip3, depending) to install python packages, which are saved here:
python2: /Library/Frameworks/Python.framework/Versions/X.X/lib/python2.7/site-packages
python3: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages
I am about to begin an online course which requires that I install miniconda in order to install the packages that I need for the course. But after having a very bad experience having Anaconda installed on my machine years ago, and it taking me months to remove it and re-create a clean Python install, I am nervous. I don't want my machine to become a mess again, rife with error messages that it can't find this or that python version or this or that python library.
Question(s)
Assuming there will now be multiple installations of Python on my machine (in addition to the versions that I already have, since I believe yet another version of python is installed along with conda), and there will be new places where python packages will be saved, what can I expect to change about how I write python after I install miniconda?
Is there anything I will need to do differently from what I am doing now when I write python code?
How can I ensure that everything remains working and compatible and I avoid the fate I suffered many years ago?
Happy to edit/clarify this question per user suggestions. Thanks!
Use conda environments and install Python into those environments. This ignores, by design, the versions of Python that are installed at the system level. Also, what's installed into environments (other than the one you're in) is ignored. As somebody who uses conda for all Python development, I don't even know what I have installed in /usr/local/bin/ or other places. Everything is in /path/to/miniconda3/envs/env_name_n. You don't have to uninstall other versions of Python you have elsewhere; the point of environments is to keep everything isolated.
What can I expect to change about how I write python after I install miniconda?
Is there anything I will need to do differently from what I am doing now when I write python code?
Use conda environments and use conda as your package manager. For almost every major project, it nearly a drop-in replacement for pip. Outside of managing dependencies, not much changes.
It seems I have several Python versions installed on my Mac (High Sierra). Do I really need more than one each of Python 2.7 and the most recent Python3?
I am primarily concerned about confusion when adding modules to the environments.
Here are the python executables I can find, excluding symlinks. Do I need them all? Does Apple need one version just for the OS? (Maybe that's Ruby I'm thinking of._)
/usr/bin/python
/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/bin/python
/usr/local/Cellar/python/2.7.13/bin/python
/usr/local/Cellar/python/2.7.13/share/python
/usr/local/Cellar/python/3.7.1/libexec/bin/python
On MacOS, you need to have the system-installed Python 2 in /usr/bin/python because some system functionality depends on it.
If you have other versions installed via Homebrew or etc, that's strictly optional. For developing code in Python, definitely install a recent version of Python 3; you should not be developing new code in the legacy version 2. It will run out of support in a year or so, and many important third-party libraries etc have already abandoned support for it.
If you do need to maintain Python 2 code, having a Homebrew version of the most recent Python 2.7.13 is probably a good idea. The system /usr/bin/python is at version 2.7.10 as of MacOS Mojave; and if we extrapolate from Apple's update policy history for third-party open-source software in the system, it will probably remain there for many years.
Homebrew is just one of many ways to install additional Python versions, but since this is what you seem to be using already (as indicated by the /usr/local/Cellar paths) I will not go into alternatives.
For switching between versions, virtualenv is the common baseline, though there are many add-ons like pyenv etc which add convenience features or alternative models for switching between projects which require different versions of Python and various packages.
Note: this answer might be primarily based opinioned.
First, in macos, I encourage you to use brew to install python. So, if you want to instal Python 3 run:
$ brew install python
Probably it is already installed, so to update it run:
$ brew upgrade python
This will upgrade to the latest version of Python 3. In general, if you start a new project you should use Python 3. Remember that Python 2 will be deprecated at the end of this year 2019.
Second, I highly recommend you to use virtual environments to avoid having issues with dependencies. You can create a virtual environment running:
$ python3 -m venv .ven
This will create a virtual environment in .venv in your current working directory.
As an alternative you can use pipenv to track dependencies too which I also encourage you to install using brew.
$ brew install pipenv
Then you create a virtual environment running:
$ pipenv --three
Keep in mind that you will have two versions of Python in your machine python which is Python 2 (installed by default in your machine) and python3 which is Python 3. Then, if you want to run Python 2 just type python, while if you want to run Python 3 type python3.
Finally, you can also keep track of several python versions using pyenv. This is very useful if you need to use multiple versions of Python for example Python 3.1, 3.2, 3.3..., 3.6, 3.7.1, 3.7.2, 3.7.3, etc.
This is what has been useful in my experience using macOS. You could also use Docker to completely isolate your environment, but probably the first two points is enough if you are a beginner.
While Python 3.x is what python solutions should be implemented in today, many existing python applications, modules, etc, are implemented and maintained in 2.7 or lower. Sometimes you'll find multiple versions of python on your computer because applications you've downloaded in the past had you install them so their application would work properly on your machine. As long as you don't want, or need applications that may depend on past versions of python, you can update to the latest one and delete the old versions.
If you are planning on developing in python, and you can use the latest version, you should definitely do so. If you rid of the versions of python on your machine, applications, modules/scripts, etc may not work or function properly.
Python comes pre-installed on Mac OS X, but it is not required to be on your machine for your computer to function, just the applications that may be on your machine that may use Python.
So, I have seen What is the difference between pip and conda?. However, all of the answers there appear to be from Anaconda supporters. So, that made me wonder: why is pip still the standard? why hasn't everyone just moved to anaconda?
I understand that anaconda only works with its own python, but is that the only disadvantage?
Based on my limited experience, I would guess that the main advantage of pip over conda is the ability to still install packages that are not available from conda or Anaconda.org.
https://conda.io/docs/using/pkgs.html#install-non-conda-packages - says basically the same.
I have been using conda for a while now, mostly studying Machine Learning and related subjects. I am a happy user 99.99% of the time. But when one faces challenges like building and installing tensorflow with GPU support for Mac that would support his or her rather specific/outdated GPU, one can't really rely on conda.
One huge advantage of pip is the built-in ability to install packages system-wide via f.ex.
sudo -H pip install ipython
It actually is smart enough to do this by default if run as the root user, installing to some directory in the global execution path. (/usr/local/bin?)
What can actually be considered an advantage for some things is that pip compiles packages (by default). So some packages like f.ex. theano which are actually optimized upon installation should not be installed via conda, or you are possibly missing out on this.
Finally, as mentioned, pip is directly linked to Python's package archive, whereas conda assumedly needs to be told when a new package was uploaded via a new configuation.
I have three different Python 2.7s at:
/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
I use a number of packages that come from different sources. I am currently installing packages from port (MacPorts), easy_install, pip (installed by easy_install), and Mercurial. There are also some that I have to install from image or build from source. I have more control over those.
The problem is that easy_install and pip seem to be installing to one location (/Library/Frameworks/...) and MacPorts installs to another (/opt/local/Library/Frameworks/...).
What's my best action now? Delete /Library/Frameworks/.../python2.7 and move easy_install and pip to the MacPorts one at /opt/local/...? Link the two directories? Move the MacPorts installation to /Library/Frameworks/...?
How can I consolidate these Pythons? I have tried putting both site-packages locations in my path, but only certain packages are available only for one Python and not the other and others vice versa, and I need them all available at once.
It seems that you have control over the stuff you're building yourself. This is how I consolidate macports with pip:
I like using Macports for all my stuff, so I just make sure that pip and easy_install build into macports' installation of python (the one in /opt/local/...).
You can tell where pip and easy_install will install things by using:
readlink `which pip`
(those are backticks)
If you want pip to install to the macports direcectories, use macports to install pip:
sudo port install py-pip
Then, be sure that which pip points to something like:
askewchan#rock:~$ which pip
/opt/local/bin/pip
askewchan#rock:~$ readlink `which pip`
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pip-2.7
From the comments below (thanks #Jonathan and #Ned) you can do the same with easy_install but its port is called py-distribute:
sudo port install py-distribute
But as far as I know, you never need to use easy_install because anything that can be easy_installed can be piped better.
Note the port descriptions:
askewchan#rock:Tracking {master *}$ port search *easy*install*
py-pip #1.2.1 (python, www)
An easy_install replacement
askewchan#rock:Tracking {master *}$ port search py*distribute
py-distribute #0.6.35 (python, devel)
Replacement for setuptools
I suggest deciding on one and only one Python for your development work ( personally, I use distribution from Python.org )
You can't get rid of /Library/Frameworks - that's the default OSX one, and you could break things
of the two remaining Pythons, I'm assuming one is Macports and the other is Python.org -- you need to choose which one you want to be your development env and to stick with that.
I would strongly recommend against using pip or easy_install from one Python to install modules for another. The reason is that there can be differences in the compile options. It can be hard enough as-is to get certain packages to compile on OSX properly -- if you start compiling against different binaries ( which might support different architectures ) you're just going to increase your headaches.
I personally chose the following path:
I use the Python.org package for all development.
On a terminal login, I run shell scripts to prioritize my Python choice
All of my projects have their own virtualenv , and I disable system packages
When starting to work on any project, I tend to have an environment setup script. I just type in go_myproject.source ; that cds me to the right directory and runs the source /path/to/virtualenv/bin/activate to get me set up for that project.
There's a tiny bit of overhead on getting things setup, but I have been in complete heaven ever since. Managing projects and not needing to worry about dependencies/upgrades for one thing killing something else is... blissful.
While not a general solution, I install Mercurial and other Python-based applications using virtualenv. In particular, pip and easy_install will install to the respective virtual environment only and not clutter any system folder. The downside is, of course, that I will have duplicates of some packages; the advantage is that I have a clean, self-contained environment with a known version of Python (which for things such as Mercurial and other mission-critical applications is more important for me).
Another downside is that I need to link individual applications to my personal bin directory or add the bin directories of the virtual environments to my path. (Personally, I manage this with some simple scripts that do the symlinking for me.)
I sugest to move all python installations to the one place and create symlinks.
After that configure python environment to avoid problems with imports and "visibility" of the modules.
Try to use commands:
# easy_install
env PYTHONPATH=/custom/path easy_install –install-dir /custom/path
#pip
pip install --install-option="--prefix=$PREFIX_PATH" package_name
Can someone please explain to me what is going on with python in ubuntu 9.04?
I'm trying to spin up virtualenv, and the --no-site-packages flag seems to do nothing with ubuntu. I installed virtualenv 1.3.3 with easy_install (which I've upgraded to setuptools 0.6c9) and everything seems to be installed to /usr/local/lib/python2.6/dist-packages
I assume that when installing a package using apt-get, it's placed in /usr/lib/python2.6/dist-packages/ ?
The issue is, there is a /usr/local/lib/python2.6/site-packages as well that just sits there being empty. It would seem (by looking at the path in a virtualenv) that this is the folder virtualenv uses as backup. Thus even thought I omit --no-site-packages, I cant access my local systems packages from any of my virtualenv's.
So my questions are:
How do I get virtualenv to point to one of the dist-packages?
Which dist-packages should I point it to? /usr/lib/python2.6/dist-packages or /usr/local/lib/python2.6/dist-packages/
What is the point of /usr/lib/python2.6/site-packages? There's nothing in there!
Is it first come first serve on the path? If I have a newer version of package XYZ installed in /usr/local/lib/python2.6/dist-packages/ and and older one (from ubuntu repos/apt-get) in /usr/lib/python2.6/dist-packages, which one gets imported when I import xyz? I'm assuming this is based on the path list, yes?
Why the hell is this so confusing? Is there something I'm missing here?
Where is it defined that easy_install should install to /usr/local/lib/python2.6/dist-packages?
Will this affect pip as well?
Thanks to anyone who can clear this up!
I believe Mike Orr's answer from the virtual-env mailing list seems to be the best. Note the OP published this question in both places.
Original content of mail:
Years ago Debian created /usr/local/lib/pythonVERSION/site-packages,
and compiled the Python binary to include it in the default search
path. Ubuntu followed Debian's lead as it normally does. The Python
developers did not like this because you'd get interference with a
locally-installed /usr/local/bin/python using the same site-packages
directory. Ubuntu finally decided to abandon site-packages and use
dist-packages instead, a name which they invented so it wouldn't
interfere with anything. The loing story is out there somewhere if
you google it, somewhere in the Python bug tracker or distutils SIG or
such.
The system works, at least if you use the Ubuntu virtualenv package.
Some people have had problems using a locally-installed virtualenv on
Ubuntu because the magic sys.path entries weren't being added or
something. I'm not sure about --no-site-packages because I never use
that option: I run PIL and mysqldb from the Ubuntu packages because it
can be hard to compile their C dependencies sometimes. (Need the
right header files, Python is ignoring the header files, etc.)
So Ubuntu Python packages go into
/usr/lib/pythonVERSION/dist-packages. Or that python-support
directory for some reason. Locally-installed Python packages go into
/usr/local/lib/pythonVERSION/dist-packages by default. Whenever I
install an Ubuntu 9.04 system I run:
$ sudo apt-get install python-setuptools (6.0c9)
$ sudo apt-get install python-virtualenv (1.3.3)
$ sudo easy_install pip
$ sudo pip install virtualenvwrapper
The virtualenvs work fine this way, although I haven't tried --no-site-packages.
I'm trying to spin up virtualenv, and the --no-site-packages flag
seems to do nothing with ubuntu. I installed virtualenv 1.3.3 with
easy_install (which I've upgraded to setuptools 0.6c9)
These versions are both in Ubuntu 9.04, so you're making it harder on
yourself by installing them locally.
and everything
seems to be installed to /usr/local/lib/python2.6/dist-packages
Yes
I assume that when installing a package using apt-get, it's placed in /
usr/lib/python2.6/dist-packages/ ?
Yes
Is it first come first serve on the path? If I have a newer
version of package XYZ installed in /usr/local/lib/python2.6/dist-
packages/ and and older one (from ubuntu repos/apt-get) in /usr/lib/
python2.6/dist-packages, which one gets imported when I import xyz?
I'm assuming this is based on the path list, yes?
sys.path is scanned in order. The only funny thing is that .pth eggs
get put earlier or later in the path than some people expect. But if
you're using pip for everything it can do (i.e. except to install pip
itself, precompiled eggs, and a snapshot of a local directory that's a
copy rather than an egg link), you won't have many .pth eggs anyway.
Why the hell is this so confusing? Is there something I'm
missing here?
It's not well documented. I figured it out by scanning the web.
Will this affect pip as well?
Yes, pip will automatically install to
/usr/local/lib/pythonVERSION/site-packages. Use "pip install -E
$VIRTUAL_ENV packagename" to install into a virtualenv.
I'd be tempted to hack it by making site-packages a link to dist-packages, but I guess this might affect other cases where you want to install some extension other than from the ubuntu dist. I can't think of another answer to 1 except tweaking virtualenv's sources (with both ubuntu and virtualenv being so popular I wouldn't be surprised to find tweaked versions already exist).
Re 2, if you're using /usr/local/bin/python you should use the /usr/local version of the lib (including site-packages) and conversely if you're using /usr/bin/python.
Re 3, there will be something there if you ever install an extension for /usr/bin/python from sources (not via easy_install or from ubuntu's distro).
Re 4, yes, earlier entries on the path take precedence.
Re 5, easy_install is easy only in its name -- it does so much dark magic that it's been carefully kept out of the standard python library despite its convenience because the consensus among us python committers is that deep dark magic for convenience is "easy" only on the surface.
Re 6, I think that's an ubuntu modification to easy_install -- if that's right then it's defined wherever Canonical or other ubuntu maintainers make their collective decisions.
Re 7, sorry, no idea -- I have no reasonably recent ubuntu at hand to check.
You really should not touch Ubuntu's Python installation unless you are building system admin tools, or building something that could be considered to be a new system service.
If you are using Ubuntu to develop or deploy Python applications, always build your own Python from source, tar it up, and use that for deployment. That way you will have all the directories in the right place and virtualenv will work normally. If you will deploy several Python apps on the server, then make your Python live in some place like /home/python or /opt/python or somewhere outside of your home directory. Make sure that you have write permissions for the developers group (users?) so that people can easily add packages.
This also allows you to have two tiers of packages. The ones that are your in-house standard tools can be installed in your Python distro and be part of the tarball that you deploy, and only the app-specific packages would be in a virtualenv.
Do not upgrade or modify the Ubuntu system installed Python.
Well I have a Ubuntu 9.04 and quickly tried setting up a couple sandboxes with site-packages and one without. And things are working fine.
The only difference in the approach I took is I used Ubuntu's python-virtualenv package (1.3.3). And presume that it is tweaked by Ubuntu team to suit Ubuntu setups.
To sum up disable easy_installed virtualenv for a while, use packaged python-virtualenv and see if that meets your expectations.
In fact we use similar setup for production without any problem. Rest is already answered by Alex.
Another way to fix it:
https://stackoverflow.com/a/17265840/202168
Have to remember to do that in each virtualenv where you need it, but doesn't rely on hacks or a special version of virtualenv