Today I decided to install python and the scipy stack manually, instead of using Anaconda (or Canopy) as I had previously done. I use homebrew on my mac and have python2 and python3 (2.7 and 3.6) installed via homebrew. But reading through the documentation, there are multiple ways to install the scipy stack and I want to know what the differences are. I have tested they independently and they all work.
From the Homebrew documentation:
python2 -m pip install numpy scipy matplotlib
python3 -m pip install numpy scipy matplotlib
These are the same two commands that the Matplotlib installation docuentation lists for how to install matplotlib through homebrew. Why does this use pip (the system Python 2.7.x's pip) instead of pip2 and pip3 respectively? Is it because you call python2/python3 first?
However, the SciPy documentation for installing these modules when using homebrew is different:
brew tap homebrew/science && brew install numpy scipy matplotlib
(NOTE: the matplotlib formula is located in the homebrew/science repository, which is why you need to use brew tap.)
Finally, from the command line readout when installing python2 and python3 via homebrew:
pip2 install numpy scipy matplotlib
pip3 install numpy scipy matplotlib
which are based on the following readouts:
Pip and setuptools have been installed. To update them
pip2 install --upgrade pip setuptools
You can install Python packages with
pip2 install <package>
They will install into the site-package directory
/usr/local/lib/python2.7/site-packages
See: https://docs.brew.sh/Homebrew-and-Python.html
...
Pip, setuptools, and wheel have been installed. To update them
pip3 install --upgrade pip setuptools wheel
You can install Python packages with
pip3 install <package>
They will install into the site-package directory
/usr/local/lib/python3.6/site-packages
See: https://docs.brew.sh/Homebrew-and-Python.html
So between four sources of documentation, there are three different ways to install scipy when using homebrew and they all work; but how is each different and should one be preferred?
From what I can tell, the first and third methods, which both invoke pip (pip2/pip3), are functionally equivalent - both invoke Homebrew's Python X.X.X's pip - but one implicitly, the other explicitly. I assume this means both methods install the pre-built binary packages from pip in the form of wheels. For the second method, I think it installs homebrew's own formulae for these packages (i.e. maintained separately by homebrew in it's repository).
If this is true, then I assume one should use the second method if you are using a version of python which is maintained by homebrew (i.e. installed via brew install python or python3). My reasoning is that if you later decide to install another formula via homebrew that has any of the scipy stack as a dependency, it will install those modules again from homebrew's repository if you installed them using pip previously.
As mentioned, I am not sure if my understanding is correct and I have not been able to find any answers, so any insights or confirmations would be appreciated.
Your analysis seems correct: variants 1 and 3 will install numpy/scipy from the python package index (PyPI) and will use pre-built wheels (if available for your platform, which they most likely are).
Variant 2 installs the brew formula.
As mentioned by #Evhz, the conda packages for numpy and scipy use the Intel Math Kernel library, which can provide significant speedups (not just on Intel processors) versus the packages installed from PyPI or brew, both of which are linked against OpenBLAS.
Concerning which method to prefer: it's not entirely straightforward.
Yes, on the surface, using brew to manage both the python interpreter and the python packages would seem consistent.
However, homebrew only provides formulae for a handful of python packages, so you'll end up needing to mix with pip in any case.
If you want performance, you go with conda, which will be managing both the interpreter and python packages.
However, also anaconda / conda-forge still have some catching up to do with PyPI, so you'll likely need to mix with pip again.
In the end, there is no perfect solution but as long as you knowingly decide for one, you're unlikely to run into issues.
Related
I am using python 3.6.0 within a venv. I would like to "pip install" matplotlib==2.0.0, however when I do this, pip seems to automatically grab the newest versions of all other required supporting packages for matplotlib. i.e. cycler 0.11.0, pyparsing==3.0.7, etc. These latest supporting package versions do not seem to work with the older version of matplotlib and it throws errors when attempting to import matplotlib.
How do I install matplotlib without pip attempting to install all its supporting packages automatically?
My current temporary solution is to go back and manually install each package before installing matplotlib but I'm sure I will run into this issue again so would like to find a better solution.
Pip has a built-in feature:
pip install matplotlib --no-dependencies
To exclude specific, you can put it in requirements file and pass it:
pip install --no-deps -r requirements.txt
I am trying to install numpy and some other python packages in their intel optimized versions. They are not distributed as wheels via PyPI, but only via anaconda cloud. But that index doesn't contain all the dependencies, so PyPI must also be used. The command to run is
python -m pip install -i https://pypi.anaconda.org/intel/simple --extra-index-url https://pypi.org/simple numpy
Unfortunately, there seems to be a version race, and if PyPI has a newer version of NumPy, I won't get the intel-version of NumPy. To check if I got the right NumPy version, I run python -c "import numpy; numpy.show_config()" and verify it has MKL support. For SciPy, I don't know of a corresponding command. I have tried this a few times now, and it is using a cached scipy version, so I don't get any information from the pip output specifying which repository it is using.
How can I see after the installation has been completed which repository was used?
When I pip install (or pip install --upgrade) packages that require numpy, they have a tendency to uninstall my existing numpy+mkl (which has a high enough version to satisfy the numpy version requirement). Afterwards, they install numpy without +mkl, which causes problems for other packages that do require MKL. An example for which this happens is gym (which has 'numpy>=1.10.4' in its install_requires in setup.py).
I understand that this is related to the +mkl suffix that probably somehow messes with the versions, and understand I can fix it afterwards by downloading and installing numpy+mkl from https://www.lfd.uci.edu/~gohlke/pythonlibs/, but it gets annoying to manually do this every time over again when upgrading a package like gym to a new version. Is there any way to prevent numpy+mkl from getting uninstalled during the pip install --upgrade?
For me, this is happening on Windows 10, Python 3.6. I did not yet check if the same happens on Linux, but would be interested in an answer for that too if it's different there.
My currently installed version of numpy+mkl (which often gets automatically uninstalled) is 1.13.3+mkl.
Using --upgrade-strategy, as suggested by cgohlke in a comment, addresses this problem. So, taking the example where we want to install gym from scratch without it replacing our existing numpy+mkl installation with regular numpy, the full command to run is:
pip install --upgrade-strategy only-if-needed gym
Or, if we just want to upgrade an existing installation, we also add --upgrade
pip install --upgrade --upgrade-strategy only-if-needed gym
I have found that some packages force pip to reinstall numpy. The safest (and only) way to ensure that numpy is installed with mkl (from conda) is to uninstall using conda and pip and then reinstall using conda:
conda uninstall numpy
pip uninstall nump
conda install numpy
After a failed attempt at a "streamlined" install of the SimpleCV framework superpack for Windows. I'm now working through a manual installation guide (which I'm OK with as I have more control over the installation and might finally learn about installing Python Packages properly in Windows!)
Rather than just blindly follow the guide I'm trying to understand each step, so I'm confused by this..
easy_install pyreadline
easy_install PIL
easy_install cython
easy_install pip
pip install ipython
pip install https://github.com/ingenuitas/SimpleCV/zipball/1.3
Why not easy_install pip as soon as possible then pip the other packages?..
easy_install pip {{{I intend to research and probably use get-pip.py here}}}
pip install pyreadline
pip install PIL
pip install cython
pip install ipython
pip install https://github.com/ingenuitas/SimpleCV/zipball/1.3
Is there a pitfall doing it this way? (My limited understanding is that it's always preferable to use pip rather than easy_install.)
I know this question relates directly to SimpleCV but I want to learn the correct approach for when I'm installing package collections in the future without the benefit of a guide.
pip fetches the source code of the packages you're trying to install and compiles them. So if you don't have a compiler installed and configured it will fail to do so for packages which contain extensions written in C, which in this case applies to pyreadline, PIL and cython.
easy_install uses the precompiled packages from pypi (at least for windows if they're available), which means you don't need to compile everything yourself.
For pure python packages it's no problem using pip instead of easy_install, and if you have a compiler and the neccessary build dependencies installed it should also work.
I believe the answer is that pip does not currently support the installation of binary distributions, i.e. Python packages that include pre-compiled C extension modules. easy_install does.
BTW, there is work afoot to provide replacements for pip (and easy_install) that will fully support binary distributions on all platforms. See here for an overview.
Hi recently i installed setup tools module and google app engine gives me errors . Is there a way to uninstall setuptool? can any one tell me step by step because i tried hard
The answer depends on how it was installed.
If it was installed using the ubuntu (debian) package manager, try:
sudo apt-get remove --purge python-setuptools
[updated]
If you installed manually, probably the setuptools final location will be something like (adjust for your environment/python version):
/usr/local/lib/python2.6/dist-packages
Just delete the setuptools stuff there.
Lame, I know, but it is your burden for not using the excellent package manager provided by ubuntu: stick to dpkg unless you need bleeding edge stuff. For other python modules installed by setuptools, it provides no "uninstall" feature (but pip does, that is why there is a lot of enthusiasm around virtualenv, pip and yolk).
[2017 update]
It is 2017 and installing Python modules changed a bit:
pip is now the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers.
venv is the standard tool for creating virtual environments (semi-isolated Python environments that allow packages to be installed for use by a particular application, rather than being installed system wide), and has been part of Python since Python 3.3. Starting with Python 3.4, it defaults to installing pip into all created virtual environments.
virtualenv is a third party alternative (and predecessor) to venv and if not official it is still very popular because it allows virtual environments to be used on versions of Python prior to 3.4, which either don’t provide venv at all, or aren’t able to automatically install pip into created environments.
easy_install pip
pip uninstall pip setuptools
(pip and setuptools both use the same package formats, but pip has uninstall support. kinda hilarious that installing something is the easiest way to uninstall.)
I was having trouble with the method below because my pip wasn't up to date.
easy_install pip
pip uninstall pip setuptools
After upgrading pip like this:
sudo -H pip install --upgrade pip
I was able to successfully uninstall setuptools like so:
pip uninstall setuptools