I have been reading a lot of the tutorials around the web and answers on the site about using Homebrew. When it comes to Python though, the advice leaves me with more questions than answers.
I understand the how, but none of the answers I have seen so far have really explained the why behind using Homebrew to install Python and what the benefits are to installing Python with Homebrew as opposed to using OS-X installers provided by the Python Foundation?
The newest versions of the installers from Python and the current implementation of PIP seem to be working pretty well, so I would really appreciate any input on my question. I have worked with Python for a while but from more of a tactical, one off problem solving perspective and I am brand new to tools like Brew and version control software such as Git. I am trying to get up the learning curve. Finding an answer to why I would choose to go with a Homebrew install over just heading over to python.org and downloading from them and then using pip to install packages might help me to understand the benefits of a tool like Homebrew.
So I guess, what does Homebrew give me that going through the installation put in place by TPF does not?
Are there advantages/disadvantages to where Homebrew installs Python and Python packages over the /Library/Frameworks/ and the site-packages folder within that framework?
Though this last question is too broad and likely out of scope, if anyone would also address or provide a link to a good answer on what the benefits are of using Homebrew in general, I'd appreciate it?
Thank you,
The big advantage of using a package manager like Homebrew is it makes it easier to keep your Python installation up to date. If you download Python from the website, then to update it means you'll need to go back to the website and download a new copy of Python (or whatever it is that you need to update that could have been installed with Homebrew).
Also, when downloading installers, I find they tend to clutter up my downloads folder and require me to periodically clean up unused files. I'd rather spend my time coding instead of managing my disk space usage.
When it comes to updating any package with Homebrew, the command is simple:
brew upgrade
And this will update all outdated packages that you installed with Brew.
Now, this isn't something unique to Homebrew. Macports, PIP, npm, Maven, and other package management tools are also able to manage the versions of modules or tools you install.
For more information, see Safari Books Online - Keeping Your Homebrew Up To Date.
A few reasons not to use system python on OS X from this post,
Apple doesn’t always do a good job on keeping the Python runtime environment up to date,
it can be cumbersome to play with permissions just to install third-party Python libraries,
finally, Apple has a tendency to wipe-out your site-packages with every major OS upgrade.
The use of an independent package manager for Python modules, such as Homebrew, conda, Macports, ets. is thus preferred.
Related
I'm trying to build a speech recognition application that works in a browser. Currently using pyodide with a web worker. I have my own package, built alongside pyaudio, that I use for the web worker.
Is there a way that I can include a brew instillation, specifically portaudio, inside of my python package so that when I build the package, portaudio is included in the wheel file? I need portaudio included for this to work in the browser.
Thank you!
I'm understanding two different questions here, so I'll try to answer them both.
Can I have a python build fetch a Homebrew project during buildtime
To my knowledge, the answer is no. The Python distribution system is separate from Homebrew, and they can't interact in this fashion.
Even if they could, this wouldn't necessarily be desirable:
What happens if the user isn't on macOS (or Linux)? Then the build would fail.
The prefix that Homebrew will install the package in isn't very deterministic. The user might be using a custom prefix, or they might be on Apple Silicon (which has a different default prefix to Intel).
Your python package might run into some difficulty locating the package.
What about if they don't have Homebrew installed? They might have another package manager like MacPorts or Fink, or maybe none at all.
Can I bundle portaudio into the build distribution?
Maybe? Even if you could, I almost certainly wouldn't recommend it.
Bundling dependencies increases the size of the distribution unnecessarily.
It would take a reasonable amount of effort to setup, assuming you can do it.
All these reasons are why for the majority of projects that have a similar setup, you will find that they recommend installing certain packages with their system package manager first, before building the Python source code.
This allows them to choose whatever package manager they have installed, and it should also be a quick and painless process.
Therefore, just change your installation instructions to the following:
# On macOS
brew install portaudio
pip install ...
I have just spent 2 days trying to build Tensorflow from source, and finally succeeded when I realized that sudo pip (even with the -H flag) was not finding my anaconda pip, but instead finding a pip installed with apt. Running, then, sudo -H ~/anaconda3/bin/pip ... fixed my problem.
In order to avoid this kind of issue ever again (I had several issues in this process with the "wrong" python being used), is it possible for me to completely remove python from my system, keeping only Anaconda? Is it advisable?
No, you can't. Python is an essential part of ubuntu (and of just any linux distrib nowadays), so removing it will just break your system. Well, if you want a broken system then it's of course up to you ;)
The right solution is to use virtualenvs for your own workspaces instead, so you can have different isolated installs with different python and 3rd part packages versions. This also avoids breaking anything when two projects depends on different versions of a same package. For a local use you probably want to install virtualenvwrapper too (it's an addon over virtualenv that makes working with virtualenvs easier - switching between virtualenvs, running hooks on env switch etc).
Oh and yes: those virtualenvs are per-user so no more sudo and you don't risk messing up your system install (thx Attie for mentioning this).
And to be even more exhaustive: darthbith mentions in a comment that anaconda has it's own virtualization/package management system conda that might make virtualenv redundant and is anyway a best fit if you're using anaconda.
This is not just a Ubuntu issue but also a linux world wide issue. The system python is at the core of apt-get and yum package managers. Also the modern grub is based on python so removing it can make your machine unbootable.
In short, this will affect RHEL related distributions (CentOS/Fedora) and Debian related distributions (Debian/Ubuntu).
I work on a system and the hosting guys don't want to use an install script that uses pip. Now we have a large pip requirements file that install the dependencies. Is there any other way to do it than using pip? Can it be done using yum or apt-get ? We are using Linux.
For god's sake, please do not fall back to using the distribution's package manager just because your hosting guys do not understand what pip+virtualenv is good for.
Python packages in Linux distribution repositories are often outdated and may come with quirks that other Python package authors did not plan for. This is especially true for Python packages with compiled code. If a documentation tells you that a certain dependency should be obtained directly from PyPI via pip, then you better follow that requirement. Convince your hosting guys to use the right tools, namely pip combined with virtualenv. The latter will create an isolated environment and make sure that the system will stay clean (really, nobody needs to do a sudo pip install, which probably is the thing your hosting guys are afraid of).
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
If I'm learning python development on OSX (Snow Leopard), how should I best manage my python packages?
I am very excited about the large number of great python packages out there. I would love to start learning them, but package management has got me totally confused. Clearly I have no idea what I'm doing. I see many questions on Stack Overflow that address package management, and people frequently answer them by saying "Why don't you just use x? Look how easy it is in this single line I copied from my shell,"
$ x install something
Which is a great answer until x fails to handle some specific dependencies or can't handle package y. Here's what I've already attempted:
I started out just installing things with easy_install.
I soon found that some important packages had missing dependencies (PIL)
so I hunted around and discovered MacPorts.
I started using MacPorts and all my environment variables got really confusing, so I learned enough bash to reconfigure PATH, PYTHONPATH, and sys.path to get things working (and to make my terminal window look sweet). MacPorts seemed great for a while.
But then I got confused about which version of different modules I was using, and I found more dependency issues (GDAL) ...
Then several very intelligent people strongly recommended I switch to Homebrew.
So I uninstalled MacPorts and installed Homebrew.
Now that I have Homebrew, I can't seem to install much of anything, because I don't know how to create Formula for most of the python packages I want to use. If a package is listed at pypi, I don't know if I should actually use Homebrew, or just figure out how to configure easy_install to install to Homebrew directories. And I still don't know how I should best approach missing dependencies. Maybe there's some secret repository of python-related Formula? or an easy way of converting portfiles into Formula?
I know that the right answer might depend on what I'm trying to install. I'm trying to install sets of packages that let me:
play around and automate everything with python
appscript
PIL
pygame
do GIS related work in python
gdal
shapely
postgis
scipy
explore the wondrous possibilities of web apps
django
google-app-engine
I'm fully willing to wipe everything off my machine, which must be a bit cluttered with scraps of packages by now. If I should be learning virtualenv, or just learning more about the basic process of checking dependencies and installing them myself, I'm happy to do that. If I should just dig into Homebrew, I happy to do that as well. How should I deal with these issues, which often prevent me from doing what I really want to do (make neat stuff with python)?
I would suggest the following strongly. I use them for basic package management.
virtualenv: http://pypi.python.org/pypi/virtualenv
virtualenvwrapper: http://pypi.python.org/pypi/virtualenvwrapper
yolk: http://pypi.python.org/pypi/yolk
pip : http://pypi.python.org/pypi/pip
Tutorial / blogs that should help:
http://simononsoftware.com/virtualenv-tutorial/
http://paintincode.blogspot.com/2010/08/install-pip-virtualenv.html
In my opinion, the best way to handle python packages is to have a set of Python installations separate from the system Python, one for each version you need.
This way I do not pollute the main python with old versions of packages, and I can delete them and reinstall them without any of my projects having problems.
Details:
On OS X, using the Macports versions are fine. The same is probably true for homebrew, although I never used it. (On Linuxes I install Python from source into /opt/pythonxx/, for example /opt/python25/, /opt/python26/, /opt/python27 and /opt/python31.)
I then never install any packages into these Pythons, with the exception of virtualenv, distribute and pip, which I install in all of them. When I want to install a package for a project, I make a virtualenv for that project with the python version I use for it:
$ cd /home/projects
$ /opt/python27/bin/virtualenv acoolproject
$ cd acoolproject
$ ./bin/pip install ThepackageIneed.
As you install each projects dependencies separately, you avoid dependency issues and version conflicts and version confusion.
You can enable and disable environments with virtualenv, I tend never to do that, I use the python executable explicitly instead:
$ ./bin/python main.py
or
$ /home7projects/acoolproject/bin/python
If I need things that you can't just install with pip, like Plone, or nginx or varnish, I use zc.buildout to make a replicatable environment configuration, zc.buildout will also run in isolation, so no packages gets installed in the main pythons.
If you're using Homebrew, you'll probably want to use pip to install Python packages. pip can be installed via Homebrew, but Homebrew doesn't include formulae for other Python packages. But it's pretty easy to install packages using pip -- as simple as
$ pip install <package>
(The package name can be found on PyPI.) If you installed pip via Homebrew, that's all you need to do.
Does Python have a package/module management system, similar to how Ruby has rubygems where you can do gem install packagename?
On Installing Python Modules, I only see references to python setup.py install, but that requires you to find the package first.
Recent progress
March 2014: Good news! Python 3.4 ships with Pip. Pip has long been Python's de-facto standard package manager. You can install a package like this:
pip install httpie
Wahey! This is the best feature of any Python release. It makes the community's wealth of libraries accessible to everyone. Newbies are no longer excluded from using community libraries by the prohibitive difficulty of setup.
However, there remains a number of outstanding frustrations with the Python packaging experience. Cumulatively, they make Python very unwelcoming for newbies. Also, the long history of neglect (ie. not shipping with a package manager for 14 years from Python 2.0 to Python 3.3) did damage to the community. I describe both below.
Outstanding frustrations
It's important to understand that while experienced users are able to work around these frustrations, they are significant barriers to people new to Python. In fact, the difficulty and general user-unfriendliness is likely to deter many of them.
PyPI website is counter-helpful
Every language with a package manager has an official (or quasi-official) repository for the community to download and publish packages. Python has the Python Package Index, PyPI. https://pypi.python.org/pypi
Let's compare its pages with those of RubyGems and Npm (the Node package manager).
https://rubygems.org/gems/rails RubyGems page for the package rails
https://www.npmjs.org/package/express Npm page for the package express
https://pypi.python.org/pypi/simplejson/ PyPI page for the package simplejson
You'll see the RubyGems and Npm pages both begin with a one-line description of the package, then large friendly instructions how to install it.
Meanwhile, woe to any hapless Python user who naively browses to PyPI. On https://pypi.python.org/pypi/simplejson/ , they'll find no such helpful instructions. There is however, a large green 'Download' link. It's not unreasonable to follow it. Aha, they click! Their browser downloads a .tar.gz file. Many Windows users can't even open it, but if they persevere they may eventually extract it, then run setup.py and eventually with the help of Google setup.py install. Some will give up and reinvent the wheel..
Of course, all of this is wrong. The easiest way to install a package is with a Pip command. But PyPI didn't even mention Pip. Instead, it led them down an archaic and tedious path.
Error: Unable to find vcvarsall.bat
Numpy is one of Python's most popular libraries. Try to install it with Pip, you get this cryptic error message:
Error: Unable to find vcvarsall.bat
Trying to fix that is one of the most popular questions on Stack Overflow: "error: Unable to find vcvarsall.bat"
Few people succeed.
For comparison, in the same situation, Ruby prints this message, which explains what's going on and how to fix it:
Please update your PATH to include build tools or download the DevKit from http://rubyinstaller.org/downloads and follow the instructions at http://github.com/oneclick/rubyinstaller/wiki/Development-Kit
Publishing packages is hard
Ruby and Nodejs ship with full-featured package managers, Gem (since 2007) and Npm (since 2011), and have nurtured sharing communities centred around GitHub. Npm makes publishing packages as easy as installing them, it already has 64k packages. RubyGems lists 72k packages. The venerable Python package index lists only 41k.
History
Flying in the face of its "batteries included" motto, Python shipped without a package manager until 2014.
Until Pip, the de facto standard was a command easy_install. It was woefully inadequate. The was no command to uninstall packages.
Pip was a massive improvement. It had most the features of Ruby's Gem. Unfortunately, Pip was--until recently--ironically difficult to install. In fact, the problem remains a top Python question on Stack Overflow: "How do I install pip on Windows?"
And just to provide a contrast, there's also pip.
The Python Package Index (PyPI) seems to be standard:
To install a package:
pip install MyProject
To update a package
pip install --upgrade MyProject
To fix a version of a package pip install MyProject==1.0
You can install the package manager as follows:
curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py
easy_install pip
References:
http://guide.python-distribute.org/
http://pypi.python.org/pypi/distribute
As a Ruby and Perl developer and learning-Python guy, I haven't found easy_install or pip to be the equivalent to RubyGems or CPAN.
I tend to keep my development systems running the latest versions of modules as the developers update them, and freeze my production systems at set versions. Both RubyGems and CPAN make it easy to find modules by listing what's available, then install and later update them individually or in bulk if desired.
easy_install and pip make it easy to install a module ONCE I located it via a browser search or learned about it by some other means, but they won't tell me what is available. I can explicitly name the module to be updated, but the apps won't tell me what has been updated nor will they update everything in bulk if I want.
So, the basic functionality is there in pip and easy_install but there are features missing that I'd like to see that would make them friendlier and easier to use and on par with CPAN and RubyGems.
There are at least two, easy_install and its successor pip.
As of at least late 2014, Continuum Analytics' Anaconda Python distribution with the conda package manager should be considered. It solves most of the serious issues people run into with Python in general (managing different Python versions, updating Python versions, package management, virtual environments, Windows/Mac compatibility) in one cohesive download.
It enables you to do pretty much everything you could want to with Python without having to change the system at all. My next preferred solution is pip + virtualenv, but you either have to install virtualenv into your system Python (and your system Python may not be the version you want), or build from source. Anaconda makes this whole process the click of a button, as well as adding a bunch of other features.
That'd be easy_install.
It's called setuptools. You run it with the "easy_install" command.
You can find the directory at http://pypi.python.org/
I don't see either MacPorts or Homebrew mentioned in other answers here, but since I do see them mentioned elsewhere on Stack Overflow for related questions, I'll add my own US$0.02 that many folks seem to consider MacPorts as not only a package manager for packages in general (as of today they list 16311 packages/ports, 2931 matching "python", albeit only for Macs), but also as a decent (maybe better) package manager for Python packages/modules:
Question
"...what is the method that Mac python developers use to manage their modules?"
Answers
"MacPorts is perfect for Python on the Mac."
"The best way is to use MacPorts."
"I prefer MacPorts..."
"With my MacPorts setup..."
"I use MacPorts to install ... third-party modules tracked by MacPorts"
SciPy
"Macs (unlike Linux) don’t come with a package manager, but there are a couple of popular package managers you can install.
Macports..."
I'm still debating on whether or not to use MacPorts myself, but at the moment I'm leaning in that direction.
On Windows install http://chocolatey.org/ then
choco install python
Open a new cmd-window with the updated PATH. Next, do
choco install pip
After that you can
pip install pyside
pip install ipython
...
Since no one has mentioned pipenv here, I would like to describe my views why everyone should use it for managing python packages.
As #ColonelPanic mentioned there are several issues with the Python Package Index and with pip and virtualenv also.
Pipenv solves most of the issues with pip and provides additional features also.
Pipenv features
Pipenv is intended to replace pip and virtualenv, which means pipenv will automatically create a separate virtual environment for every project thus avoiding conflicts between different python versions/package versions for different projects.
Enables truly deterministic builds, while easily specifying only what you want.
Generates and checks file hashes for locked dependencies.
Automatically install required Pythons, if pyenv is available.
Automatically finds your project home, recursively, by looking for a Pipfile.
Automatically generates a Pipfile, if one doesn’t exist.
Automatically creates a virtualenv in a standard location.
Automatically adds/removes packages to a Pipfile when they are un/installed.
Automatically loads .env files, if they exist.
If you have worked on python projects before, you would realize these features make managing packages way easier.
Other Commands
check checks for security vulnerabilities and asserts that PEP 508 requirements are being met by the current environment. (which I think is a great feature especially after this - Malicious packages on PyPi)
graph will show you a dependency graph, of your installed dependencies.
You can read more about it here - Pipenv.
Installation
You can find the installation documentation here
P.S.: If you liked working with the Python Package requests , you would be pleased to know that pipenv is by the same developer Kenneth Reitz
In 2019 poetry is the package and dependency manager you are looking for.
https://github.com/sdispater/poetry#why
It's modern, simple and reliable.
Poetry is what you're looking for. It takes care of dependency management, virtual environments, running.