I'm trying to build a speech recognition application that works in a browser. Currently using pyodide with a web worker. I have my own package, built alongside pyaudio, that I use for the web worker.
Is there a way that I can include a brew instillation, specifically portaudio, inside of my python package so that when I build the package, portaudio is included in the wheel file? I need portaudio included for this to work in the browser.
Thank you!
I'm understanding two different questions here, so I'll try to answer them both.
Can I have a python build fetch a Homebrew project during buildtime
To my knowledge, the answer is no. The Python distribution system is separate from Homebrew, and they can't interact in this fashion.
Even if they could, this wouldn't necessarily be desirable:
What happens if the user isn't on macOS (or Linux)? Then the build would fail.
The prefix that Homebrew will install the package in isn't very deterministic. The user might be using a custom prefix, or they might be on Apple Silicon (which has a different default prefix to Intel).
Your python package might run into some difficulty locating the package.
What about if they don't have Homebrew installed? They might have another package manager like MacPorts or Fink, or maybe none at all.
Can I bundle portaudio into the build distribution?
Maybe? Even if you could, I almost certainly wouldn't recommend it.
Bundling dependencies increases the size of the distribution unnecessarily.
It would take a reasonable amount of effort to setup, assuming you can do it.
All these reasons are why for the majority of projects that have a similar setup, you will find that they recommend installing certain packages with their system package manager first, before building the Python source code.
This allows them to choose whatever package manager they have installed, and it should also be a quick and painless process.
Therefore, just change your installation instructions to the following:
# On macOS
brew install portaudio
pip install ...
The situation
You have 2 software products in development, a library which presents an API and a GUI tool that exposes the library for end users. Additionally, you expect a lot of technical staff at your place to use the library as a building block for all sorts of relevant custom code, tools and assets.
Both software products (library and GUI tool) are actively in development and influence each other. For both you want the most easy way of distribution and devenv setup, using pip:
pip install gui_tool
or
pip install library
Gui Tool (use case 1)
The installation of the GUI tool happens via pip and the dependencies are noted with a hard version number within the setup.py. Your library is one of those dependencies:
...
install_requires = ['library==1.2, package_x==0.3, package_y==0.6'],
...
The installation procedure consists of installing the tools and resolving the dependencies into a fresh virtualenv. Because every dependency version is hardwired, you control a stable and consistent installation. Nightly builds/bleeding edge dependencies can be controlled by devs through manually updating library to newer versions:
pip install --upgrade library # get the latest nightly build/hotfix release on your own
Custom code/tools (use case 2)
As mentioned, a lot of code and custom tools might be built using the API that your library provides. Everybody willing to use it should install/update it through pip easily with the oneliner from the top.
Problem
Fellow GUI tool developers should be able to pull in nightly builds/hotfix releases of the library dependency with pip. Other staff, using library as a building block somewhere, should always get the latest stable version using pip. You want to remain one unique release procedure for library providing stable versions, bleeding edge and hotfix releases through X.Y.Z versioning.
There are a few possible solutions to this, like:
maintaining a readme for the custom users that states the stable versions they should be installing with pip
or setting up some magic within the setup.py for the Gui Tool cloning the git repo and using it through python setup.py develop (Versioning can then be handled by devs via checkouts in the repo).
However, none of these seem particularly elegant, so I am interested in your solutions, ideas or best practices for stable/bleeding edge/nightly build dependency management for Python?
Managing a "bleeding edge" version in pip can be achieved by using the --pre flag of pip
From pip install --help:
--pre Include pre-release and development versions. By default, pip only finds stable versions.
You should add classifiers to your project, in particular set your stable release to Development Status :: 5 - Production/Stable and your "bleeding edge releases" to anything below 5.
This is how major python packages manage their alphas, e.g.: the django project with currently 1.9a in alpha and 1.8.5 in stable.
To upgrade to the latest version flagged as Development Status :: 3 - Alpha
:
pip install --pre --upgrade library
Users using the library as building blocks need not now about the alpha releases and will go with a regular pip install library which will install the latest stable flagged version.
I am not sure if you need to manage on library or several libraries?
It seems like you have a case of complex Python dependency tracking and deployment. There are few large Python projects which face the same issue with the need to track several different release channels, betas and bleeding edge. Most notable them is Plone which sports more than 300 MB of Python egg source code.
Plone used a buildout as the dependency manager instead of pip to tackle the complexity. Buildout offers mix and match configuration files for Python dependencies.
To quickly grasp how this happens see Plone core buildouts
Example of Plone 5.0 latest: http://dist.plone.org/release/5.0-latest/versions.cfg
Example of older Plone 4.x latest release: http://dist.plone.org/release/4.3rc1/versions.cfg
Github repository which drives the process https://github.com/plone/buildout.coredev - however the complexity here has grown so high that it is no longer very easy for an outsider to grasp what is going on. Release process described here http://buildoutcoredev.readthedocs.org/en/latest/release.html
Warning Buildout is arcane and rough on the edges like just surfaced volcano island of hot lava. Is it more elegant than generating pip requirements.txt with custom scripts? Yes if you have more than one library which needs to be managed.
I work on a system and the hosting guys don't want to use an install script that uses pip. Now we have a large pip requirements file that install the dependencies. Is there any other way to do it than using pip? Can it be done using yum or apt-get ? We are using Linux.
For god's sake, please do not fall back to using the distribution's package manager just because your hosting guys do not understand what pip+virtualenv is good for.
Python packages in Linux distribution repositories are often outdated and may come with quirks that other Python package authors did not plan for. This is especially true for Python packages with compiled code. If a documentation tells you that a certain dependency should be obtained directly from PyPI via pip, then you better follow that requirement. Convince your hosting guys to use the right tools, namely pip combined with virtualenv. The latter will create an isolated environment and make sure that the system will stay clean (really, nobody needs to do a sudo pip install, which probably is the thing your hosting guys are afraid of).
I am new to Python and I want to install VLFeat on Ubuntu (13.04).
I am using Eclipse 3.8. For Python, I have installed the PyDev extension on Eclipse.
I have installed Numpy, but I don't know how to install VLFeat. I tried to use their website, but I can't get anything for Python. I have downloaded packages, but I don't know how to install them.
The Menpo Project provides a wrapper around VLFeat: it's called cyvlfeat.
To install cyvlfeat, we strongly suggest you use conda:
conda install -c menlo cyvlfeat
If you don't want to use conda, your
mileage will vary. In particular, you must satisfy the
linking/compilation requirements for the package, which include the
vlfeat dynamic library.
In other words, the nice thing about installing with conda is that it will install (and link) VLFeat dependencies as well.
It may not include all functionalities of VLFeat. Current State as of March 2017:
fisher
fisher
generic
set_simd_enabled, get_simd_enabled, cpu_has_avx, cpu_has_sse3, cpu_has_sse2
get_num_cpus,
get_max_threads, set_num_threads, get_thread_limit
hog
hog
kmeans
kmeans
kmeans_quantize
ikmeans, ikmeans_push
hikmeans, hikmeans_push
sift
dsift
sift
Relevant reading
Dev blog by Simmi Mourya. This includes descriptions and usage examples.
More alternatives:
vlfeat-ctypes: minimal VLFeat interface for python
pyvlfeat fork by jchazalon: A high-level Python wrapper around a subset of the VLFeat library [more recently updated than the original]
Note about the IDE
Installing python packages should be independent of the IDE (Eclipse + PyDev, in the OP case), as long as the interpreter and libraries paths are correctly set up.
Note about conda
It is not required to install the Anaconda distribution in order to use conda. The much lighter Miniconda is enough.
Assuming you are getting VLFeat from the Python Package Index, the instructions are
Download distribution, extract it, get to command prompt and type:
$ python setup.py install
Does Python have a package/module management system, similar to how Ruby has rubygems where you can do gem install packagename?
On Installing Python Modules, I only see references to python setup.py install, but that requires you to find the package first.
Recent progress
March 2014: Good news! Python 3.4 ships with Pip. Pip has long been Python's de-facto standard package manager. You can install a package like this:
pip install httpie
Wahey! This is the best feature of any Python release. It makes the community's wealth of libraries accessible to everyone. Newbies are no longer excluded from using community libraries by the prohibitive difficulty of setup.
However, there remains a number of outstanding frustrations with the Python packaging experience. Cumulatively, they make Python very unwelcoming for newbies. Also, the long history of neglect (ie. not shipping with a package manager for 14 years from Python 2.0 to Python 3.3) did damage to the community. I describe both below.
Outstanding frustrations
It's important to understand that while experienced users are able to work around these frustrations, they are significant barriers to people new to Python. In fact, the difficulty and general user-unfriendliness is likely to deter many of them.
PyPI website is counter-helpful
Every language with a package manager has an official (or quasi-official) repository for the community to download and publish packages. Python has the Python Package Index, PyPI. https://pypi.python.org/pypi
Let's compare its pages with those of RubyGems and Npm (the Node package manager).
https://rubygems.org/gems/rails RubyGems page for the package rails
https://www.npmjs.org/package/express Npm page for the package express
https://pypi.python.org/pypi/simplejson/ PyPI page for the package simplejson
You'll see the RubyGems and Npm pages both begin with a one-line description of the package, then large friendly instructions how to install it.
Meanwhile, woe to any hapless Python user who naively browses to PyPI. On https://pypi.python.org/pypi/simplejson/ , they'll find no such helpful instructions. There is however, a large green 'Download' link. It's not unreasonable to follow it. Aha, they click! Their browser downloads a .tar.gz file. Many Windows users can't even open it, but if they persevere they may eventually extract it, then run setup.py and eventually with the help of Google setup.py install. Some will give up and reinvent the wheel..
Of course, all of this is wrong. The easiest way to install a package is with a Pip command. But PyPI didn't even mention Pip. Instead, it led them down an archaic and tedious path.
Error: Unable to find vcvarsall.bat
Numpy is one of Python's most popular libraries. Try to install it with Pip, you get this cryptic error message:
Error: Unable to find vcvarsall.bat
Trying to fix that is one of the most popular questions on Stack Overflow: "error: Unable to find vcvarsall.bat"
Few people succeed.
For comparison, in the same situation, Ruby prints this message, which explains what's going on and how to fix it:
Please update your PATH to include build tools or download the DevKit from http://rubyinstaller.org/downloads and follow the instructions at http://github.com/oneclick/rubyinstaller/wiki/Development-Kit
Publishing packages is hard
Ruby and Nodejs ship with full-featured package managers, Gem (since 2007) and Npm (since 2011), and have nurtured sharing communities centred around GitHub. Npm makes publishing packages as easy as installing them, it already has 64k packages. RubyGems lists 72k packages. The venerable Python package index lists only 41k.
History
Flying in the face of its "batteries included" motto, Python shipped without a package manager until 2014.
Until Pip, the de facto standard was a command easy_install. It was woefully inadequate. The was no command to uninstall packages.
Pip was a massive improvement. It had most the features of Ruby's Gem. Unfortunately, Pip was--until recently--ironically difficult to install. In fact, the problem remains a top Python question on Stack Overflow: "How do I install pip on Windows?"
And just to provide a contrast, there's also pip.
The Python Package Index (PyPI) seems to be standard:
To install a package:
pip install MyProject
To update a package
pip install --upgrade MyProject
To fix a version of a package pip install MyProject==1.0
You can install the package manager as follows:
curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py
easy_install pip
References:
http://guide.python-distribute.org/
http://pypi.python.org/pypi/distribute
As a Ruby and Perl developer and learning-Python guy, I haven't found easy_install or pip to be the equivalent to RubyGems or CPAN.
I tend to keep my development systems running the latest versions of modules as the developers update them, and freeze my production systems at set versions. Both RubyGems and CPAN make it easy to find modules by listing what's available, then install and later update them individually or in bulk if desired.
easy_install and pip make it easy to install a module ONCE I located it via a browser search or learned about it by some other means, but they won't tell me what is available. I can explicitly name the module to be updated, but the apps won't tell me what has been updated nor will they update everything in bulk if I want.
So, the basic functionality is there in pip and easy_install but there are features missing that I'd like to see that would make them friendlier and easier to use and on par with CPAN and RubyGems.
There are at least two, easy_install and its successor pip.
As of at least late 2014, Continuum Analytics' Anaconda Python distribution with the conda package manager should be considered. It solves most of the serious issues people run into with Python in general (managing different Python versions, updating Python versions, package management, virtual environments, Windows/Mac compatibility) in one cohesive download.
It enables you to do pretty much everything you could want to with Python without having to change the system at all. My next preferred solution is pip + virtualenv, but you either have to install virtualenv into your system Python (and your system Python may not be the version you want), or build from source. Anaconda makes this whole process the click of a button, as well as adding a bunch of other features.
That'd be easy_install.
It's called setuptools. You run it with the "easy_install" command.
You can find the directory at http://pypi.python.org/
I don't see either MacPorts or Homebrew mentioned in other answers here, but since I do see them mentioned elsewhere on Stack Overflow for related questions, I'll add my own US$0.02 that many folks seem to consider MacPorts as not only a package manager for packages in general (as of today they list 16311 packages/ports, 2931 matching "python", albeit only for Macs), but also as a decent (maybe better) package manager for Python packages/modules:
Question
"...what is the method that Mac python developers use to manage their modules?"
Answers
"MacPorts is perfect for Python on the Mac."
"The best way is to use MacPorts."
"I prefer MacPorts..."
"With my MacPorts setup..."
"I use MacPorts to install ... third-party modules tracked by MacPorts"
SciPy
"Macs (unlike Linux) don’t come with a package manager, but there are a couple of popular package managers you can install.
Macports..."
I'm still debating on whether or not to use MacPorts myself, but at the moment I'm leaning in that direction.
On Windows install http://chocolatey.org/ then
choco install python
Open a new cmd-window with the updated PATH. Next, do
choco install pip
After that you can
pip install pyside
pip install ipython
...
Since no one has mentioned pipenv here, I would like to describe my views why everyone should use it for managing python packages.
As #ColonelPanic mentioned there are several issues with the Python Package Index and with pip and virtualenv also.
Pipenv solves most of the issues with pip and provides additional features also.
Pipenv features
Pipenv is intended to replace pip and virtualenv, which means pipenv will automatically create a separate virtual environment for every project thus avoiding conflicts between different python versions/package versions for different projects.
Enables truly deterministic builds, while easily specifying only what you want.
Generates and checks file hashes for locked dependencies.
Automatically install required Pythons, if pyenv is available.
Automatically finds your project home, recursively, by looking for a Pipfile.
Automatically generates a Pipfile, if one doesn’t exist.
Automatically creates a virtualenv in a standard location.
Automatically adds/removes packages to a Pipfile when they are un/installed.
Automatically loads .env files, if they exist.
If you have worked on python projects before, you would realize these features make managing packages way easier.
Other Commands
check checks for security vulnerabilities and asserts that PEP 508 requirements are being met by the current environment. (which I think is a great feature especially after this - Malicious packages on PyPi)
graph will show you a dependency graph, of your installed dependencies.
You can read more about it here - Pipenv.
Installation
You can find the installation documentation here
P.S.: If you liked working with the Python Package requests , you would be pleased to know that pipenv is by the same developer Kenneth Reitz
In 2019 poetry is the package and dependency manager you are looking for.
https://github.com/sdispater/poetry#why
It's modern, simple and reliable.
Poetry is what you're looking for. It takes care of dependency management, virtual environments, running.