I'm accustomed to pre-downloading packages using Pip, then copying them over to a target machine for deployment. With the newly introduced Python Wheels, I'm forced to "pip ... --no-use-wheel", as some of the downloaded packages are platform specific (I'm developing on OSX and deploying to Debian) and will not install on the target machine. Is there a way to download Wheels for target platforms (or platform independent)?
The pip download command now has the --platform argument, which you can use to specify the desired platform:
pip download --platform=manylinux1_x86_64 --only-binary=:all: lxml
the --platform=manylinux1_x86_64 option indicates that you want wheels for this specific platform. manylinux1_x86_64 means roughly "compatible with most distributions and with an intel CPU architecture". This answer links to some PEPs that describe which platforms exist and what OS/CPU they are compatible with.
the --only-binary=:all: forces the use of binary distribution packages (ie. wheels, as opposed to sdist "source distribution packages") for ":all:" the things that will be installed in this command. Instead of :all:, one can pass a comma-separated list of specific distribution packages; see pip install --help for more info.
Note: I use the term "distribution package" to avoid confusion with the other kind of "package" (the ones one can import in a python script).
The easiest way to achieve that is IMO to use a custom script.
You can access the whole of the PyPI index via the simple interface, if the package of interest offers one or more wheels, they will be listed at the same address + /<package-name>.
For example: if you were to install setuptools all wheels would be listed at: https://pypi.python.org/simple/setuptools/
In your script, remember to implement the recommended tag priority as specified by PEP-425. Essentially that boils down to download the most specific (as opposed to the most general) version of the package as this normally translate into performance advantages, with for example C extensions replacing pure python implementations of some algorithm.
Related
I am building a python module using C++ with pybind11. This project will depend on two separate packages, that provide bindings for native C++ libraries:
opencv (provided by the pip package opencv-python)
imgui (provided by the pip package pyimgui).
This packages, once installed, consist mainly of dynamic libraries, plus some glue code in python.
My question concerns the installation/build process on the final user computer, i.e when pip install . is launched (in build isolation mode): how to find the path to the third parties dynamic libraries when linking?
In the end, they will be located somewhere in the virtual environment of the user; but I need to link them during the pip build (in isolation mode), and I also need to ensure that they will be found when the module will be used.
Note: as far a opencv is concerned, I could link to a local installation of OpenCV (i.e require the user to apt install opencv before, and use find_package(OpenCV)), but I suspect this is not the way to go. Concerning imgui, this solution would not apply.
I'm trying to build a speech recognition application that works in a browser. Currently using pyodide with a web worker. I have my own package, built alongside pyaudio, that I use for the web worker.
Is there a way that I can include a brew instillation, specifically portaudio, inside of my python package so that when I build the package, portaudio is included in the wheel file? I need portaudio included for this to work in the browser.
Thank you!
I'm understanding two different questions here, so I'll try to answer them both.
Can I have a python build fetch a Homebrew project during buildtime
To my knowledge, the answer is no. The Python distribution system is separate from Homebrew, and they can't interact in this fashion.
Even if they could, this wouldn't necessarily be desirable:
What happens if the user isn't on macOS (or Linux)? Then the build would fail.
The prefix that Homebrew will install the package in isn't very deterministic. The user might be using a custom prefix, or they might be on Apple Silicon (which has a different default prefix to Intel).
Your python package might run into some difficulty locating the package.
What about if they don't have Homebrew installed? They might have another package manager like MacPorts or Fink, or maybe none at all.
Can I bundle portaudio into the build distribution?
Maybe? Even if you could, I almost certainly wouldn't recommend it.
Bundling dependencies increases the size of the distribution unnecessarily.
It would take a reasonable amount of effort to setup, assuming you can do it.
All these reasons are why for the majority of projects that have a similar setup, you will find that they recommend installing certain packages with their system package manager first, before building the Python source code.
This allows them to choose whatever package manager they have installed, and it should also be a quick and painless process.
Therefore, just change your installation instructions to the following:
# On macOS
brew install portaudio
pip install ...
I just want to know if this is possible. For example, let's say if I want to build a pip package only for Linux or build a pip package for Windows and Mac but exclude Linux, is it possible to do that? Or do all pip packages become available for all 3 platforms (Windows, Mac, and Linux) when they get released?
Another example, let's say if Stripe wanted to release its Python SDK and make it Mac-only for Python users through pip, would that be possible?
I definitely would appreciate all answers as I am curious to know whether this is possible! I just want to know if this is possible or not. Thank you to those who do respond.
You have to consider both sdist and wheel distributions.
For sdist there is not much you can do, they are multi platforms period. You can add safe-guards in setup.py, so that running it fails on some platforms, but that is far from elegant. The best practice is to document clearly that your project is only for specific platforms via the "trove classifiers", and also for example via your long_description and/or README.
For wheel, you can make it so that you only build platform-specific wheels, that's one of the things wheels are good at:
How to force a python wheel to be platform specific when building it?
But best practices recommend you to publish the sdist as well in any case, so on other platforms the installer will fall-back to the sdist.
So to sum it up, you can build platform specific wheels but not platform-specific sdist. Document your project correctly (with trove classifiers) to declare which platforms are supported.
I've seen 2 ways of installing OpenCV (there might be more ways which I don't know):
Installing from the source
Installing with pip: pip install opencv-python
My question is, why we need to install OpenCV from the source while we can simply install it using pip? Since people are using both of them, both must be useful. If so, there are any conditions for selecting one of them?
I will list out the differences between both
1.
Installation using pip
Installation is done at the default location where all the python packages resides.
Installation from Source
Installation location is provided by the developer.
2.
Installation using pip
In terms of performance, the packages installed might run slower because of the hidden conflicts between features.
Installation from Source
The developer can select the optimization flags during the compilation of packages which are responsible for the fast performance of library.
3.
Installation using pip
The developers can neither add nor remove features provided in the installation done by pip.
Installation from Source
The developer has all the rights to add or remove the features during the installation of library.
4.
Installation using pip
The package manager will do the work on behalf of developer. Package Manager is also responsible for taking care of library updation.
Installation from Source
The developers are responsible for feature selection and updation of library. They must be aware of new package updates, latest security patches etc, to keep themselves updated about the library.
Hope this helps you!
OpenCV is always under development, and the thing is some parts of the library is not going to published, due to compatibility and copyright issues, but if you use the source then you can have all the capabilities that you need. SURF & SIFT are examples of this problem.
This question already has answers here:
What is a Python egg?
(4 answers)
Closed 6 years ago.
When I need a Python library, I use pip to fetch it from PyPi and if I create a project and want to share it, I just need to have in place the setup.py file and that would make it easily installable. Therefore, I was wondering what is the use case for egg or wheel packages.
The Python Packaging User Guide has to say the following on this topic:
Wheel and Egg are both packaging formats that aim to support the use case of needing an install artifact that doesn’t require building or compilation, which can be costly in testing and production workflows.
These formats can be used to distribute packages that contain binary extension modules. These would otherwise require compilation during installation.
If no compilation is involved a source distribution is in principle sufficient, but the user guide still recommends to create a wheel for performance reasons:
Minimally, you should create a Source Distribution:
python setup.py sdist
A “source distribution” is unbuilt (i.e, it’s not a Built Distribution), and requires a build step when installed by pip. Even if the distribution is pure python (i.e. contains no extensions), it still involves a build step to build out the installation metadata from setup.py.
[...]
You should also create a wheel for your project. A wheel is a built package that can be installed without needing to go through the “build” process. Installing wheels is substantially faster for the end user than installing from a source distribution.
In short, packages are a convenience thing - mostly for the user.
Wheel packages unify the process of distributing and installing projects that contain pure python, platform dependent code, or compiled extensions. The user does not need to worry if the package is written in Python or in C - it just works.
Egg packages are an older standard, you should ignore them nowadays. Use pip install . instead of ./setup.py install to prevent creating them. (addendum: They are also .zips in disguise, from which Python reads package data — not exactly the most performant solution)
Wheel packages, on the other hand, are the new standard. They allow for creation of portable binary packages for Windows, macOS, and Linux (yes, Linux!). Nowadays, you can just do pip install PyQt5 (as an example) and it will just work, no C++ compiler and Qt libraries required on the system. Everything is pre-compiled and included in the wheel. Non-binary packages also benefit, because it’s safer not to run setup.py (all the metadata is in the wheel). (addendum: those are also .zips, but they are unpacked when installed)