What's the best way to deploy a Python package to a Windows Server if some of the dependencies need to be compiled? Installing Visual Studio is out of the question and I'm reluctant to pass a 250MB file around everytime things need updating. pip install -r requirements.txt is the goal.
Any suggestions?
easy_install - allows installing from exe
easy_install does install from exe installers, if they are available. pip does not install from exe, so in case of C/C++ packages, easy_install is often the way to install.
making pip to install binaries
pip is supporting installation from wheel format.
In case you have a wheel on pypi or another index server you use, allow pip to use wheels (--use-wheels). Latest version of pip is using wheel format by default, older ones required setting --use-wheel switch.
building wheels
You can either build (compile) wheels, or you can convert eggs or exe installers to wheels.
For this, use wheel command.
Serving packages in wheel format
Some packages (e.g. lxml) do not provide wheels on pypi server. If you want to use them, you have to manage your own index. Easiest one is to use dedicated directory and pip configured with --find-links pointing there.
See SO answer to caching packages for more tips and details.
Very good alternative is devpi index server, which is easy to set up, has very good workflow and will integrate with your pip very well.
Using a compile-once, deploy-many-times strategy might help, with the use of Python's newest package distribution format, wheels. They are an already-compiled, binary format. So the requirements are just a development / build platform that's similar / running the same python as the deployment platform (the latter won't require a build chain anymore).
Install the latest pip, setuptools and wheel packages. You can then use the pip wheel command as you would pip install, except it will instead produce a folder called (by default) wheelhouse full of the necessary, already-compiled wheels for another python/pip to install what you required.
You can then pass those wheel filepaths directly to a pip install command (e.g on your deployment machine), or use the normal package name(s) with --no-index --find-links= flags to point to the folder location where those wheels are. This location can also be a simple http folder - for example.
Related
So, I have just started using python and I installed conda to manage the packages, but now I need to install a package (which comes in a "wheel" file, whatever that is?) which is not available from conda repositories and I am not sure what to do. I might be able to use pip but I have read somewhere that this is not a good way since I will not be able to use conda features to manage it later. I saw this https://docs.conda.io/projects/conda-build/en/latest/user-guide/wheel-files.html article, but it talks about something called "conda recipe" and other thing I have no clue about and doesn't provide a step-by-step instruction on what to do. So, how do I fully incorporate such "non-conda" packages into conda?
A wheel file is like an .exe file for windows. It installes an application, or in this case a Python package. To install wheel files with conda run the following command in your terminal:
conda install -c 'wheel'
Replace 'wheel' with your downloaded file. Check out this source for wheel files Unofficial windows binaries.
I have a Python script which uses open source third party libraries for geoprocessing (OGR and Shapely). My plan is to execute this script on a computer without having to install the required libraries.
I know that there are tools such as py2exe available for this purpose. However, compiling an executable file is not my first choice as I noticed that such files can get pretty large in size. Besides, I would like to use the code within another script. I would therefore like to create a portable python script which already includes the third party methods needed for executing.
Is it possible to include third party methods in a Python script in order to avoid the installation of third party libraries? And if not, what can I do instead, besides compiling an executable file? I work on Windows OS.
You can export your libraries using pip and embbed them into your application.
pip wheel requires the wheel package to be installed, which provides the "bdist_wheel" setuptools extension that it uses.
To build wheels for your requirements and all their dependencies to a local directory:
pip install wheel
pip freeze > requirements.txt
At this point check requirements.txt, clean it up, then you can download wheels in a local folder :
pip wheel --wheel-dir=/local/wheels -r requirements.txt
And then to install those requirements just using your local directory of wheels (and not from PyPI):
pip install --no-index --find-links=/local/wheels -r requirements.txt
Then though you'll need pip, though it's shipped with latest versions of python.
Check this : https://pip.readthedocs.io/en/latest/user_guide/#requirements-files
If your third party lib does not have any dependencies you can get the source files (.py) put into your project folder and use it as package by using import, else it has dependencies your project size grow more better create exe for your script.
I am still relatively new to python packaging, each time I think I find "the" solution, I am thrown another curve ball, here is my problem followed by what I've tried:
I have CentOS and Ubuntu systems with Python 2.7.3 installed that is partitioned from the net so I have to create an "all in one package"
The target system does NOT have setuptools, easy_install, pip, virtualenv installed (this is the problem I'm trying to solve here)
The requirements.txt (or setup.py install_dependencies) is fairly heavy (Flask, etc...) for the application (though really, this isn't the problem)
My packaging sophistication has progressed slowly:
For connected systems, I had a really nice process going with
packaging: python2.7 setup.py sdist
installation: create a virtualenv, untar the distribution, python setup.py install
For the disconnected system, I've tried a few things. Wheels seem to be appropriate but I can't get to the "final" installation that includes setuptools, easy_install, pip. I am new to wheels so perhaps I am missing something obvious.
I started with these references:
Python on Wheels, this was super helpful but I could not get my .sh scripts, test data, etc... installed so I am actually using a wheel/sdist hybrid right now
Wheel, the Docs, again, very helpful but I am stuck on "the final mile of a disconnected system"
I then figured out I could package virtualenv as a wheel :-) Yay
I then figured out I could package easy_install as a python program :-) Yay, but it depends on setuptools, boo, I can't find how to get these packaged / installed
Is there a reference around for bootstrapping a system that has Python, is disconnected, but does not have setuptools, pip, wheels, virtualenv? My list of things a person must do to install this simple agent is becoming just way too long :/ I suppose if I can finish the dependency chain there must be a way to latch in a custom script to setup.py to shrink the custom steps back down ...
Your process will likely vary according to what platform you are targeting, but in general, a typical way to get what you are trying to achieve is to download packages on an online machine, copy them over to the offline one, and then install them from a file rather than from a URL or repository).
A possible workflow for RPM-based distros may be:
Install python-pip through binary packages (use rpm or yum-downloadonly, to download the package on an online machine, then copy it over and install it on the offline one with rpm -i python-pip.<whatever-version-and-architecture-you-downloaded>).
On your online machine, use pip install --download <pkgname> to download the packages you need.
scp or rsync the packages to a given directory X onto your offline machine
Use pip install --find-links=<your-dir-here> <pkgname> to install packages on your offline machine.
If you have to replicate the process on many servers, I'd suggest you set up your own repositories behind a firewall. In case of pip, it is very easy, as it's just a matter of telling pip to use a directory as its own index:
$ pip install --no-index --find-links=file:///local/dir/ SomePackage
For RPM or DEB repos is a bit more complicated (but not rocket science!), but possibly also not that necessary, as you really only ought to install python-pip once.
The pip install --download option that #mac mentioned has been deprecated and removed. Instead the documentation states that the pip download method should be used instead. So the workflow should be:
Download the python package or installer using your online machine.
Install python using the offline method used by your package manager or the python installer for windows on the offline machine.
On the online machine use pip download -r requirements.txt where "requirments.txt" contains the packages you will be needing the proper format
Use pip install --find-links=<your-dir-here> <pkgname> to install packages on your offline machine.
I have an external package I want to install into my python virtualenv from a tar file.
What is the best way to install the package?
I've discovered 2 ways that can do it:
Extract the tar file, then run python setup.py install inside of the extracted directory.
pip install packagename.tar.gz from example # 7 in https://pip.pypa.io/en/stable/reference/pip_install/#examples
Is if there is any difference doing them in these 2 ways.
On the surface, both do the same thing: doing either python setup.py install or pip install <PACKAGE-NAME> will install your python package for you, with a minimum amount of fuss.
However, using pip offers some additional advantages that make it much nicer to use.
pip will automatically download all dependencies for a package for you. In contrast, if you use setup.py, you often have to manually search out and download dependencies, which is tedious and can become frustrating.
pip keeps track of various metadata that lets you easily uninstall and update packages with a single command: pip uninstall <PACKAGE-NAME> and pip install --upgrade <PACKAGE-NAME>. In contrast, if you install a package using setup.py, you have to manually delete and maintain a package by hand if you want to get rid of it, which could be potentially error-prone.
You no longer have to manually download your files. If you use setup.py, you have to visit the library's website, figure out where to download it, extract the file, run setup.py... In contrast, pip will automatically search the Python Package Index (PyPi) to see if the package exists there, and will automatically download, extract, and install the package for you. With a few exceptions, almost every single genuinely useful Python library can be found on PyPi.
pip will let you easily install wheels, which is the new standard of Python distribution. More info about wheels.
pip offers additional benefits that integrate well with using virtualenv, which is a program that lets you run multiple projects that require conflicting libraries and Python versions on your computer. More info.
pip is bundled by default with Python as of Python 2.7.9 on the Python 2.x series, and as of Python 3.4.0 on the Python 3.x series, making it even easier to use.
So basically, use pip. It only offers improvements over using python setup.py install.
If you're using an older version of Python, can't upgrade, and don't have pip installed, you can find more information about installing pip at the following links:
Official instructions on installing pip for all operating systems
Instructions on installing pip on Windows (including solutions to common problems)
Instructions on installing pip for Mac OX
pip, by itself, doesn't really require a tutorial. 90% of the time, the only command you really need is pip install <PACKAGE-NAME>. That said, if you're interested in learning more about the details of what exactly you can do with pip, see:
Quickstart guide
Official documentation.
It is also commonly recommended that you use pip and virtualenv together. If you're a beginner to Python, I personally think it'd be fine to start of with just using pip and install packages globally, but eventually I do think you should transition to using virtualenv as you tackle more serious projects.
If you'd like to learn more about using pip and virtualenv together, see:
Why you should be using pip and virtualenv
A non-magical introduction to Pip and Virtualenv for Python beginners
Virtual Environments
python setup.py install is the analog of make install: it’s a limited way to compile and copy files to destination directories. This doesn’t mean that it’s the best way to really install software on your system.
pip is a package manager, which can install, upgrade, list and uninstall packages, like familiar package managers including: dpkg, apt, yum, urpmi, ports etc. Under the hood, it will run python setup.py install, but with specific options to control how and where things end up installed.
In summary: use pip.
The question is about the preferred method to install a local tarball containing a python package, NOT about the advantage of uploading package to an indexing service like PyPi.
As lest I know some software distributor does not upload their package to PyPi, instead asking developers to download package from their website and install.
python setup.py install
This can work but not recommended. It's not necessary to unwrap the tarball file and go into it to run setup.py file.
pip install ../path/to/packagename.tar.gz
This is the way designed and preferred. Concise and align with PyPi-style packages.
More information about pip install can be found here: https://pip.readthedocs.io/en/stable/reference/pip_install/
Is there a way to configure easy_install to avoid having to download the files again when an installation fails?
Update 13 years later: easy_install was removed from Python in January 2021. The python package manager is pip, it caches downloaded packages.
pip (http://pypi.python.org/pypi/pip/) is a drop-in replacement for the easy_install tool and can do that.
Just run easy_install pip and set an environment variable PIP_DOWNLOAD_CACHE to the path you want pip to store the files.
Note that the cache won't work with dependencies that checkout from a source code repository (like svn/git/hg/bzr).
Then use pip install instead of easy_install
Here is my solution using pip, managing even installation of binary packages and usable on both, Linux and Windows. And as requested, it will limit download from PyPi to the minimum, and as extra bonus, on Linux, it allows to speed up repeated installation of packages usually requiring compilation to a fraction of a second.
Setup takes few steps, but I thing, it is worth to do.
Create pip config file
Create pip configuration file (on linux: ~/.pip/pip.conf, on Windows %HOME%\pip\pip.ini)
My one has this content:
[global]
download-cache = /home/javl/.pip/cache
find-links = /home/javl/.pip/packages
[install]
use-wheel = yes
[wheel]
wheel-dir = /home/javl/.pip/packages
Populating cache dir - goes automatically
The cache dir will get cached version of data downloaded from pypi each time, pip attempts to get some package from pypi. It is easy to get it there (no special care needed), but note, that from pip point of view, these are just cashed data downloaded from PyPi, not packages, so in case you use an option --no-index, it will not work.
pip download to populate packages dir
The packages dir is place to put real package files to. E.g. for my favorite package plac, I would do:
$ pip download --dest ~/.pip/packages plac
and the plac package file would appear in that dir. You may even use -r requirements.txt file to do this for multiple packages at once.
These packages are used even with $ pip install --no-index <something>.
Prevent repeated compilation of the same package on Linux
E.g. lxml package requires compliation, and download and compile may take from 45 seconds to minutes. Using wheel format, you may save here a lot.
Install wheel tool, if you do not have it yet:
$ pip install wheel
Create the wheel for lxml (assuming, you have managed to install lxml in past - it requires some libs in the system to be installed):
$ pip wheel lxml
This goes over download, compile, but finally results in lxml whl file being in packages dir.
Since then
$ pip install lxml
or even faster
$ pip install --no-index lxml
will take fraction of a second, as it uses wheel formatted package.
Prepare wheel package from Window setup exe package
(note: this can be prepared even on Linux machine, there is no compilation, only some repacking from exe file into whl.)
download the exe form of the package from pypi, e.g:
$ wget https://pypi.python.org/packages/2.7/l/lxml/lxml-3.2.3.win32-py2.7.exe#md5=14ab978b7f0a3382719b65a1ca938d33
$ dir
lxml-3.2.3.win32-py2.7.exe
convert it to whl
$ wheel convert lxml-3.2.3.win32-py2.7.exe
$ dir
lxml-3.2.3.win32-py2.7.exe
lxml-3.2.3-cp27-none-win32.whl
Test it:
$ pip install lxml
or
$ pip install --no-index lxml
shall be very quick.
Note, that wheel convert can do exactly the same conversion for egg formatted packages.
Let easy_install and setup.py install reuse your packages dir
easy_install and $ python setup.py install do not seem to offer download cache, but allow to use packages we have in our packages dir.
To do so, edit config file for these two tools:
On Linux: $HOME/.pydistutils.cfg
On Windows: %HOME%\pydistutils.cfg
In my case I have here in /home/javl/.pydistutils.cfg:
[easy_install]
find_links = /home/javl/.pip/packages
This config may help even some cases of pip install calls, when pip attempts to install a package, declaring dependency on other ones. As it delegates this task to setup.py call, without the .pydistutils.cfg config it would download the file from PyPi.
Unfortunately, wheel format is not supported in this case (as far as I am aware of).