I have a python project hosted on PyPI. I've discovered I have some dependency conflicts that need to be pinned. I know pip looks at the install_requires key in setup.py for dependencies, but I've read it's best to place pinned dependencies in a requirements.txt file. I've included this file (see below) using pip freeze, but I am unsure whether pip install project is sufficient to install dependencies as well.
# requirments.txt
numpy==1.9.2
pandas==0.16.2
I would like to make the simplest installation process for the user. For a package hosted on PyPI:
How do I setup requirements to simply pip install a project and include all of it's pinned dependencies automatically (similar to conda)?
Must install_requires=['numpy', 'pandas'] be included? If so, how do I best set it up to install the pinned versions only.
Related
I'm looking to pin Python dependencies installed as part of the pip build system. Here is the scenario I'm working with:
I'm using a requirements file to install a 3rd party package (from a git+ssh source, but I doubt that matters).
i.e. requirements.txt
git+ssh://git#gitremote/some/path/included-project.git#1.2.3#egg=included-project
Then, of course:
pip install -r requirements.txt
That "included-project" has a pyproject.toml that looks like this:
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
Because poetry is not pinned, it upgrades to the newest version. However, the new version of poetry appears to be pulling in keyring, which pulls in SecretStorage, which pulls in cryptography... none of which are pinned. The new version of cryptography that's pulled in more recently started breaking my build because it's missing Rust.
I want my build to be deterministic, so I'd like to pin all the dependencies that poetry installs as part of the "build-system". This build-system and pep517 is new to me; my understanding is that it creates a virtualenv of sorts just to install the build-backend (poetry and deps) and then abandons that after the build is done. And as I understand it, that virtualenv is different than the virtualenv that is created by the normal install.
As such, I'm not sure how to pin packages inside of that temporary virtualenv that build-system creates. How can I do this?
In my requirements.txt I have packages defined in following manner:
Django ~= 2.2.0
It means that when I use pip install -r requirements.txt pip will find the latest available 2.2.x version and install it along with all dependencies.
What I need is requirements-formatted list of all packages with explicit versions that will be installed but without actually installing any packages. So example output would be something like:
Django==2.2.23
package1==0.2.1
package2==1.4.3
...
So in other words I'm looking for something like pip freeze results but without installing anything.
pip-compile is what you need!
Doc: https://github.com/jazzband/pip-tools)
python -m pip install pip-tools
pip-compile requirements.txt --output-file requirements-all.txt
The pip-compile command lets you compile a requirements.txt file from your dependencies, this way you can pip install you dependencies to always have the same environment
TL;DR
Try pipdetree or pip-tree.
Explanation
pip, contrary to most package managers, doesn't have a big dependency graph to look up. What it does is that it lets arbitrary setup code to be executed, which automatically pulls the dependencies. This means that, for example, a package could manage their dependencies in an other way than putting them in requirements.txt (see fastai for an example of a project that handles the dependencies differently).
So, there is, theoretically, no other way to see all the dependencies than to actually run an install on an isolated environment, see what was pulled, then delete the environment (because it could potentially be the same part of the code that does the installation and that brings the dependencies). You could actually do that with venv.
In practice, tools like pipdetree or pip-tree fetch the dependencies based on some standardization of the requirements (most packages separate the dependencies and the installation, and actually let pip handle both).
I am working in a team and wrote some python code that uses libraries that need to be installed separately (because they are not part of standard python distribution). How should I specify those ? What is the right/correct/pythonic way to do this?
I personally use pip install -r requirements.txt
https://pip.pypa.io/en/latest/user_guide.html#requirements-files
Check out tool called pip. It's what most python projects use these days.
Typically, one would do the following (for example, we want to install the requests package for our new project):
pip install requests
and then
pip freeze > requirements.txt
Now, we have installed requests on our system and saved the dependency version to a file which we can distribute with our project.
At this point, requirements.txt contains:
requests==2.7.0
To install the same set of requirements (in our case only the requests package) on some other system, one would do the following:
pip install -r requirements.txt
You need to make a setup.py file for your package that specifies required packages. You may need to reorganize your file structure and include data and a manifest.
Then create a distribution of your package, EG: a wheel file.
Then when using pip install your_package_distro.whl, pip will determine your packages dependencies are met and install them from PyPI unless you specify another package source (EG: https://pypi.anaconda.org/)
Read through the following references to distribute your code:
Python 2.7.10 documentation - Distributing Python Modules, Section 2: Writing the Setup Script
Setuptools - Building and Distributing Packages with Setuptools
Hitchhiker's Guide to Packaging
Hitchhiker's Guide to Python - Packaging your Code
I am using some custom modules, not available on PyPI. Is it possible to manage the dependency through virtualenv?
Yes. pip can install packages from -
PyPI (and other indexes) using requirement specifiers.
VCS project urls.
Local project directories.
Local or remote source archives.
So all you have to do it provide the location of the module from some VCS or local directory in the requirements.txt file and pip install -r requirements.txt after activating virtualenv, and it'll work. More examples can be found at pip documentation.
Just keep in mind that pip will run python setup.py install after downloading and extracting your custom module. So you must package your module to support that.
One thing I like about Rails projects is that when deploying to a remote server, if everything is set up correctly you can just do:
$: bundle install
And the system will install the various dependencies (ruby gems) needed to run the project.
Is there something similar for Python/Django?
You can freeze requirements. This generates a list of all the Python modules that your project needs. I believe bundle is similar in concept.
For example:
virtualenv --no-site-packages myproject_env # create a blank Python virtual environment
source myproject_env/bin/activate # activate it
(myproject_env)$ pip install django # install django into the virtual environment
(myproject_env)$ pip install other_package # etc.
...
(myproject_env)$ pip freeze > requirements.txt
The last line generates a text file will all the packages that were installed in your custom environment. You can use that file to install the same requirements on other servers:
pip install -r requirements.txt
Of course you don't need to use pip, you can create the requirements file by hand; it doesn't have any special syntax requirements. Just a package and (possibly) version identifier on each line. Here is a sample of a typical django project with some extra packages:
Django==1.4
South==0.7.4
Werkzeug==0.8.3
amqplib==1.0.2
anyjson==0.3.1
celery==2.5.1
django-celery==2.5.1
django-debug-toolbar==0.9.4
django-extensions==0.8
django-guardian==1.0.4
django-picklefield==0.2.0
kombu==2.1.4
psycopg2==2.4.5
python-dateutil==2.1
six==1.1.0
wsgiref==0.1.2
xlwt==0.7.3
The closest is probably virtualenv, pip and a requirements file.
With those 3 ingredients is quite easy to write a simple bootstrap scripts.
More demanding and complex is buildout. But I would only go for it if virtualenv and pip are not sufficient.
And if you extend this approach with fabric and optional cuisine, you already have your project deployment automated. Check out these links for more information:
http://www.caktusgroup.com/blog/2010/04/22/basic-django-deployment-with-virtualenv-fabric-pip-and-rsync/
http://www.saltycrane.com/blog/2009/05/notes-using-pip-and-virtualenv-django/
http://www.saltycrane.com/blog/2009/05/notes-using-pip-and-virtualenv-django/