I want to encode the git hash of the lastest commit at the time of installing as the output of a function in my library.
I am using a structure
.git/
setup.py
|- mylib/
| - __init__.py
| - git.py
whereby git.py contains a dummy, unconfigured, function
def git():
return None
At the time of installing I want to overwrite git.py such that it returns the hash of the latest git commit, e.g.:
def git():
return "2d04d1a10f81d24183df7622c95398d60106dfff"
(whereby the hash is the output of git rev-parse HEAD at the time of installing).
So my question:
How can I overwrite the to-be-installed git.py, using setuptools, without overwriting the file in the source directory?
Current work-around setup.py
import subprocess
git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).strip().decode('UTF-8')
cmd = '''def git():
return "{0:s}"
'''.format(git_hash)
os.rename('mylib/git.py', 'mylib/git.py.bak')
with open('mylib/git.py', 'w') as f:
f.write(cmd)
setup(
name = 'mylib',
version = __version__,
# ...
packages = find_packages(),
)
os.rename('mylib/git.py.bak', 'mylib/git.py')
Which has the down-sight of temporarily changing git.py in the source directory. I'd rather leave that file completely untouched.
With the help of #webknjaz on setuptools' git repository (see issue) I found a solution using setuptools_scm that works:
File structure
setup.py
|- mylib/
| - __init__.py
| - _version.py
setup.py
setup(
name = 'mylib',
packages = find_packages(),
use_scm_version = {'write_to': 'mylib/_version.py'},
setup_requires = ['setuptools_scm'],
)
init.py
from ._version import *
_version.py (dummy, overwritten at install-time)
version = "None"
version_tuple = (0, 0, 0, "None", "None")
I have a framework which working under py.test. py.test can generate beauty reports with params --html and --junitxml. But clients that using my framework not always type this params to command line where they using py.test. I want make py.test to generate reports always when the py.test used with my framework. And i want to put this reports with log folder. So i need to generate path for report in runtime. Can i do this by fixtures? Or maybe by the plugin API?
Putting this in conftest.py will suffice:
def pytest_configure(config):
if config.option.xmlpath is None:
config.option.xmlpath = get_custom_xml_path() # implement this
The accepted answer is probably a bit more complicated than necessary for most people for a few reasons:
The decorator doesn't help. It doesn't matter when this executes.
There is no need make a custom LogXML since you can just set the property here and it will be used.
slaveinput is specific to a pytest plugin xdist. I don't think there is any need to check for that, especially if you don't use xdist.
First of all, if you want to implicitly add the command line args to pytest, you can use the pytest.ini placed in the tests root dir with the addopts config value:
[pytest]
addopts=--verbose --junit-xml=/tmp/myreport.xml # etc
Of course, if you want to dynamically calculate the directory to store the reports, then you can't put it in the config and will need to extend pytest. The best spot would be the pytest_configure hook. Example:
# conftest.py
import tempfile
import pytest
from _pytest.junitxml import LogXML
#pytest.hookimpl(tryfirst=True)
def pytest_configure(config):
if config.option.xmlpath: # was passed via config or command line
return # let pytest handle it
if not hasattr(config, 'slaveinput'):
with tempfile.NamedTemporaryFile(suffix='.xml') as tmpfile:
xmlpath = tmpfile.name
config._xml = LogXML(xmlpath, config.option.junitprefix, config.getini('junit_suite_name'))
config.pluginmanager.register(config._xml)
If you remove the first if block, then pytest will completely ignore --junit-xml arg passed via command line or in addopts value in config.
Example run:
$ pytest
=================================== test session starts ====================================
platform darwin -- Python 3.6.3, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
rootdir: /Users/hoefling/projects/private/stackoverflow/so-48320357, inifile:
plugins: forked-0.2, asyncio-0.8.0, xdist-1.22.0, mock-1.6.3, hypothesis-3.44.4
collected 1 item
test_spam.py . [100%]
--- generated xml file: /var/folders/_y/2qk6029j4c7bwv0ddk3p96r00000gn/T/tmp1tleknm3.xml ---
================================ 1 passed in 0.01 seconds ==================================
The xml report is now put in a tempfile.
Configure pytest.ini file with parameters:
# content of pytest.ini
[pytest]
addopts = --html=report.html --self-contained-html
;addopts = -vv -rw --html=./results/report.html --self-contained-html
#hoefling's answer worked perfectly for me in conftest.py. the code looks simpler there.
def pytest_configure(config):
if not config.option.xmlpath and not hasattr(config, 'slaveinput'):
xmlpath = "test_report_" + str(int(time.time())) + ".xml"
config._xml = LogXML(xmlpath, config.option.junitprefix, config.getini('junit_suite_name'))
config.pluginmanager.register(config._xml)
Just to keep things more clear, pytest uses argparse and the request.config.option is a argparse.Namespace object. Then, if you would like to simulate a command line option as pytest ... --docker-compose-remove-volumes, you can directly attribute the option docker_compose_remove_volumes to request.config.option (because --docker-compose-remove-volumes is converted to docker_compose_remove_volumes by argparse module).
This examples inverts the default option for --docker-compose-remove-volumes which is false. But allow you to enable it back by providing --keep-containers option to pytest.
def pytest_addoption(parser):
parser.addoption("--keep-containers", action="store_true", default=False,
help="Keeps docker-compose on failure.")
#pytest.fixture(scope='session', autouse=True)
def load_env(request):
is_to_keep_container = request.config.getoption("--keep-containers")
if not is_to_keep_container:
request.config.option.docker_compose_remove_volumes = True
I have a requirements.txt file that I'm using with Travis-CI. It seems silly to duplicate the requirements in both requirements.txt and setup.py, so I was hoping to pass a file handle to the install_requires kwarg in setuptools.setup.
Is this possible? If so, how should I go about doing it?
Here is my requirements.txt file:
guessit>=0.5.2
tvdb_api>=1.8.2
hachoir-metadata>=1.3.3
hachoir-core>=1.3.3
hachoir-parser>=1.3.4
On the face of it, it does seem that requirements.txt and setup.py are silly duplicates, but it's important to understand that while the form is similar, the intended function is very different.
The goal of a package author, when specifying dependencies, is to say "wherever you install this package, these are the other packages you need, in order for this package to work."
In contrast, the deployment author (which may be the same person at a different time) has a different job, in that they say "here's the list of packages that we've gathered together and tested and that I now need to install".
The package author writes for a wide variety of scenarios, because they're putting their work out there to be used in ways they may not know about, and have no way of knowing what packages will be installed alongside their package. In order to be a good neighbor and avoid dependency version conflicts with other packages, they need to specify as wide a range of dependency versions as can possibly work. This is what install_requires in setup.py does.
The deployment author writes for a very different, very specific goal: a single instance of an installed application or service, installed on a particular computer. In order to precisely control a deployment, and be sure that the right packages are tested and deployed, the deployment author must specify the exact version and source-location of every package to be installed, including dependencies and dependency's dependencies. With this spec, a deployment can be repeatably applied to several machines, or tested on a test machine, and the deployment author can be confident that the same packages are deployed every time. This is what a requirements.txt does.
So you can see that, while they both look like a big list of packages and versions, these two things have very different jobs. And it's definitely easy to mix this up and get it wrong! But the right way to think about this is that requirements.txt is an "answer" to the "question" posed by the requirements in all the various setup.py package files. Rather than write it by hand, it's often generated by telling pip to look at all the setup.py files in a set of desired packages, find a set of packages that it thinks fits all the requirements, and then, after they're installed, "freeze" that list of packages into a text file (this is where the pip freeze name comes from).
So the takeaway:
setup.py should declare the loosest possible dependency versions that are still workable. Its job is to say what a particular package can work with.
requirements.txt is a deployment manifest that defines an entire installation job, and shouldn't be thought of as tied to any one package. Its job is to declare an exhaustive list of all the necessary packages to make a deployment work.
Because these two things have such different content and reasons for existing, it's not feasible to simply copy one into the other.
References:
install_requires vs Requirements files from the Python packaging user guide.
You can flip it around and list the dependencies in setup.py and have a single character — a dot . — in requirements.txt instead.
Alternatively, even if not advised, it is still possible to parse the requirements.txt file (if it doesn't refer any external requirements by URL) with the following hack (tested with pip 9.0.1):
install_reqs = parse_requirements('requirements.txt', session='hack')
This doesn't filter environment markers though.
In old versions of pip, more specifically older than 6.0, there is a public API that can be used to achieve this. A requirement file can contain comments (#) and can include some other files (--requirement or -r). Thus, if you really want to parse a requirements.txt you can use the pip parser:
from pip.req import parse_requirements
# parse_requirements() returns generator of pip.req.InstallRequirement objects
install_reqs = parse_requirements(<requirements_path>)
# reqs is a list of requirement
# e.g. ['django==1.5.1', 'mezzanine==1.4.6']
reqs = [str(ir.req) for ir in install_reqs]
setup(
...
install_requires=reqs
)
It can't take a file handle. The install_requires argument can only be a string or a list of strings.
You can, of course, read your file in the setup script and pass it as a list of strings to install_requires.
import os
from setuptools import setup
with open('requirements.txt') as f:
required = f.read().splitlines()
setup(...
install_requires=required,
...)
Requirements files use an expanded pip format, which is only useful if you need to complement your setup.py with stronger constraints, for example specifying the exact urls some of the dependencies must come from, or the output of pip freeze to freeze the entire package set to known-working versions. If you don't need the extra constraints, use only a setup.py. If you feel like you really need to ship a requirements.txt anyway, you can make it a single line:
.
It will be valid and refer exactly to the contents of the setup.py that is in the same directory.
While not an exact answer to the question, I recommend Donald Stufft's blog post at https://caremad.io/2013/07/setup-vs-requirement/ for a good take on this problem. I've been using it to great success.
In short, requirements.txt is not a setup.py alternative, but a deployment complement. Keep an appropriate abstraction of package dependencies in setup.py. Set requirements.txt or more of 'em to fetch specific versions of package dependencies for development, testing, or production.
E.g. with packages included in the repo under deps/:
# fetch specific dependencies
--no-index
--find-links deps/
# install package
# NOTE: -e . for editable mode
.
pip executes package's setup.py and installs the specific versions of dependencies declared in install_requires. There's no duplicity and the purpose of both artifacts is preserved.
Using parse_requirements is problematic because the pip API isn't publicly documented and supported. In pip 1.6, that function is actually moving, so existing uses of it are likely to break.
A more reliable way to eliminate duplication between setup.py and requirements.txt is to specific your dependencies in setup.py and then put -e . into your requirements.txt file. Some information from one of the pip developers about why that's a better way to go is available here: https://caremad.io/blog/setup-vs-requirement/
First, I believe parsing requirements.txt to fill the list of dependencies in package metadata is not a good idea. The requirements.txt file and the list of "install dependencies" are two different concepts, they are not interchangeable. It should be the other way around, the list of dependencies in package metadata should be considered as some kind of source of truth, and files such as requirements.txt should be generated from there. For example with a tool such as pip-compile. See the notes at the bottom of this answer.
But everyone has different needs, that lead to different workflows. So with that said... There are 3 possibilities to handle this, depending on where you want your project's package metadata to be written: pyproject.toml, setup.cfg, or setup.py.
Words of caution!
If you insist on having the list of dependencies in package metadata be read from a requirements.txt file then make sure that this requirements.txt file is included in the "source distribution" (sdist) otherwise installation will fail, for obvious reasons.
These techniques will work only for simple requirements.txt files. See Requirements parsing in the documentation page for pkg_resources to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not supported and it will cause a failure.
pyproject.toml
[project]
# ...
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
# ...
dependencies = requirements.txt
setup.cfg
Since setuptools version 62.6 it is possible to write something like this in setup.cfg:
[options]
install_requires = file: requirements.txt
setup.py
It is possible to parse a relatively simple requirements.txt file from a setuptools setup.py script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources.
It could more or less look like this:
#!/usr/bin/env python
import pathlib
import pkg_resources
import setuptools
with pathlib.Path('requirements.txt').open() as requirements_txt:
install_requires = [
str(requirement)
for requirement
in pkg_resources.parse_requirements(requirements_txt)
]
setuptools.setup(
install_requires=install_requires,
)
Notes:
https://github.com/pypa/setuptools/issues/1951#issuecomment-1431345869
https://caremad.io/posts/2013/07/setup-vs-requirement/
https://setuptools.pypa.io/en/latest/history.html#v62-6-0
See also this other answer: https://stackoverflow.com/a/59971236
Most of the other answers above don't work with the current version of pip's API. Here is the correct* way to do it with the current version of pip (6.0.8 at the time of writing, also worked in 7.1.2. You can check your version with pip -V).
from pip.req import parse_requirements
from pip.download import PipSession
install_reqs = parse_requirements(<requirements_path>, session=PipSession())
reqs = [str(ir.req) for ir in install_reqs]
setup(
...
install_requires=reqs
....
)
* Correct, in that it is the way to use parse_requirements with the current pip. It still probably isn't the best way to do it, since, as posters above said, pip doesn't really maintain an API.
Install the current package in Travis. This avoids the use of a requirements.txt file.
For example:
language: python
python:
- "2.7"
- "2.6"
install:
- pip install -q -e .
script:
- python runtests.py
from pip.req import parse_requirements did not work for me and I think it's for the blank lines in my requirements.txt, but this function does work
def parse_requirements(requirements):
with open(requirements) as f:
return [l.strip('\n') for l in f if l.strip('\n') and not l.startswith('#')]
reqs = parse_requirements(<requirements_path>)
setup(
...
install_requires=reqs,
...
)
The following interface became deprecated in pip 10:
from pip.req import parse_requirements
from pip.download import PipSession
So I switched it just to simple text parsing:
with open('requirements.txt', 'r') as f:
install_reqs = [
s for s in [
line.split('#', 1)[0].strip(' \t\n') for line in f
] if s != ''
]
BEWARE OF parse_requirements BEHAVIOUR!
Please note that pip.req.parse_requirements will change underscores to dashes. This was enraging me for a few days before I discovered it. Example demonstrating:
from pip.req import parse_requirements # tested with v.1.4.1
reqs = '''
example_with_underscores
example-with-dashes
'''
with open('requirements.txt', 'w') as f:
f.write(reqs)
req_deps = parse_requirements('requirements.txt')
result = [str(ir.req) for ir in req_deps if ir.req is not None]
print result
produces
['example-with-underscores', 'example-with-dashes']
If you don't want to force your users to install pip, you can emulate its behavior with this:
import sys
from os import path as p
try:
from setuptools import setup, find_packages
except ImportError:
from distutils.core import setup, find_packages
def read(filename, parent=None):
parent = (parent or __file__)
try:
with open(p.join(p.dirname(parent), filename)) as f:
return f.read()
except IOError:
return ''
def parse_requirements(filename, parent=None):
parent = (parent or __file__)
filepath = p.join(p.dirname(parent), filename)
content = read(filename, parent)
for line_number, line in enumerate(content.splitlines(), 1):
candidate = line.strip()
if candidate.startswith('-r'):
for item in parse_requirements(candidate[2:].strip(), filepath):
yield item
else:
yield candidate
setup(
...
install_requires=list(parse_requirements('requirements.txt'))
)
I created a reusable function for this. It actually parses an entire directory of requirements files and sets them to extras_require.
Latest always available here: https://gist.github.com/akatrevorjay/293c26fefa24a7b812f5
import glob
import itertools
import os
# This is getting ridiculous
try:
from pip._internal.req import parse_requirements
from pip._internal.network.session import PipSession
except ImportError:
try:
from pip._internal.req import parse_requirements
from pip._internal.download import PipSession
except ImportError:
from pip.req import parse_requirements
from pip.download import PipSession
def setup_requirements(
patterns=[
'requirements.txt', 'requirements/*.txt', 'requirements/*.pip'
],
combine=True):
"""
Parse a glob of requirements and return a dictionary of setup() options.
Create a dictionary that holds your options to setup() and update it using this.
Pass that as kwargs into setup(), viola
Any files that are not a standard option name (ie install, tests, setup) are added to extras_require with their
basename minus ext. An extra key is added to extras_require: 'all', that contains all distinct reqs combined.
Keep in mind all literally contains `all` packages in your extras.
This means if you have conflicting packages across your extras, then you're going to have a bad time.
(don't use all in these cases.)
If you're running this for a Docker build, set `combine=True`.
This will set `install_requires` to all distinct reqs combined.
Example:
>>> import setuptools
>>> _conf = dict(
... name='mainline',
... version='0.0.1',
... description='Mainline',
... author='Trevor Joynson <github#trevor.joynson,io>',
... url='https://trevor.joynson.io',
... namespace_packages=['mainline'],
... packages=setuptools.find_packages(),
... zip_safe=False,
... include_package_data=True,
... )
>>> _conf.update(setup_requirements())
>>> # setuptools.setup(**_conf)
:param str pattern: Glob pattern to find requirements files
:param bool combine: Set True to set install_requires to extras_require['all']
:return dict: Dictionary of parsed setup() options
"""
session = PipSession()
# Handle setuptools insanity
key_map = {
'requirements': 'install_requires',
'install': 'install_requires',
'tests': 'tests_require',
'setup': 'setup_requires',
}
ret = {v: set() for v in key_map.values()}
extras = ret['extras_require'] = {}
all_reqs = set()
files = [glob.glob(pat) for pat in patterns]
files = itertools.chain(*files)
for full_fn in files:
# Parse
reqs = {
str(r.req)
for r in parse_requirements(full_fn, session=session)
# Must match env marker, eg:
# yarl ; python_version >= '3.0'
if r.match_markers()
}
all_reqs.update(reqs)
# Add in the right section
fn = os.path.basename(full_fn)
barefn, _ = os.path.splitext(fn)
key = key_map.get(barefn)
if key:
ret[key].update(reqs)
extras[key] = reqs
extras[barefn] = reqs
if 'all' not in extras:
extras['all'] = list(all_reqs)
if combine:
extras['install'] = ret['install_requires']
ret['install_requires'] = list(all_reqs)
def _listify(dikt):
ret = {}
for k, v in dikt.items():
if isinstance(v, set):
v = list(v)
elif isinstance(v, dict):
v = _listify(v)
ret[k] = v
return ret
ret = _listify(ret)
return ret
__all__ = ['setup_requirements']
if __name__ == '__main__':
reqs = setup_requirements()
print(reqs)
Another possible solution...
def gather_requirements(top_path=None):
"""Captures requirements from repo.
Expected file format is: requirements[-_]<optional-extras>.txt
For example:
pip install -e .[foo]
Would require:
requirements-foo.txt
or
requirements_foo.txt
"""
from pip.download import PipSession
from pip.req import parse_requirements
import re
session = PipSession()
top_path = top_path or os.path.realpath(os.getcwd())
extras = {}
for filepath in tree(top_path):
filename = os.path.basename(filepath)
basename, ext = os.path.splitext(filename)
if ext == '.txt' and basename.startswith('requirements'):
if filename == 'requirements.txt':
extra_name = 'requirements'
else:
_, extra_name = re.split(r'[-_]', basename, 1)
if extra_name:
reqs = [str(ir.req) for ir in parse_requirements(filepath, session=session)]
extras.setdefault(extra_name, []).extend(reqs)
all_reqs = set()
for key, values in extras.items():
all_reqs.update(values)
extras['all'] = list(all_reqs)
return extras
and then to use...
reqs = gather_requirements()
install_reqs = reqs.pop('requirements', [])
test_reqs = reqs.pop('test', [])
...
setup(
...
'install_requires': install_reqs,
'test_requires': test_reqs,
'extras_require': reqs,
...
)
Cross posting my answer from this SO question for another simple, pip version proof solution.
try: # for pip >= 10
from pip._internal.req import parse_requirements
from pip._internal.download import PipSession
except ImportError: # for pip <= 9.0.3
from pip.req import parse_requirements
from pip.download import PipSession
requirements = parse_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt'), session=PipSession())
if __name__ == '__main__':
setup(
...
install_requires=[str(requirement.req) for requirement in requirements],
...
)
Then just throw in all your requirements under requirements.txt under project root directory.
Yet another parse_requirements hack that also parses environment markers into extras_require:
from collections import defaultdict
from pip.req import parse_requirements
requirements = []
extras = defaultdict(list)
for r in parse_requirements('requirements.txt', session='hack'):
if r.markers:
extras[':' + str(r.markers)].append(str(r.req))
else:
requirements.append(str(r.req))
setup(
...,
install_requires=requirements,
extras_require=extras
)
It should support both sdist and binary dists.
As stated by others, parse_requirements has several shortcomings, so this is not what you should do on public projects, but it may suffice for internal/personal projects.
I did this:
import re
def requirements(filename):
with open(filename) as f:
ll = f.read().splitlines()
d = {}
for l in ll:
k, v = re.split(r'==|>=', l)
d[k] = v
return d
def packageInfo():
try:
from pip._internal.operations import freeze
except ImportError:
from pip.operations import freeze
d = {}
for kv in freeze.freeze():
k, v = re.split(r'==|>=', kv)
d[k] = v
return d
req = getpackver('requirements.txt')
pkginfo = packageInfo()
for k, v in req.items():
print(f'{k:<16}: {v:<6} -> {pkginfo[k]}')
Here is a complete hack (tested with pip 9.0.1) based on Romain's answer that parses requirements.txt and filters it according to current environment markers:
from pip.req import parse_requirements
requirements = []
for r in parse_requirements('requirements.txt', session='hack'):
# check markers, such as
#
# rope_py3k ; python_version >= '3.0'
#
if r.match_markers():
requirements.append(str(r.req))
print(requirements)