I have package "A" with a setup.py and an extras_requires line like:
extras_require = {
'ssh': ['paramiko'],
},
And a package "B" that depends on util:
install_requires = ['A[ssh]']
If I run python setup.py install on package B, which uses setuptools.command.easy_install under the hood, the extras_requires is correctly resolved, and paramiko is installed.
However, if I run pip /path/to/B or pip hxxp://.../b-version.tar.gz, package A is installed, but paramiko is not.
Because pip "installs from source", I'm not quite sure why this isn't working. It should be invoking the setup.py of B, then resolving & installing dependencies of both B and A.
Is this possible with pip?
We use setup.py and pip to manage development dependencies for our packages, though you need a newer version of pip (we're using 1.4.1 currently).
#!/usr/bin/env python
from setuptools import setup
from myproject import __version__
required = [
'gevent',
'flask',
...
]
extras = {
'develop': [
'Fabric',
'nose',
]
}
setup(
name="my-project",
version=__version__,
description="My awsome project.",
packages=[
"my_project"
],
include_package_data=True,
zip_safe=False,
scripts=[
'runmyproject',
],
install_requires=required,
extras_require=extras,
)
To install the package:
$ pip install -e . # only installs "required"
To develop:
$ pip install -e .[develop] # installs develop dependencies
This is suppported since pip 1.1, which was released in February 2012 (one year after this question was asked).
The answer from #aaronfay is completely correct but it may be nice to point out that if you're using zsh that the install command pip install -e .[dev] needs to be replaced by pip install -e ".[dev]".
Related
I have a project that makes use of certain C package. Building the package is done as below in a setup.py file:
from setuptools import setup
import torch
if torch.cuda.is_available():
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
setup(
name='iou3d',
ext_modules=[
CUDAExtension('iou3d_cuda', [
'src/iou3d.cpp',
'src/iou3d_kernel.cu',
],
extra_compile_args={'cxx': ['-g'],
'nvcc': ['-O2']})
],
cmdclass={'build_ext': BuildExtension})
Now this is being setup correctly if I already have pytorch installed in my environment.
But if I am doing a fresh install on a clean environment using requirements.txt, and I want to install everything in one shot by pip install -r requirements.txt I am not sure how can I get to install pytorch first to be able to import it in the setup.py.
Appreciate your help.
This is an extension of SO setup.py ignores full path dependencies, instead looks for "best match" in pypi
I am trying to write setup.py to install a proprietary package from a .tar.gz file on an internal web site. Unfortunately for me the prop package name duplicates a public package in the public PyPI, so I need to force install of the proprietary package at a specific version. I'm building a docker image from a Debian-Buster base image, so pip, setuptools and tox are all freshly installed, the image brings python 3.8 and pip upgrades itself to version 21.2.4.
Solution 1 - dependency_links
I followed the instructions at the post linked above to put the prop package in install_requires and dependency_links. Here are the relevant lines from my setup.py:
install_requires=["requests", "proppkg==70.1.0"],
dependency_links=["https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"]
Installation is successful in Debian-Buster if I run python3 setup.py install in my package directory. I see the proprietary package get downloaded and installed.
Installation fails if I run pip3 install . also tox (version 3.24.4) fails similarly. In both cases, pip shows a message "Looking in indexes" then fails with "ERROR: Could not find a version that satisfies the requirement".
Solution 2 - PEP 508
Studying SO answer pip ignores dependency_links in setup.py which states that dependency_links is deprecated, I started over, revised setup.py to have:
install_requires=[
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
Installation is successful in Debian-Buster if I run pip3 install . in my package directory. Pip shows a message "Looking in indexes" but still downloads and installs the proprietary package successfully.
Installation fails in Debian-Buster if I run python3 setup.py install in my package directory. I see these messages:
Searching for proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0
..
Reading https://pypi.org/simple/proppkg/
..
error: Could not find suitable distribution for Requirement.parse(...).
Tox also fails in this scenario as it installs dependencies.
Really speculating now, it almost seems like there's an ordering issue. Tox invokes pip like this:
python -m pip install --exists-action w .tox/.tmp/package/1/te-0.3.5.zip
In that output I see "Collecting proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0" as the first step. That install fails because it fails to import package requests. Then tox continues collecting other dependencies. Finally tox reports as its last step "Collecting requests" (and that succeeds). Do I have to worry about ordering of install steps?
I'm starting to think that maybe the proprietary package is broken. I verified that the prop package setup.py has requests in its install_requires entry. Not sure what else to check.
Workaround solution
My workaround is installing the proprietary package in the docker image as a separate step before I install my own package, just by running pip3 install https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz. The setup.py has the PEP508 URL in install_requires. Then pip and tox find the prop package in the pip cache, and work fine.
Please suggest what to try for the latest pip and tox, or if this is as good as it gets, thanks in advance.
Update - add setup.py
Here's a (slightly sanitized) version of my package's setup.py
from setuptools import setup, find_packages
def get_version():
"""
read version string
"""
version_globals = {}
with open("te/version.py") as fp:
exec(fp.read(), version_globals)
return version_globals['__version__']
setup(
name="te",
version=get_version(),
packages=find_packages(exclude=["tests.*", "tests"]),
author="My Name",
author_email="email#mycompany.com",
description="My Back-End Server",
entry_points={"console_scripts": [
"te-be=te.server:main"
]},
python_requires=">=3.7",
install_requires=["connexion[swagger-ui]",
"Flask",
"gevent",
"redis",
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
package_data={"te": ["openapi_te.yml"]},
include_package_data=True, # read MANIFEST.in
)
I have a Python project that doesn't contain requirements.txt.
But it has a pyproject.toml file.
How can I download packages (dependencies) required by this Python project and declared in pyproject.toml using the Pip package manager (instead of the build tool Poetry).
So instead of pip download -r requirements.txt, something like pip download -r pyproject.toml.
Here is an example of .toml file:
[build-system]
requires = [
"flit_core >=3.2,<4",
]
build-backend = "flit_core.buildapi"
[project]
name = "aedttest"
authors = [
{name = "Maksim Beliaev", email = "beliaev.m.s#gmail.com"},
{name = "Bo Yang", email = "boy#kth.se"},
]
readme = "README.md"
requires-python = ">=3.7"
classifiers = ["License :: OSI Approved :: MIT License"]
dynamic = ["version", "description"]
dependencies = [
"pyaedt==0.4.7",
"Django==3.2.8",
]
[project.optional-dependencies]
test = [
"black==21.9b0",
"pre-commit==2.15.0",
"mypy==0.910",
"pytest==6.2.5",
"pytest-cov==3.0.0",
]
deploy = [
"flit==3.4.0",
]
to install core dependencies you run:
pip install .
if you need test(develop) environment (we use test because it is a name defined in .toml file, you can use any):
pip install .[test]
To install from Wheel:
pip install C:\git\aedt-testing\dist\aedttest-0.0.1-py3-none-any.whl[test]
pip supports installing pyproject.toml dependencies natively.
As of version 10.0, pip supports projects declaring dependencies that are required at install time using a pyproject.toml file, in the form described in PEP 518. When building a project, pip will install the required dependencies locally, and make them available to the build process. Furthermore, from version 19.0 onwards, pip supports projects specifying the build backend they use in pyproject.toml, in the form described in PEP 517.
From the project's root, use pip's local project install:
python -m pip install .
You can export the dependencies to a requirements.txt and use pip download afterwards:
poetry export -f requirements.txt > requirements.txt
pip download -r requirements.txt
I am unable to install the local packages using setup.py
Here is the project structure:
my-project/
lib/
local1/
local1.1.0.whl
index.html
local2/
local2.1.0.whl
index.html
setup.py
setup.py
import os
from setuptools import setup
setup(name='my project',
version='1.0',
description='my project',
install_requires=[
'lxml >= 4.3.0',
'local1 # file://localhost/{}/lib/local1/local1.1.0.whl'.format(os.getcwd()),
'local2 # file://localhost/{}/lib/local2/local2.2.0.whl'.format(os.getcwd()),
]
)
I can install if I put the dependencies in a requirements.txt file and use pip install -r requirements.txt --extra-index-url lib/, but I want to know why is it not possible to do python setup.py install or if I am missing something.
This is the error that I get -
No local packages or working download links found for local2# file://localhost//Users/anusha/Documents/my-project/lib/local2/local2.1.0.whl
error: Could not find suitable distribution for Requirement.parse('local2# file://localhost//Users/anusha/Documents/my-project/lib/local2/local2.1.0.whl')
On searching, I found this issue on github, but does not give me any pointers or solution as to how it worked.
Any help is welcome, thanks in advance!
Note this comment from pganssle in the discussion "Setuptools install fails with PEP508 URLs" in setuptools's issue tracker:
Our policy to date has been that if using pip install fixes your problem, you should use pip install and we won't fix the issue.
I believe this is in line with the current evolution of the packaging tools and techniques in the Python community. So if your setuptools-based project with this requirement notation can be installed via pip install . and pip install --editable ., then look no further.
A more complete (exhaustive) article on the topic:
Paul Ganssle's "Why you shouldn't invoke setup.py directly"
I have a structure of the directory as such with foobar and alphabet data directories together with the code something.py:
\mylibrary
\packages
\foobar
foo.zip
bar.zip
\alphabet
abc.zip
xyz.zip
something.py
setup.py
And the goal is such that users can pip install the module as such:
pip install mylibrary[alphabet]
And that'll only include the data from the packages/alphabet/* data and the python code. Similar behavior should be available for pip install mylibrary[foobar].
If the user installs without the specification:
pip install mylibrary
Then it'll include all the data directories under packages/.
Currently, I've tried writing the setup.py with Python3.5 as such:
import glob
from setuptools import setup, find_packages
setup(
name = 'mylibrary',
packages = ['packages'],
package_data={'packages':glob.glob('packages' + '/**/*.txt', recursive=True)},
)
That will create a distribution with all the data directories when users do pip install mylibrary.
How should I change the setup.py such that specific pip installs like pip install mylibrary[alphabet] is possible?
Firs you have to package and publish alphabet and foobar as a separate packages as pip install mylibrary[alphabet] means
pip install mylibrary
pip install alphabet
After that add alphabet and foobar as extras:
setup(
…,
extras = {
'alphabet': ['alphabet'],
'foobar': ['foobar'],
}
)
The keys in the dictionary are the names used in pip install mylibrary[EXTRA_NAME], the values are a list of package names that will be installed from PyPI.
PS. And no, you cannot use extras to install some data files that are not available as packages from PyPI.