I'm trying to collect python code in a package gnn_pylib and install it in my conda environment. My package will require opencv, which has been installed in my conda environment via:
conda install -c conda-forge opencv
I can run cv2 functions correctly, and I can call functions in the packages using cv2 functions successfully:
import gnn_pylib
gnn_pylib.show()
But when i try to install the package running pip install -e .
from the gnn_pylib directory I get the following error:
Collecting cv2 (from gnn-pylib==0.1)
Could not find a version that satisfies the requirement cv2 (from gnn-pylib==0.1) (from versions: )
No matching distribution found for cv2 (from gnn-pylib==0.1)
Is there something i am missing? should I some way inform pip but my conda opencv?
The package has the following structure:
gnn_pylib/
gnn_pylib/
__init__.py
show.py
setup.py
__init__.py is as follows:
from .show import foo
show.py is as follows:
import cv2
import numpy as np
def foo():
cv2.imshow("random", np.random.rand(10,10))
cv2.waitKey()
return
setup.py is as follows:
from setuptools import setup
setup(name='gnn_pylib',
version='0.1',
description='General purpose python library',
url='http://github.com/whatever/gnn_pylib',
author='whatever',
author_email='whatever#gmail.com',
license='MIT',
packages=['gnn_pylib'],
install_requires=[
'numpy',
'cv2',
],
zip_safe=False)
Rather than using cv2 as the required package name instead use opencv-python since that's the name of the OpenCV bindings package available from PyPI. So your setup.py file will instead look like this (same as above with different entry for the OpenCV bindings package requirement):
from setuptools import setup
setup(name='gnn_pylib',
version='0.1',
description='General purpose python library',
url='http://github.com/whatever/gnn_pylib',
author='whatever',
author_email='whatever#gmail.com',
license='MIT',
packages=['gnn_pylib'],
install_requires=[
'numpy',
'opencv-python',
],
zip_safe=False)
#James Adams answer the specific case for cv2, replacing with more compatible opencv-python.
However, if you still want to have dependencies install from conda, consider make a conda package.
See similar question with answer:
setup.py with dependecies installed by conda (not pip)
Use 'conda install' instead of 'pip install' for setup.py packages
I cannot find answer with detail step-by-step and example yet. But hope it help.
Related
I have been locally editing (inside a conda env) the package GSTools cloned from the github repo https://github.com/GeoStat-Framework/GSTools, to adapt it to my own purposes. The package is c++ wrapped in python (cython).
I've thus far used pip install -e . in the main package dir for my local changes. But I want to now use their OpenMP support by setting the env variable export GSTOOLS_BUILD_PARALLEL=1 . Then doing pip install -e . I get among other things in the terminal ...
Installing collected packages: gstools
Running setup.py develop for gstools
Successfully installed gstools-1.3.6.dev37
The issue is nothing actually changed because, setup.py (shown below) is supposed to print "OpenMP=True" if the env variable is set to GSTOOLS_BUILD_PARALLEL=1 in the linux terminal , and print something else if its not set to 1.
here is setup.py.
# -*- coding: utf-8 -*-
"""GSTools: A geostatistical toolbox."""
import os
import numpy as np
from Cython.Build import cythonize
from extension_helpers import add_openmp_flags_if_available
from setuptools import Extension, setup
# cython extensions
CY_MODULES = [
Extension(
name=f"gstools.{ext}",
sources=[os.path.join("src", "gstools", *ext.split(".")) + ".pyx"],
include_dirs=[np.get_include()],
define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
)
for ext in ["field.summator", "variogram.estimator", "krige.krigesum"]
]
# you can set GSTOOLS_BUILD_PARALLEL=0 or GSTOOLS_BUILD_PARALLEL=1
if int(os.getenv("GSTOOLS_BUILD_PARALLEL", "0")):
added = [add_openmp_flags_if_available(mod) for mod in CY_MODULES]
print(f"## GSTools setup: OpenMP used: {any(added)}")
else:
print("## GSTools setup: OpenMP not wanted by the user.")
# setup - do not include package data to ignore .pyx files in wheels
setup(ext_modules=cythonize(CY_MODULES), include_package_data=False)
I tried instead just python setup.py install but that gives
UNKNOWN 0.0.0 is already the active version in easy-install.pth
Installed /global/u1/b/benabou/.conda/envs/healpy_conda_gstools_dev/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8-linux-x86_64.egg
Processing dependencies for UNKNOWN==0.0.0
Finished processing dependencies for UNKNOWN==0.0.0
and import gstools
no longer works correctly.
So how can I install my edited version of the package with OpenMP support?
developer of GSTools here.
I guess you don't see the printed message, because pip is suppressing output for the setup. So you could try making pip verbose with:
GSTOOLS_BUILD_PARALLEL=1 pip install -v -e .
BTW, we are always interested in enhancements. So maybe you are willing the share your edits on GSTools? :-)
Cheers,
Sebastian
I have a project that makes use of certain C package. Building the package is done as below in a setup.py file:
from setuptools import setup
import torch
if torch.cuda.is_available():
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
setup(
name='iou3d',
ext_modules=[
CUDAExtension('iou3d_cuda', [
'src/iou3d.cpp',
'src/iou3d_kernel.cu',
],
extra_compile_args={'cxx': ['-g'],
'nvcc': ['-O2']})
],
cmdclass={'build_ext': BuildExtension})
Now this is being setup correctly if I already have pytorch installed in my environment.
But if I am doing a fresh install on a clean environment using requirements.txt, and I want to install everything in one shot by pip install -r requirements.txt I am not sure how can I get to install pytorch first to be able to import it in the setup.py.
Appreciate your help.
This is an extension of SO setup.py ignores full path dependencies, instead looks for "best match" in pypi
I am trying to write setup.py to install a proprietary package from a .tar.gz file on an internal web site. Unfortunately for me the prop package name duplicates a public package in the public PyPI, so I need to force install of the proprietary package at a specific version. I'm building a docker image from a Debian-Buster base image, so pip, setuptools and tox are all freshly installed, the image brings python 3.8 and pip upgrades itself to version 21.2.4.
Solution 1 - dependency_links
I followed the instructions at the post linked above to put the prop package in install_requires and dependency_links. Here are the relevant lines from my setup.py:
install_requires=["requests", "proppkg==70.1.0"],
dependency_links=["https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"]
Installation is successful in Debian-Buster if I run python3 setup.py install in my package directory. I see the proprietary package get downloaded and installed.
Installation fails if I run pip3 install . also tox (version 3.24.4) fails similarly. In both cases, pip shows a message "Looking in indexes" then fails with "ERROR: Could not find a version that satisfies the requirement".
Solution 2 - PEP 508
Studying SO answer pip ignores dependency_links in setup.py which states that dependency_links is deprecated, I started over, revised setup.py to have:
install_requires=[
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
Installation is successful in Debian-Buster if I run pip3 install . in my package directory. Pip shows a message "Looking in indexes" but still downloads and installs the proprietary package successfully.
Installation fails in Debian-Buster if I run python3 setup.py install in my package directory. I see these messages:
Searching for proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0
..
Reading https://pypi.org/simple/proppkg/
..
error: Could not find suitable distribution for Requirement.parse(...).
Tox also fails in this scenario as it installs dependencies.
Really speculating now, it almost seems like there's an ordering issue. Tox invokes pip like this:
python -m pip install --exists-action w .tox/.tmp/package/1/te-0.3.5.zip
In that output I see "Collecting proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0" as the first step. That install fails because it fails to import package requests. Then tox continues collecting other dependencies. Finally tox reports as its last step "Collecting requests" (and that succeeds). Do I have to worry about ordering of install steps?
I'm starting to think that maybe the proprietary package is broken. I verified that the prop package setup.py has requests in its install_requires entry. Not sure what else to check.
Workaround solution
My workaround is installing the proprietary package in the docker image as a separate step before I install my own package, just by running pip3 install https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz. The setup.py has the PEP508 URL in install_requires. Then pip and tox find the prop package in the pip cache, and work fine.
Please suggest what to try for the latest pip and tox, or if this is as good as it gets, thanks in advance.
Update - add setup.py
Here's a (slightly sanitized) version of my package's setup.py
from setuptools import setup, find_packages
def get_version():
"""
read version string
"""
version_globals = {}
with open("te/version.py") as fp:
exec(fp.read(), version_globals)
return version_globals['__version__']
setup(
name="te",
version=get_version(),
packages=find_packages(exclude=["tests.*", "tests"]),
author="My Name",
author_email="email#mycompany.com",
description="My Back-End Server",
entry_points={"console_scripts": [
"te-be=te.server:main"
]},
python_requires=">=3.7",
install_requires=["connexion[swagger-ui]",
"Flask",
"gevent",
"redis",
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
package_data={"te": ["openapi_te.yml"]},
include_package_data=True, # read MANIFEST.in
)
I am unable to install the local packages using setup.py
Here is the project structure:
my-project/
lib/
local1/
local1.1.0.whl
index.html
local2/
local2.1.0.whl
index.html
setup.py
setup.py
import os
from setuptools import setup
setup(name='my project',
version='1.0',
description='my project',
install_requires=[
'lxml >= 4.3.0',
'local1 # file://localhost/{}/lib/local1/local1.1.0.whl'.format(os.getcwd()),
'local2 # file://localhost/{}/lib/local2/local2.2.0.whl'.format(os.getcwd()),
]
)
I can install if I put the dependencies in a requirements.txt file and use pip install -r requirements.txt --extra-index-url lib/, but I want to know why is it not possible to do python setup.py install or if I am missing something.
This is the error that I get -
No local packages or working download links found for local2# file://localhost//Users/anusha/Documents/my-project/lib/local2/local2.1.0.whl
error: Could not find suitable distribution for Requirement.parse('local2# file://localhost//Users/anusha/Documents/my-project/lib/local2/local2.1.0.whl')
On searching, I found this issue on github, but does not give me any pointers or solution as to how it worked.
Any help is welcome, thanks in advance!
Note this comment from pganssle in the discussion "Setuptools install fails with PEP508 URLs" in setuptools's issue tracker:
Our policy to date has been that if using pip install fixes your problem, you should use pip install and we won't fix the issue.
I believe this is in line with the current evolution of the packaging tools and techniques in the Python community. So if your setuptools-based project with this requirement notation can be installed via pip install . and pip install --editable ., then look no further.
A more complete (exhaustive) article on the topic:
Paul Ganssle's "Why you shouldn't invoke setup.py directly"
I'm building a new PyPI package based on an existing open source project using setuptools and add some code modifications (they are not the same).
Example:
opensource-custom=2.13.1
Since this project requires dependencies that will look for opensource
what options can I pass to my setup.py when building my wheel files so when I do pip freeze/pip list I can see both?
opensource-custom=2.13.1
opensource=2.13.0
An example of this scenario is intel-numpy if you do a pip install of it, it will generate a copy of numpy.
>pip install intel-numpy
>pip freeze
icc-rt==2019.0
intel-numpy==1.15.1
intel-openmp==2019.0
mkl==2019.0
mkl-fft==1.0.6
mkl-random==1.0.1.1
numpy==1.15.1
tbb==2019.0
tbb4py==2019.0
It sounds like you want to make opensource a dependency of opensource-custom. To do this, you can specify the install_requires parameter in setup.py:
from setuptools import setup
setup(
name='opensource-custom',
install_requires=[
'opensource',
],
...
)
See https://packaging.python.org/guides/distributing-packages-using-setuptools/#install-requires