We Keep Coding

(Python3) How handle C/C++ dependencies with Pip/Pypi?

I have C/C++ dependencies (TA-Lib) for the Python package that I developed. Not being user-friendly, I don't want the user to install or compile C to use my package. Simply, I just want the user to be able to install my package with pip install <package>.
So, I installed my C dependency in the setup.py before the setup() function by substituing the setup's commands (install, develop, build, etc.).
Here is the code:
import platform
from setuptools import setup
from setuptools.command.install import install
from setuptools.command.develop import develop
from distutils.command.build import build
from distutils.command.build_ext import build_ext
from distutils.command.sdist import sdist
from wheel.bdist_wheel import bdist_wheel
import os
WINDOW_OS = "Windows"
LINUX_OS = "Linux"
CURRENT_OS = platform.system()
def talib_install():
dir = "build_helper"
if CURRENT_OS == LINUX_OS:
os.system(f"cd {dir} && sh talib-install.sh")
elif CURRENT_OS == WINDOW_OS:
os.system(f"pip install {dir} TA_Lib-0.4.24-cp39-cp39-win_amd64.whl")
else:
raise Exception("Unknown OS")
class TalibBuild(build):
def get_sub_commands(self):
# Force "build_ext" invocation.
commands = build.get_sub_commands(self)
for c in commands:
if c == 'build_ext':
return commands
return ['build_ext'] + commands
class TalibBuildExt(build_ext):
def run(self):
talib_install()
build_ext.run(self)
class TalibSdit(sdist):
def run(self) -> None:
talib_install()
return super().run()
class TalibDevelop(develop):
def run(self) -> None:
talib_install()
return super().run()
def finalize_options(self) -> None:
return super().finalize_options()
class TalibInstall(install):
def run(self):
talib_install()
install.run(self)
def finalize_options(self) -> None:
return super().finalize_options()
class TalibBdistWheel(bdist_wheel):
def run(self):
talib_install()
# Run wheel build command
super().run()
def finalize_options(self):
super().finalize_options()
cmdclass = {
"install": TalibInstall,
"develop": TalibDevelop,
"sdist": TalibSdit,
"bdist_wheel": TalibBdistWheel,
"build": TalibBuild,
"build_ext": TalibBuildExt
}
install_requirements = [
"setuptools==59.5.0",
"tensorboard",
"torch-tb-profiler",
"geneticalgorithm2"
]
talib_install()
setup(
data_files=["build_helper/talib-install.sh"],
install_requires=install_requirements,
cmdclass=cmdclass
)
In local, the following commands are working well: pip install -e . and pip setup.py install.
The problem occurs when I upload my package on Pypi and try installing it on Colab using pip install mypackage. I have the feeling that the installation does not pass through the C code installation step. (I make sure that I have talib-install.sh script in the data_files.)
Moreover, in my release I have two files: wheel archive and source archive. When I execute pip install <link_to_the_source_achive> on Colab, the C code installation works. But pip install <link_to_the_wheel> doesn't work.
I thought that, by default, pip uses the wheel archive to install. I deleted it so that not having a choice, pip chooses the source. But it doesn't work either...
What is the difference between pip install <package> and pip install <link_to_source>? How pip is working? Any thoughts about how I can resolve my issue?
(I am working on Linux but I want it to work on Windows and MacOs too.)
Thank you very much.

Install by default, "optional" dependencies in Python (setuptools)

Is there a way to specify optional dependencies for a Python package that should be installed by default from pip but for which an install should not be considered a failure if they cannot be installed?
I know that I can specify install_requires so that the packages will be installed for the 90% of users using OSes that can easily install certain optional dependencies, and I also know I can specify extra_require to specify that users can declare they want a full install to get these features, but I haven't found a way to make a default pip install try to install the packages but not complain if they cannot be installed.
(The particular package I'd like to update the setuptools and setup.py for is called music21 for which 95% of the tools can be run without matplotlib, IPython, scipy, pygame, some obscure audio tools etc. but the package gains extra abilities and speed if these packages are installed, and I'd prefer to let people have these abilities by default but not report errors if they cannot be installed)

Not a perfect solution by any means, but you could setup a post-install script to try to install the packages, something like this:
from distutils.core import setup
from distutils import debug
from setuptools.command.install import install
class PostInstallExtrasInstaller(install):
extras_install_by_default = ['matplotlib', 'nothing']
#classmethod
def pip_main(cls, *args, **kwargs):
def pip_main(*args, **kwargs):
raise Exception('No pip module found')
try:
from pip import main as pip_main
except ImportError:
from pip._internal import main as pip_main
ret = pip_main(*args, **kwargs)
if ret:
raise Exception(f'Exitcode {ret}')
return ret
def run(self):
for extra in self.extras_install_by_default:
try:
self.pip_main(['install', extra])
except Exception as E:
print(f'Optional package {extra} not installed: {E}')
else:
print(f"Optional package {extra} installed")
return install.run(self)
setup(
name='python-package-ignore-extra-dep-failures',
version='0.1dev',
packages=['somewhat',],
license='Creative Commons Attribution-Noncommercial-Share Alike license',
install_requires=['requests',],
extras_require={
'extras': PostInstallExtrasInstaller.extras_install_by_default,
},
cmdclass={
'install': PostInstallExtrasInstaller,
},
)

The simplest way to do this is by adding a custom install command that simply shells out to pip to install the "optional" packages. In your setup.py:
import sys
import subprocess
from setuptools import setup
from setuptools.command.install import install
class MyInstall(install):
def run(self):
subprocess.call([sys.executable, "-m", "pip", "install", "whatever"])
install.run(self)
setup(
...
cmdclass={
'install': MyInstall,
},
)
Like hoefling said above, this will only work if you publish a source distribution (.tar.gz or .zip). It will not work if you publish your package as a built distribution (.whl).

This might be what you are looking for. It's appears to be a built in feature of setup tools that allows you to declare "optional dependencies".
https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies
Ex
setup(
name="Project-A",
...
extras_require={
'PDF': ["ReportLab>=1.2", "RXP"],
'reST': ["docutils>=0.3"],
}
)

Compile C library on pip install

I made a Python application which needs to execute C functions. To do so, I compiled the C functions in a shared library using gcc and called the library in my Python script using ctypes.
I'm trying to pack my application in a pip package but found no way to create the shared library upon pip install.
I tried the following (setup.py) :
from setuptools import setup
from setuptools.command.install import install
import subprocess
class compileLibrary(install):
def run(self):
install.run(self)
command = "cd packageName"
command += " && git clone https://mygit.com/myAwesomeCLibrary.git"
command += " && gcc -my -many -options"
process = subprocess.Popen(command, shell=True)
process.wait()
setup(
name='packageName',
version='0.1',
packages=['packageName'],
install_requires=[
...
],
cmdclass={'install': compileLibrary},
)
This works when doing python setup.py install but, when installing from pip, while the install process is successful, the shared library is not in the package folder (pythonx.x/site-packages/packageName).
I tried using distutils with the same result.
Based on the question Run Makefile on pip install I also tried the following setup.py :
from setuptools import setup
from distutils.command.build import build
import subprocess
class Build(build):
def run(self):
command = "cd packageName"
command += " && git clone https://mygit.com/myAwesomeCLibrary.git"
command += " && gcc -my -many -options"
process = subprocess.Popen(command, shell=True)
process.wait()
build.run(self)
setup(
name='packageName',
version='0.1',
packages=['packageName'],
install_requires=[
...
],
cmdclass={'build': Build},
)
Here is the architecture of my package
packageName/
├── packageName/
│ ├── __init__.py
│ ├── myFunctions.c
├── MANIFEST.in
├── README.md
├── setup.py
Note : After installing the package with pip, I only have __init__.py and __pycache__ in the install folder (pythonx.x/site-packages/packageName).
How can I create the shared library during pip install so It can be used by my package ?

After hours of research, I found the solution. The main problems were :
Use Extension from setuptools to compile the library
Use a custom install function to git clone and calling the constructor at the end so the git clone occurs before the compilation.
Here is the working setup.py :
from setuptools import setup, Extension
from setuptools.command.install import install
import subprocess
import os
class CustomInstall(install):
def run(self):
command = "git clone https://mygit.com/myAwesomeCLibrary.git"
process = subprocess.Popen(command, shell=True, cwd="packageName")
process.wait()
install.run(self)
module = Extension('packageName.library',
sources = ['packageName/library.c'],
include_dirs = ['packageName/include'],
extra_compile_args=['-fPIC'])
setup(
name='packageName',
version='0.1',
packages=['packageName'],
install_requires=[
...
],
cmdclass={'install': CustomInstall},
include_package_data=True,
ext_modules=[module],
)

you can add the shared library in MANIFEST.in.
like this:
include *.so *.dylib

Post-install setup.py doesn't work

I'm trying to add a post-install step into the setup.py. The installation works but the code inside run(self) is never executed.
Things I've tried (with no result):
Install it using both "pip install (-e) ." and "python setup.py (develop)" [and later uninstall it, reinstall it, delete .egg-info folder,...]
Variations using: do_egg_install, build, bdist_egg,...
Versions:
pip 8.1.2 (for Python 3.5)
setuptools-20.2.2
Toy example:
from setuptools import setup
from setuptools.command.install import install
class MyCommand(install):
def run(self):
print("Hello, developer, how are you? :)")
install.run(self)
if __name__ == '__main__':
setup(
cmdclass={
'install': MyCommand,
}
)

setup.py install_require with options

I need to add rjsmin to my dependencies via install_require in a setup.py.
rjsmin offers a way to disable the c-extension by using the --without-c-extensions switch like following
python setup.py install --without-c-extensions
I wonder, how to add this switch to the install_require string.

I solved my problem of installing dependencies with global-options by sub-classing setuptools.command.install class and overriding its run() method, like following code -
from setuptools import setup
from setuptools.command.install import install
from subprocess import call
class CustomInstall(install):
def run(self):
install.run(self)
call(['pip', 'install', 'pycurl', '--global-option=--with-nss'])
setup( ...
cmdclass={
'install': CustomInstall,
},
)
Here, I am installing pycurl with global option --with-nss

You need to provide an --install-option or --global-option along with the requirement text.
You can refer the doc here