I made a Python application which needs to execute C functions. To do so, I compiled the C functions in a shared library using gcc and called the library in my Python script using ctypes.
I'm trying to pack my application in a pip package but found no way to create the shared library upon pip install.
I tried the following (setup.py) :
from setuptools import setup
from setuptools.command.install import install
import subprocess
class compileLibrary(install):
def run(self):
install.run(self)
command = "cd packageName"
command += " && git clone https://mygit.com/myAwesomeCLibrary.git"
command += " && gcc -my -many -options"
process = subprocess.Popen(command, shell=True)
process.wait()
setup(
name='packageName',
version='0.1',
packages=['packageName'],
install_requires=[
...
],
cmdclass={'install': compileLibrary},
)
This works when doing python setup.py install but, when installing from pip, while the install process is successful, the shared library is not in the package folder (pythonx.x/site-packages/packageName).
I tried using distutils with the same result.
Based on the question Run Makefile on pip install I also tried the following setup.py :
from setuptools import setup
from distutils.command.build import build
import subprocess
class Build(build):
def run(self):
command = "cd packageName"
command += " && git clone https://mygit.com/myAwesomeCLibrary.git"
command += " && gcc -my -many -options"
process = subprocess.Popen(command, shell=True)
process.wait()
build.run(self)
setup(
name='packageName',
version='0.1',
packages=['packageName'],
install_requires=[
...
],
cmdclass={'build': Build},
)
Here is the architecture of my package
packageName/
├── packageName/
│ ├── __init__.py
│ ├── myFunctions.c
├── MANIFEST.in
├── README.md
├── setup.py
Note : After installing the package with pip, I only have __init__.py and __pycache__ in the install folder (pythonx.x/site-packages/packageName).
How can I create the shared library during pip install so It can be used by my package ?
After hours of research, I found the solution. The main problems were :
Use Extension from setuptools to compile the library
Use a custom install function to git clone and calling the constructor at the end so the git clone occurs before the compilation.
Here is the working setup.py :
from setuptools import setup, Extension
from setuptools.command.install import install
import subprocess
import os
class CustomInstall(install):
def run(self):
command = "git clone https://mygit.com/myAwesomeCLibrary.git"
process = subprocess.Popen(command, shell=True, cwd="packageName")
process.wait()
install.run(self)
module = Extension('packageName.library',
sources = ['packageName/library.c'],
include_dirs = ['packageName/include'],
extra_compile_args=['-fPIC'])
setup(
name='packageName',
version='0.1',
packages=['packageName'],
install_requires=[
...
],
cmdclass={'install': CustomInstall},
include_package_data=True,
ext_modules=[module],
)
you can add the shared library in MANIFEST.in.
like this:
include *.so *.dylib
Related
With the move to the new pyproject.toml system, I was wondering whether there was a way to install packages in editable mode while compiling extensions (which pip install -e . does not do).
So I want pip to:
run the build_ext I configured for Cython and generate my .so files
put them in the local folder
do the rest of the normal editable install
I found some mentions of build_wheel_for_editable on the pip documentation but I could not find any actual example of where this hook should be implemented and what it should look like. (to be honest, I'm not even completely sure this is what I'm looking for)
So would anyone know how to do that?
I'd also happy about any additional explanation as to why pip install . runs build_ext but the editable command does not.
Details:
I don't have a setup.py file anymore; the pyproject.toml uses setuptools and contains
[build-system]
requires = ["setuptools>=61.0", "numpy>=1.17", "cython>=0.18"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
package-dir = {"" = "."}
[tool.setuptools.packages]
find = {}
[tool.setuptools.cmdclass]
build_ext = "_custom_build.build_ext"
The custom build_ext looks like
from setuptools import Extension
from setuptools.command.build_ext import build_ext as _build_ext
from Cython.Build import cythonize
class build_ext(_build_ext):
def initialize_options(self):
super().initialize_options()
if self.distribution.ext_modules is None:
self.distribution.ext_modules = []
extensions = Extension(...)
self.distribution.ext_modules.extend(cythonize(extensions))
def build_extensions(self):
...
super().build_extensions()
It builds a .pyx into .cpp, then adds it with another cpp into a .so.
I created a module that looks like this:
$ tree .
.
├── pyproject.toml
├── setup.py
└── test
└── helloworld.pyx
1 directory, 3 files
My pyproject.toml looks like:
[build-system]
requires = ["setuptools>=61.0", "numpy>=1.17", "cython>=0.18"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
py-modules = ["test"]
[project]
name = "test"
version = "0.0.1"%
My setup.py:
from setuptools import setup
from Cython.Build import cythonize
setup(ext_modules=cythonize("test/helloworld.pyx"))
And helloworld.pyx just contains print("Hello world").
When I do pip install -e ., it builds the cython file as expected.
If you really don't want to have a setup.py at all, I think you'll need to override build_py instead of build_ext, but IMO just having the simple setup.py file isn't a big deal.
Environment
PC1: Dev-machine, online, Python install dir: C:\Python310
PC2: Target machine, offline, Python install dir: C:\Program Files\Python310
Doing
Write source and run command in workdir pip install -t ./out ./ on PC1.
Copy files under out dir from PC1 to PC2.
Open term and invoke exe file on PC2.
Then I got message Fatal error in launcher: Unable to create process using '"C:\Python310\python.exe" "C:\Program Files\Python310\Scripts\my_app.exe" ': ??????????????????.
How can I build for PC2?
Folder structure
┗━ my_app
┣━ setup.py
┗━ my_app
┣━ __init__.py
┣━ __main__.py
┗━ main.py
File contents:
setup.py
from setuptools import setup, find_packages
setup(
name = 'my_app',
packages = find_packages(),
entry_points = {
'console_scripts': [
'my_app = my_app.main:main',
],
},
)
my_app/__main__.py
from .main import main
main()
my_app/main.py
def main():
print('hello world')
Constraints
without cx_freeze, pyinstaller, py2exe or similer third party packages
actual my_app requires external packages(ex: tqdm)
Run command python setup.py bdist and copy dist/my_app...zip content to target machine, its resolved my question.
https://docs.python.org/3/distutils/builtdist.html
Problem Statement: when I install my pip package, a specific file inside the package get coped to Temp directory
Approach:
My Package directory Sturcture is following:
my-app/
├─ app/
│ ├─ __init__.py
│ ├─ __main__.py
├─ folder-with-extra-stuff/
│ ├─ __init__.py
│ ├─ file_I_want_to_cppy.tar.gz
├─ setup.py
├─ MANIFEST.in
I'm tweaking my setup.py file to do the job. Following is my setup.py
#!/usr/bin/env python
from setuptools import setup, find_packages
from setuptools.command.install import install
import os
import sys
import shutil
rootDir = os.path.abspath(os.path.dirname(__file__))
def run_custom_install():
print("--------Start running custom command -------")
temp_dir = r'c:\temp' if sys.platform == "win32" else r'/tmp'
temp_col_dir = temp_dir + os.sep + 'dump'
os.makedirs(temp_dir, exist_ok=True)
os.makedirs(temp_col_dir, exist_ok=True)
print("----------locate the zip file ---------------")
ColDirTests = os.path.abspath(os.path.join(rootDir, 'my-app','folder-with-extra-stuff'))
_src_file = os.path.join(ColDirTests , 'file_I_want_to_cppy.tar.gz ')
print(f"******{_src_file}**********")
if os.path.exists(_src_file):
print(f"-----zip file has been located at {_src_file}")
shutil.copy(_src_file, temp_col_dir)
else:
print("!!!!Couldn't locate the zip file for transfer!!!!")
class CustomInstall(install):
def run(self):
print("***********Custom run from install********")
install.run(self)
run_custom_install()
ver = "0.0.0"
setup(
name='my_pkg',
version=ver,
packages=find_packages(),
python_requires='>=3.6.0',
install_requires = getRequirements(),
include_package_data= True,
cmdclass={
'install' : CustomInstall,
}
)
MANIFEST.in
include README.md
include file_I_want_to_cppy.tar.gz
recursive-include my-app *
global-exclude *.pyc
include requirements.txt
prune test
Testing build:
> python setup.py bdist_wheel
It is working during build. I can see there is a directory formed C:\temp\dump and file_I_want_to_cppy.tar.gz inside it. But when I release the package in pip and try to install it from pip, the folder remains Empty!
Any idea what I might be doing wrong here?
After a lot of research I have figure out how to resolve this issue. Let me summarize my findings, it might be helpful for other who wants to do post_pip_install processing.
setup.py
Different options to install package: 1) pip install pkg_name, 2) python -m setup.py sdist
If you want to make them work in either ways, need to have install, egg_info and develop all 3 options repeated as shown in setup.py
If you create *.whl file by python -m setup.py bdist_wheel , post pip install processing won't be executed! Please upload .tar.gz format generated usingsdist to PyPi/Artifacts to make post pip install processing work. Again, Please note: It will not work when installing from a binary wheel
upload the pip package: twine upload dist/*.tar.gz
from setuptools import setup, find_packages
from setuptools.command.install import install
from setuptools.command.egg_info import egg_info
from setuptools.command.develop import develop
rootDir = os.path.abspath(os.path.dirname(__file__))
def run_post_processing():
print("--------Start running custom command -------")
# One can Run any Post Processing here that will be executed post pip install
class PostInstallCommand(install):
def run(self):
print("***********Custom run from install********")
install.run(self)
run_post_processing()
class PostEggCommand(egg_info):
def run(self):
print("***********Custom run from Egg********")
egg_info.run(self)
run_post_processing()
class PostDevelopCommand(develop):
def run(self):
print("***********Custom run from Develop********")
develop.run(self)
run_post_processing()
ver = "0.0.0"
setup(
name='my_pkg',
version=ver,
packages=find_packages(),
python_requires='>=3.6.0',
install_requires = getRequirements(),
include_package_data= True,
cmdclass={
'install' : PostInstallCommand,
'egg_info': PostEggCommand,
'develop': PostDevelopCommand
}
)
Few More Things from my research:
If you want to do pre-processing instead of post-processing, need to move install.run(self) at the end
while pip installing, if you want to see custom messages of pre/post instllation, use -vvv. Example: pip install -vvv my_pkg
is it possible to detect if current python code is running from package?
if yes - is it possible to get package metadata (name, version, description)?
package is created with this kind of setup.py
import os
from setuptools import setup, find_packages
setup(
name='my-pack-name',
description='my description ' + os.getenv('GIT_COMMIT', '*')[:7],
version=os.getenv('BUILD_VERSION', '0.0.0dev'),
packages=find_packages(),
)
build: python3 setup.py bdist_wheel -d ./artifact
install on target: pip3 install "my-pack-name-x.x.x.whl" --upgrade
now from my_pack_name/app.py that was inside my-pack-name-x.x.x.whl i want to detect that i'm running from installed package
and if so then get package metadata defined during setup.py execution
For Python >=3.8
https://docs.python.org/es/3.10/library/importlib.metadata.html
You can get the metadata for a package by:
from importlib.metadata import metadata
md = metadata("your package name")
author = md["Author"]
# etc ...
For Python <3.8
This is just an idea (not tested).
What about having the package metadata in a different module, and try relative import it in the app.py module?
# metadata.py
name='my-pack-name',
description='my description ' + os.getenv('GIT_COMMIT', '*')[:7],
version=os.getenv('BUILD_VERSION', '0.0.0dev')
In setup.py you could reuse that:
# setup.py
import os
from setuptools import setup, find_packages
import .metadata as md
setup(
name=md.name,
description=md.description + os.getenv('GIT_COMMIT', '*')[:7],
version=md.version,
packages=find_packages(),
)
And the in app.py
def get_metadata():
try:
import .metadata as md
except ImportError:
return None
else:
# return metadata here
That way if get_metadata returns None, you were not able to import the module, so you're not executing the app.py in your package, otherwise, you are in your package and as a bonus you got your metadata.
For the test if python code is running from an installed package (vs. from a development location), I use
if 'site-packages' in __file__:
...
I don't know if that's a good approach; seems to work thus far.
import os
from setuptools import setup
from distutils.command.install import install as _install
def _post_install(dir):
from subprocess import call
call([sys.executable, 'post_script.py'],
cwd=os.path.join(dir, 'script_folder'))
class install(_install):
def run(self):
_install.run(self)
self.execute(_post_install, (self.install_lib,),
msg="Running post install task")
VERSION = '123'
setup(name='XXXX',
description='hello',
url='http://giturl.com',
packages=['package_folder'],
cmdclass={'install': install},
package_data={
'package_folder': [
'*.py',
'se/*pp'
],
},
)
#
Basically the postscript should execute once I install the rpm that is being built.
Its not working.
Any other method as this is not working?
You can run python setup.py bdist_rpm --post-install=<script name>
This will create an rpm which will run the contents of the script you provide after the normal rpm installation is completed.
If you want to do it in your setup.py you can pass along
setup(
...
options={'bdist_rpm': {'post_install': '<post_install script name>'}},
...
)
This will only affect bdist_rpm, and thus only the rpm you create with python setup.py bdist_rpm