We Keep Coding

Creating a python package (deb/rpm) from cmake

I am trying to create a python package (deb & rpm) from cmake, ideally using cpack. I did read
https://cmake.org/cmake/help/latest/cpack_gen/rpm.html and,
https://cmake.org/cmake/help/latest/cpack_gen/deb.html
The installation works just fine (using component install) for my shared library. However I cannot make sense of the documentation to install the python binding (glue) code. Using the standard cmake install mechanism, I tried:
install(
FILES __init__.py library.py
DESTINATION ${ACME_PYTHON_PACKAGE_DIR}/project_name
COMPONENT python)
And then using brute-force approach ended-up with:
# debian based package (relative path)
set(ACME_PYTHON_PACKAGE_DIR lib/python3/dist-packages)
and
# rpm based package (full path required)
set(ACME_PYTHON_PACKAGE_DIR /var/lang/lib/python3.8/site-packages)
The above is derived from:
debian % python -c 'import site; print(site.getsitepackages())'
['/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.9/dist-packages']
while:
rpm % python -c 'import site; print(site.getsitepackages())'
['/var/lang/lib/python3.8/site-packages']
It is pretty clear that the brute-force approach will not be portable, and is doomed to fail on the next release of python. The only possible solution that I can think of is generating a temporary setup.py python script (using setuptools), that will do the install. Typically cmake would call the following process:
% python setup.py install --root ${ACME_PYTHON_INSTALL_ROOT}
My questions are:
Did I understand the cmake/cpack documentation correctly for python package ? If so this means I need to generate an intermediate setup.py script.
I have been searching through the cmake/cpack codebase (git grep setuptools) but did not find helper functions to handle generation of setup.py and passing the result files back to cpack. Is there an existing cmake module which I could re-use ?
I did read, some alternative solution, such as:
How to build debian package with CPack to execute setup.py?
Which seems overly complex, and geared toward Debian-only based system. I need to handle RPM in my case.

As mentionned in my other solution, the ugly part is dealing with absolute path in cmake install() commands. I was able to refactor the code to avoid usage of absolute path in install(). I simply changed the installation into:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
# "." syntax is a reliable mechanism, see:
# https://gitlab.kitware.com/cmake/cmake/-/issues/22616
DESTINATION "."
COMPONENT python)
And then one simply needs to:
set(CMAKE_INSTALL_PREFIX "/")
set(CPACK_PACKAGING_INSTALL_PREFIX "/")
include(CPack)
At this point all install path now need to include explicitely /usr since we've cleared the value for CMAKE_INSTALL_PREFIX.
The above has been tested for deb and rpm packages. CPACK_BINARY_TGZ does properly run with the above solution:
https://gitlab.kitware.com/cmake/cmake/-/issues/22925

I am going to post the temporary solution I am using at the moment, until someone provide something more robust.
So I eventually manage to stumble upon:
https://alioth-lists.debian.net/pipermail/libkdtree-devel/2012-October/000366.html and,
Using CMake with setup.py
Re-using the above to do an install step instead of a build step can be done as follow:
find_package(Python COMPONENTS Interpreter)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(SETUP_DEPS "${CMAKE_CURRENT_SOURCE_DIR}/project_name/__init__.py")
set(SETUP_OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build-python")
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
COMMAND ${Python_EXECUTABLE} ARGS ${SETUP_PY} install --root ${SETUP_OUTPUT}
COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
DEPENDS ${SETUP_DEPS})
add_custom_target(target ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp)
And then the ugly part is:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
DESTINATION "/" # FIXME may cause issues with other cpack generators
COMPONENT python)
Turns out that the documentation for install() is pretty clear about absolute paths:
https://cmake.org/cmake/help/latest/command/install.html#introduction
DESTINATION
[...]
As absolute paths are not supported by cpack installer generators,
it is preferable to use relative paths throughout.
For reference, here is my setup.py.in:
from setuptools import setup
if __name__ == '__main__':
setup(name='project_name_python',
version='${PROJECT_VERSION}',
package_dir={'': '${CMAKE_CURRENT_SOURCE_DIR}'},
packages=['project_name'])
You can be fancy and remove the __pycache__ folder using the -B flag:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT}
You can be extra fancy and add debian option such as:
if(CPACK_BINARY_DEB)
set(EXTRA_ARG "--install-layout" "deb")
endif()
use as:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT} ${EXTRA_ARG}

numpy.f2py - extension doesn't exist when source distribution is installed with pip

I'm trying to wrap my head around f2py because my organization has a lot of legacy fortran code that I would like to incorporate into some newer python-based tools I'm writing. Ideally, I would package these tools either in source packages or wheels to make it easier to distribute to the rest of the organization.
I've written a small test package based on some other examples I've seen that just sums an array of floats. The package contents are included below. If I build a source distribution tarball using py setup.py sdist, everything looks like it works. It even looks like pip successfully installs it. However, if I open a python shell and try to import the newly installed module, I get an error on the from fastadd import fadd line in the initialization script saying
AttributeError: module 'fastadd' has no attribute 'fastadd'
So it seems like it didn't actually successfully build the f2py module. Doing some troubleshooting, if I open a powershell window in the package folder and just run
py -m numpy.f2py -c fadd.pyf fadd.f90
and then open a python shell in the same folder and try to import fastadd, I get an error, ImportError: DLL load failed: The specified module could not be found. (This is after I installed the Visual Studio build tools, a fix suggested on several threads). Following the advice on this thread, changing the command to
py -m numpy.f2py -c --fcompiler=gnu95 --compiler=mingw32 fadd.pyf fadd.f90
will build a module file that I can successfully import and use. Okay, great.
However, when I change config.add_extension in the setup file to include the keyword argument f2py_options=["--fcompiler=gnu95","--compiler=mingw32"] and try to build a package distribution file with setup.py sdist command and then install using py -m pip install fastadd-1.0a1.tar.gz, I get yet a different error that says
ERROR: No .egg-info directory found in C:\Users\username\AppData\Local\Temp\pip-pip-egg-info-c7406k03
And now I'm completely flummoxed. Other configurations of the f2py_options either result in setup.py throwing an error or fail to create the extension altogether, similar to above. Using a simple string for the options gives an error, so apparently f2py_options does in fact expect a list input. I can't seem to find any good documentation on whether I'm using f2py_options correctly, and I have no idea why just adding that option would cause pip to not know where its info directory is. That makes no sense to me. I'd really appreciate some help on this one.
I'm running Python 3.7.0 32-bit, numpy 1.20.1, and pip 21.0.1 on a Windows 10 machine.
--EDIT--
Looking in the installation directory of the test module, I found a new wrinkle to this problem: the installation directory does not actually include any files listed in MANIFEST, not even the __init__.py file. If I copy __init__.py into the directory, trying to import the module gives the same ImportError: DLL load failed error I've been getting.
Also, inspecting the output of py -m pip install, it looks like numpy.distutils doesn't recognize --fcompiler or --compiler as valid options and just ignores them, even though numpy.f2py does recognize them.
--END EDIT--
PACKAGE CONTENTS:
+-fastadd
---__init__.py
---fadd.f90
---fadd.pyf
-MANIFEST.in
-README
-setup.py
fadd.f90 has the following contents:
subroutine fadd(vals,n,mysum)
integer, intent(in) :: n
real*8, intent(out):: mysum
real*8, dimension(n), intent(in) :: vals
mysum = sum(vals)
end subroutine fadd
fadd.pyf has the following contents:
python module fastadd ! in
interface ! in :fastadd
subroutine fadd(vals,n,mysum) ! in :fastadd:fadd.f90
real*8 dimension(n),intent(in) :: vals
integer, optional,intent(in),check(len(vals)>=n),depend(vals) :: n=len(vals)
real*8 intent(out) :: mysum
end subroutine fadd
end interface
end python module fastadd
__init__.py:
"""This is the documentation!"""
from .fastadd import fadd
MANIFEST.in:
include README
recursive-include fastadd *.f90
recursive-include fastadd *.pyf
recursive-include fastadd *.py
and, finally, setup.py:
def configuration(pth=None):
from numpy.distutils.misc_util import Configuration
config = Configuration(
'fastadd',
top_path=pth,
version='1.0a1',
author='John Doe',
author_email='john.doe#fake-org.biz',
url='fake-org.biz/fastadd',
description="Testing f2py build process. Sums an arbitrary-length list of numbers.")
config.add_extension(
'fastadd',
sources=['fastadd\\fadd.pyf','fastadd\\fadd.f90']
)
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(**configuration('fastadd').todict())
If it helps at all, the final MANIFEST file looks like this after the setup script is run:
# file GENERATED by distutils, do NOT edit
README
setup.py
C:\Users\username\Documents\Development\python_modules\fastadd\fastadd\fadd.f90
C:\Users\username\Documents\Development\python_modules\fastadd\fastadd\fadd.pyf
fastadd\__init__.py
fastadd\fadd.f90
fastadd\fadd.pyf

Where are Python wheels created ?

I do this
C:\WINDOWS\system32>python c:\\python34\kissdownloader\setup.py bdist_wheel
and I get this after some other stuff
creating build\bdist.win-amd64\wheel\kissdownloader-1.dist-info\WHEEL
I goto my python root and there's nothing there , no build folder or anything . Is the thing somewhere else ? Oh and I don't have a manifest.in file if that makes any difference .
My setup.py file looks like this :
from setuptools import setup
setup (
name='kissdownloader',
version='1',
description = 'A kisscartoon/kissanime downloader' ,
package_dir={'kissdownloader':'C:\\Python34'},
author='Vriska',
author_email='xyz#gmail.com',
install_requires = ['bs4','cfscrape','requests'],
package_data={'data' : ['C:\Program Files\PhantomJS\phantomjs.exe']},
)

The output is generated in your current working directory. In your case, that's C:\WINDOWS\system32, because that's where you started Python. You'll find a build and dist directory; the latter contains the completed wheel.
If you want the directories to be created next to setup.py, change your directory to c:\python34\kissdownloader\ first. Many a Python project expects you to run setup.py there anyway.
As a side note: I wouldn't bundle the PhantomJS binary in with your project. Instead, require users to install it separately; they may already have it installed anyway, and you may run into legal issues by re-distributing it in your own project without at least a compatible license.

Debianzing a Python program to get a .deb [duplicate]

This question already has answers here:
Is there a standard way to create Debian packages for distributing Python programs?
(5 answers)
Closed 5 years ago.
Aim
To create an installable .deb file (or package). Which when clicked would install the software on a Linux machine and an icon would be put on the GNOME panel. So as to launch this application from there.
What I have referred to
I referred to two debianizing guides.
Guide 1
Guide 2
The first one had a video which was impossible to understand, partly because of the accent and partly because it was hopelessly outdated.(it was uploaded in 2007)
And the second one was completely text. I got till the 4th Step, Builds the package. But when I did it I got output that did not match what was given in the guide.
What I require
I have a simple python program. It takes your age and then prints back out if the age is below, equal to, or above 18 years. There is just one file and no other dependency for this program. And I want to build this into a .deb.
Specs
-Python 2.7
-Linux Mint
Edit
I followed the exact directory structure as you instructed as you. And replaced all myscript with cowsandbulls. The build completed and I got the Debian. When I installed it and then ran the command cowsandbulls from the terminal I got the following error:
Traceback (most recent call last):
File "/usr/bin/cowsandbulls", line 9, in <module>
load_entry_point('cowsandbulls==1.0', 'gui_scripts', 'cowsandbulls')()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 337, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2311, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2017, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
ImportError: No module named cowsandbulls

I just tested stdeb (see https://pypi.python.org/pypi/stdeb), a Python package for turning any other Python package into a Debian package.
First I installed stdeb:
apt-get install python-stdeb
Then I made a simple script called myscript.py with the following contents:
def main():
print "Hello world, says myscript!"
# wait for input from the user
raw_input()
if __name__ == '__main__':
main()
Importantly, your directory structure should be:
somewhere/myscript/
setup.py
myscript/
__init__.py
myscript.py
In the setup.py file, you do something like:
import os
from setuptools import setup
from nvpy import nvpy
setup(
name = "myscript",
version = "1.0",
author = "Charl P. Botha",
author_email = "cpbotha#vxlabs.com",
description = "Demo of packaging a Python script as DEB",
license = "BSD",
url = "https://github.com/cpbotha/nvpy",
packages=['myscript'],
entry_points = {
'console_scripts' : ['myscript = myscript.myscript:main']
},
data_files = [
('share/applications/', ['vxlabs-myscript.desktop'])
],
classifiers=[
"License :: OSI Approved :: BSD License",
],
)
The console_scripts directive is important, and it'll create an executable script called my_script, which will be available system-wide after you install the resultant DEB. If your script uses something like tkinter or wxpython and has a graphical user interface, you should use gui_scripts instead of console_scripts.
The data_files directive will install a suitable desktop file into /usr/share/applications, so that you can also start myscript from your desktop environment. vxlabs-myscript.desktop looks like this:
[Desktop Entry]
Version=1.0
Type=Application
Name=myscript
Comment=Minimal stdeb example
# myscript should wait for user input at the end, else the terminal
# window will disappear immediately.
Exec=myscript
Icon=/usr/share/icons/gnome/48x48/apps/file-manager.png
Categories=Utility;
# desktop should run this in a terminal application
Terminal=true
StartupNotify=true
StartupWMClass=myscript
To build the DEB, you do the following in the top-level myscript:
python setup.py --command-packages=stdeb.command bdist_deb
Which will create a .deb in the deb_dist directory.
After having installed the DEB I created like this, I could run myscript from the command-line, and I could also invoke it from my desktop environment.
Here's a GitHub repository with the example code above: https://github.com/cpbotha/stdeb-minimal-example

The right way of building a deb package is using dpkg-buildpackage, but sometimes it is a little bit complicated. Instead you can use dpkg -b <folder> and it will create your Debian package.
These are the basics for creating a Debian package with dpkg -b <folder> with any binary or with any kind of script that runs automatically without needing manual compilation (Python, Bash, Pearl, and Ruby):
Create the files and folders in order to recreate the following structure:
ProgramName-Version/
ProgramName-Version/DEBIAN
ProgramName-Version/DEBIAN/control
ProgramName-Version/usr/
ProgramName-Version/usr/bin/
ProgramName-Version/usr/bin/your_script
The scripts placed at /usr/bin/ are directly called from the terminal. Note that I didn't add an extension to the script.
Also, you can notice that the structure of the deb package will be the structure of the program once it's installed. So if you follow this logic if your program has a single file, you can directly place it under ProgramName-Version/usr/bin/your_script, but if you have multiple files, you should place them under ProgramName-Version/usr/share/ProgramName/all your files and place only one file under /usr/bin/ that will call your scripts from /usr/share/ProgramName/.
Change all the folder permission to root:
chown root:root -R /path/to/ProgramName-Version
Change the script's permissions:
chmod 0755 /path/to/the/script
Finally, you can run: dpkg -b /path/to/the/ProgramName-Version and your deb package will be created! (You can also add the post/pre inst scripts and everything you want; it works like a normal Debian package.)
Here is an example of the control file. You only need to copy-paste it in to an empty file called "control" and put it in the DEBIAN folder.
Package: ProgramName
Version: VERSION
Architecture: all
Maintainer: YOUR NAME <EMAIL>
Depends: python2.7, etc , etc,
Installed-Size: in_kb
Homepage: http://foo.com
Description: Here you can put a one line description. This is the short Description.
Here you put the long description, indented by 1 space.
If you want to build using dpkg -b <folder> you can use this program that will do everything with one command. If you regularly build packages, it is a pain to do all the stuff that I mentioned!
*The guide was taken from Basics of Debian Packages.

problem compiling libjingle

I downloaded and installed libjingle-0.5.2.zip, and according to the README also downloaded and installed swtoolkit.0.9.1.zip, scons-local-2.1.0.alpha.20101125.tar.gz, and expat-2.0.1.tar.gz, and got nrtp by cvs download. After overwriting my Makefile twice, attempting to follow the rather poorly-written README, I came up with the following Makefile that almost works:
# First, make sure the SCONS_DIR environment variable is set correctly.
SCONS_DIR ?= /usr/src/scons-local/scons-local-2.1.0.alpha.20101125/
#SCONS_DIR ?= /usr/src/scons-local/
export
default: build
# Second, run talk/third_party/expat-2.0.1/configure...
talk/third_party/expat-2.0.1/Makefile:
cd talk/third_party/expat-2.0.1 && ./configure
# ...and talk/third_party/srtp/configure.
talk/third_party/srtp/Makefile:
cd talk/third_party/srtp && ./configure
# Third, go to the talk/ directory and run $path_to_swtoolkit/hammer.sh. Run
# $path_to_swtoolkit/hammer.sh --help for information on how to build for
# different modes.
build: talk/third_party/expat-2.0.1/Makefile talk/third_party/srtp/Makefile
cd talk && ../../swtoolkit/hammer.sh
help:
../swtoolkit/hammer.sh --help
However, make gives me the following errors:
jcomeau#intrepid:/usr/src/libjingle-0.5.2$ make
cd talk && ../../swtoolkit/hammer.sh
*** Error loading site_init file './../../swtoolkit/site_scons/site_init.py':
AttributeError: 'Dir' object has no attribute 'endswith':
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 1338:
_exec_main(parser, values)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 1302:
_main(parser)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 929:
_load_site_scons_dir(d.path, options.site_dir)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 719:
exec fp in site_m
File "./../../swtoolkit/site_scons/site_init.py", line 455:
SiteInitMain()
File "./../../swtoolkit/site_scons/site_init.py", line 451:
SCons.Node.FS.get_default_fs().SConstruct_dir, None)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 677:
site_dir = os.path.join(topdir, site_dir_name)
File "/usr/lib/python2.6/posixpath.py", line 67:
elif path == '' or path.endswith('/'):
make: *** [build] Error 2
I'm guessing that something new (a 'Dir' object being where a POSIX path string is expected) in one of the packages is breaking the build process, but which one? There are just too many layers of cruft here for me to follow. For sure I could just keep trying older packages, particularly for swtoolkit and scons, but if anyone here has successfully compiled libjingle and could prod me in the right direction, I'd appreciate it.

I'm not familiar with the project, but think I have a fix to get you past that point. You need to cast those Dir instances using str() in swtoolkit/site_scons/site_init.py. That way they can safely be evaluated by path.endswith('/'). Odd that such an issue would exist for very long in the main part of the build infrastructure:
Line 330:
SCons.Script.Main._load_site_scons_dir(
str(SCons.Node.FS.get_default_fs().SConstruct_dir), site_dir)
Line 450:
SCons.Script.Main._load_site_scons_dir(
str(SCons.Node.FS.get_default_fs().SConstruct_dir), None)

I did following to build libjingle :
Building LibJingle for Linux
How to Build
Libjingle is built with swtoolkit ( http://code.google.com/p/swtoolkit/), which
is a set of extensions to the open-source SCons build tool ( http://www.scons.org).
First, install Python 2.4 or later from http://www.python.org/.
Please note that since swtoolkit only works with Python 2.x, you will
not be able to use Python 3.x.
Second, install the stand alone scons-local package 2.0.0 or later from
http://www.scons.org/download.php and set an environment variable,
SCONS_DIR, to point to the directory containing SCons, for example,
/src/libjingle/scons-local/scons-local-2.0.0.final.0/.
Third, install swtoolkit from http://code.google.com/p/swtoolkit/.
Finally, Libjingle depends on two open-source projects, expat and srtp.
Download expat from http://sourceforge.net/projects/expat/ to
talk/third_party/expat-2.0.1/. Follow the instructions at
http://sourceforge.net/projects/srtp/develop to download latest srtp to
talk/third_party/srtp. Note that srtp-1.4.4 does not work since it misses
the extensions used by Libjingle.
If you put expat or srtp in a different directory, you need to edit
talk/libjingle.scons correspondingly.
2.1 Build Libjingle under Linux or OS X
First, make sure the SCONS_DIR environment variable is set correctly.
Second, run talk/third_party/expat-2.0.1/configure and
talk/third_party/srtp/configure.
Third, go to the talk/ directory and run $path_to_swtoolkit/hammer.sh. Run
$path_to_swtoolkit/hammer.sh --help for information on how to build for
different modes.
Other than above given steps, See following as reference
Set SCONS_DIR Path
export SCONS_DIR=/home/esumit/libjingle/libjingle-0.5.2/talk/third_party/scons-local/scons-local-2.0.1
Install libasound2-dev Lib to compile libJingle, otherwise you will encounter errors.
sudo apt-get install libasound2-dev
Download SRTP using the following command. If it asks for a passowrd, just hit Enter.
cvs -z3 -d:pserver:anonymous#srtp.cvs.sourceforge.net:/cvsroot/srtp co -P srtp
Possible components in LibJingle Directory
libjingle-0.5.2/talk/third_party$ ls
expat-2.0.1 libudev scons-local srtp swtoolkit
Execute following command to build LibJingle
libjingle-0.5.2/talk$ ./third_party/swtoolkit/hammer.sh