EDIT: The question is a bit too long. Here is my real question: How can I build and install a python package with setuptools (setup.py) inside CMake? The detail of my code is shown below (but with an out-of-source build method, the method with the source is working).
I have a project where I need to distribute my own python package. I made a setup.py script but I would like to build & install it with CMake.
I followed Using CMake with setup.py but it only works with one CMakeLists.txt alongside the setup.py and the python folder and without executing cmake from a build directory.
With this layout :
Project/
--build/
--lib/
----python/
------folder1/
------folder2/
------data/
------...
------__init__.py
----setup.py
----CMakeLists.txt
--CMakeLists.txt
and with CMakeLists.txt:
cmake_minimum_required(VERSION 2.8.8 FATAL_ERROR)
add_subdirectory(lib)
(..)
and with lib/CMakeLists.txt:
find_program(PYTHON "python")
if (PYTHON)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(DEPS "${CMAKE_CURRENT_SOURCE_DIR}/python/__init__.py")
set(OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build")
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(OUTPUT ${OUTPUT}
COMMAND ${PYTHON}
ARGS setup.py build
DEPENDS ${DEPS})
add_custom_target(target ALL DEPENDS ${OUTPUT})
install(CODE "execute_process(COMMAND ${PYTHON} ${SETUP_PY} install)")
endif()
and with setup.py:
from setuptools import setup, find_packages
setup(name="python",
version="xx",
author="xx",
packages = find_packages(),
package_data = {'': ['*.txt']},
description="Python lib for xx")
When I run CMake from build directory and then make, the target is built but with nothing. It is as if no packages were found. The installation installs the python package without .py files.
setuptools doesn't know about the out of source build and therefore doesn't find any python source files (because you do not copy them to the binary dir, only the setup.py file seems to exist there). In order to fix this, you would have to copy the python source tree into the CMAKE_CURRENT_BINARY_DIR.
https://bloerg.net/2012/11/10/cmake-and-distutils.html suggests setting package_dir to ${CMAKE_CURRENT_SOURCE_DIR} in setup.py.
As pointed out previously you can copy your python files to the build folder, e.g. something like this
set(TARGET_NAME YourLib)
file(GLOB_RECURSE pyfiles python/*.py)
foreach (filename ${pyfiles})
get_filename_component(target "${filename}" NAME)
message(STATUS "Copying ${filename} to ${TARGET_NAME}/${target}")
configure_file("${filename}"
"${CMAKE_CURRENT_BINARY_DIR}/${TARGET_NAME}/${target}" COPYONLY)
endforeach (filename)
and then have a build target like this
add_custom_target(PyPackageBuild
COMMAND "${PYTHON_EXECUTABLE}" -m pip wheel .
WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
COMMENT "Building python wheel package"
)
add_dependencies(PyPackageBuild ${TARGET_NAME})
In case you do not want to use pip you have to adjust the PyPackageBuld target.
If you want to include some shared library, e.g. written in C++, which is build by other parts of your cmake project you have to copy the shared object file as well to the binary folder
set_target_properties(${TARGET_NAME} PROPERTIES
PREFIX "${PYTHON_MODULE_PREFIX}"
SUFFIX "${PYTHON_MODULE_EXTENSION}"
BUILD_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
LIBRARY_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/${TARGET_NAME}/libs"
BUILD_WITH_INSTALL_RPATH TRUE)
set(TARGET_PYMODULE_NAME "${PYTHON_MODULE_PREFIX}${TARGET_NAME}${PYTHON_MODULE_EXTENSION}")
and add it to the package_data in setup.py
....
package_data={
'': ['libs/YourLib.cpython-39-x86_64-linux-gnu.so']
}
You can find a working example using pybind11 for `C++ยด bindings at https://github.com/maximiliank/cmake_python_r_example
Related
I have created a custom python package following this guide, so I have the following structure:
mypackage/ <-- VCS root
mypackage/
submodule1/
submodule2/
setup.py
And setup.py contains exactly the same information as in the guide:
from setuptools import setup, find_packages
setup(name='mypackage',
version='0.1',
description='desc',
url='vcs_url',
author='Hodossy, Szabolcs',
author_email='myemail#example.com',
license='MIT',
packages=find_packages(),
install_requires=[
# deps
],
zip_safe=False)
I have noticed if I go into the folder where setup.py is, and then call python setup.py install in a virtual environment, in site-packages the following structure is installed:
.../site-packages/mypackage-0.1-py3.6.egg/mypackage/
submodule1/
submodule2/
but if I call it from one folder up like python mypackage/setup.py install, then the structure is the following:
.../site-packages/mypackage-0.1-py3.6.egg/mypackage/
mypackage/
submodule1/
submodule2/
This later one ruins all imports from my module, as the path is different for the submodules.
Could you explain what is happening here and how to prevent that kind of behaviour?
This is experienced with Python 3.6 on both Windows and Linux.
Your setup.py does not contain any paths, but seems to only find the files via find_packages. So of course it depends from where you run it. The setup.py isn't strictly tied to its location. Of course you could do things like chdir to the basename of the setup file path in sys.argv[0], but that's rather ugly.
The question is, WHY do you want to build it that way? It looks more like you would want a structure like
mypackage-source
mypackage
submodule1
submodule2
setup.py
And then execute setup.py from the work directory. If you want to be able to run it from anywhere, the better workaround would be to put a shellscript next to it, like
#!/bin/sh
cd ``basename $0``
python setup.py $#
which separates the task of changing to the right directory (here I assume the directory with setup.py in the workdir) from running setup.py
how can I make setup.py file for my own script? I have to make my script global.
(add it to /usr/bin) so I could run it from console just type: scriptName arguments.
OS: Linux.
EDIT:
Now my script is installable, but how can i make it global? So that i could run it from console just name typing.
EDIT: This answer deals only with installing executable scripts into /usr/bin. I assume you have basic knowledge on how setup.py files work.
Create your script and place it in your project like this:
yourprojectdir/
setup.py
scripts/
myscript.sh
In your setup.py file do this:
from setuptools import setup
# you may need setuptools instead of distutils
setup(
# basic stuff here
scripts = [
'scripts/myscript.sh'
]
)
Then type
python setup.py install
Basically that's it. There's a chance that your script will land not exactly in /usr/bin, but in some other directory. If this is the case, type
python setup.py install --help
and search for --install-scripts parameter and friends.
I know that this question is quite old, but just in case, I post how I solved the problem for myself, that was wanting to setup a package for PyPI, that, when installing it with pip, would install it as a system package, not just for Python.
setup(
# rest of setup
console_scripts={
'console_scripts': [
'<app> = <package>.<app>:main'
]
},
)
Details
I read a lot of answers on this question, but no solution works for me.
Project layout:
generators_data\
en_family_names.txt
en_female_names.txt
__init__.py
generators.py
setup.py
I want include "generators_data" with it's content into installation. My setup.py:
from distutils.core import setup
setup(name='generators',
version='1.0',
package_data={'generators': ['generators_data/*']}
)
I tried
python setup.py install
got
running install
running build
running install_egg_info
Removing c:\Python27\Lib\site-packages\generators-1.0-py2.7.egg-info
Writing c:\Python27\Lib\site-packages\generators-1.0-py2.7.egg-info
but generators_data directory doesn't appear in "c:\Python27\Lib\site-packages\". Why?
The code you posted contains two issues: setup.py should be sibling to the package you want to distribute, not inside it, and you need to list packages in setup.py.
Try with this this layout:
generators/ # project root, the directory you get from git clone or equivalent
setup.py
generators/ # Python package
__init__.py
# other modules
generators_data/
names.txt
And this setup.py:
setup(name='generators',
version='1.0',
packages=['generators'],
package_data={'generators': ['generators_data/*']},
)
I'm using setuptools 0.6 to package my code. By default when I type python setup.py sdist, the tool creates a folder dist directly in the same folder of my script. What I can do to change this default folder?
Edit: Another question is, if my setup script and my package folder are not in the same folder, what can I do to specify the exact path of the package?
Thanks
Use the --dist-dir=[differentdir] option. From python setup.py sdist --help:
--dist-dir (-d) directory to put the source distribution archive(s) in
[default: dist]
You can specify the top-level package directory with the package_dir keyword argument to setup():
package_dir = {'': 'src'},
I can recommend the Python Packaging User Guide for a good tutorial on how to package your python projects.
I'm using setuptools 0.6 to package my code. By default when I type python setup.py sdist, the tool creates a folder dist directly in the same folder of my script. What I can do to change this default folder?
Edit: Another question is, if my setup script and my package folder are not in the same folder, what can I do to specify the exact path of the package?
Thanks
Use the --dist-dir=[differentdir] option. From python setup.py sdist --help:
--dist-dir (-d) directory to put the source distribution archive(s) in
[default: dist]
You can specify the top-level package directory with the package_dir keyword argument to setup():
package_dir = {'': 'src'},
I can recommend the Python Packaging User Guide for a good tutorial on how to package your python projects.