We Keep Coding

Proper way to copy configuration files during installation?

I am trying to distribute a mplstyle I wrote such that I can share it easily. It boils down to copying a text file to the proper configuration direction (which is known for any architecture) during installation. I want to be able to install using either python setup.py install or pip install .... Currently I do not seem to get either of the two ways robust (see current approach below).
Installing with pip install ... does not seem to invoke the copying at all.
Installing with python setup.py install works well on my machine, but ReadTheDocs gives me the following error:
python setup.py install --force
running install
error: [Errno 2] No such file or directory: u'/home/docs/.config/matplotlib/stylelib/goose.mplsty
What is the proper way to copy configuration files during installation in a robust way?
Current approach
File structure
setup.py
goosempl/
| __init__.py
| stylelib/
| goose.mplstyle
| ...
setup.py
from setuptools import setup
from setuptools.command.install import install
class PostInstallCommand(install):
def run(self):
import goosempl
goosempl.copy_style()
install.run(self)
setup(
name = 'goosempl',
...,
install_requires = ['matplotlib>=2.0.0'],
packages = ['goosempl'],
cmdclass = {'install': PostInstallCommand},
package_data = {'goosempl/stylelib':['goosempl/stylelib/goose.mplstyle']},
)
goosempl/__init__.py
def copy_style():
import os
import matplotlib
from pkg_resources import resource_string
files = [
'stylelib/goose.mplstyle',
]
for fname in files:
path = os.path.join(matplotlib.get_configdir(),fname)
text = resource_string(__name__,fname).decode()
print(path, text)
open(path,'w').write(text)
Upload to PyPi
python setup.py bdist_wheel --universal
twine upload dist/*

First of all, based on the project structure you've provided, you're not specifying the package_data correctly. If goosempl is a package and stylelib a directory inside it containing the mplstyle files (what I assume from your code), then your package_data configuration line should be:
package_data = {'goosempl': ['stylelib/goose.mplstyle']},
As described in Building and Distributing Packages with Setuptools:
The package_data argument is a dictionary that maps from package names to lists of glob patterns. The globs may include subdirectory names, if the data files are contained in a subdirectory of the package.
So your package is goosempl and stylelib/goose.mplstyle is the file to be included in package data for goosempl.
Your second issue (No such file or directory) is simple: in the copy_style() function, you never check if the parent directory of the file exists before writing the file. You should be able to reproduce this locally by removing the directory /home/<user>/.config/matplotlib/stylelib/ (or moving it temporarily).
The fix is also quite simple, actually there are lots of them. Use whatever you want to create missing directories.
distutils.dir_util.mkpath is suitable for both python2 and python3:
for fname in files:
path = os.path.join(matplotlib.get_configdir(), fname)
distutils.dir_util.mkpath(os.dirname(path))
My preferred one is using pathlib, but it is available only since Python 3.4:
for fname in files:
path = pathlib.Path(matplotlib.get_configdir(), fname)
path.parent.mkdir(parents=True, exist_ok=True)

Possible to create python sdist from different directory?

Want to create python source distribution by running python setup.py sdist from a directory outside of the one I want to package up. Can't seem to find a way to do this. I have a script that generates a setup.py and MANIFEST.in dynamically, and I'd like to tell python to use those files to create an sdist of the source in a different directory "over there".
What I'm doing is creating a script that lets a user create an sdist w/o any setup.py etc. They just say "package up this directory and everything under it". So I generate a setup.py and MANIFEST.in (with recursive-include * to grab all files) in a python tempfile.mkdtemp directory (in an unrelated file path like /tmp/whatever) that I can clean up afterwards...but I can't seem to use those to package their directory. I don't want to create those files in their source dir.

You can use setuptools's, --dist-dir=DIR / -d DIR option to specify where the otherwise default dist/-folder is made. In other words, this changes the output directory.
E.g.:
python setup.py sdist -d /tmp/whatever
If you are using distutils.core: Instead of using from distutils.core import setup you can use from setuptools import setup.
In order to define where the source directories come from, I think you can add the directory to sys.path and then setup() will discover the content files automatically:
import sys
from os import path
# ...
# Add other folders to sys.path
sys.path.append('/tmp/whatever')
sys.path.append(path.join(path.abspath('..'), 'some', 'folder'))

Sort of a hack, but this worked for me. Right before running setup(), use os.chdir() to change the directory to that of the base path where setup.py would normally run. To specify where the distribution packages go, I use the arguments to setup.py, specifically:
python setup.py sdist --formats=gztar -d 'directory_for_the_distribution' egg_info --egg-base 'directory_for_the_egg_info'
Thus you can run setuptools from a directory other than at the base of the package directories and the distribution and temp egg directories go wherever you want.

Build a python package with setup.py in CMake

EDIT: The question is a bit too long. Here is my real question: How can I build and install a python package with setuptools (setup.py) inside CMake? The detail of my code is shown below (but with an out-of-source build method, the method with the source is working).
I have a project where I need to distribute my own python package. I made a setup.py script but I would like to build & install it with CMake.
I followed Using CMake with setup.py but it only works with one CMakeLists.txt alongside the setup.py and the python folder and without executing cmake from a build directory.
With this layout :
Project/
--build/
--lib/
----python/
------folder1/
------folder2/
------data/
------...
------__init__.py
----setup.py
----CMakeLists.txt
--CMakeLists.txt
and with CMakeLists.txt:
cmake_minimum_required(VERSION 2.8.8 FATAL_ERROR)
add_subdirectory(lib)
(..)
and with lib/CMakeLists.txt:
find_program(PYTHON "python")
if (PYTHON)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(DEPS "${CMAKE_CURRENT_SOURCE_DIR}/python/__init__.py")
set(OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build")
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(OUTPUT ${OUTPUT}
COMMAND ${PYTHON}
ARGS setup.py build
DEPENDS ${DEPS})
add_custom_target(target ALL DEPENDS ${OUTPUT})
install(CODE "execute_process(COMMAND ${PYTHON} ${SETUP_PY} install)")
endif()
and with setup.py:
from setuptools import setup, find_packages
setup(name="python",
version="xx",
author="xx",
packages = find_packages(),
package_data = {'': ['*.txt']},
description="Python lib for xx")
When I run CMake from build directory and then make, the target is built but with nothing. It is as if no packages were found. The installation installs the python package without .py files.

setuptools doesn't know about the out of source build and therefore doesn't find any python source files (because you do not copy them to the binary dir, only the setup.py file seems to exist there). In order to fix this, you would have to copy the python source tree into the CMAKE_CURRENT_BINARY_DIR.

https://bloerg.net/2012/11/10/cmake-and-distutils.html suggests setting package_dir to ${CMAKE_CURRENT_SOURCE_DIR} in setup.py.

As pointed out previously you can copy your python files to the build folder, e.g. something like this
set(TARGET_NAME YourLib)
file(GLOB_RECURSE pyfiles python/*.py)
foreach (filename ${pyfiles})
get_filename_component(target "${filename}" NAME)
message(STATUS "Copying ${filename} to ${TARGET_NAME}/${target}")
configure_file("${filename}"
"${CMAKE_CURRENT_BINARY_DIR}/${TARGET_NAME}/${target}" COPYONLY)
endforeach (filename)
and then have a build target like this
add_custom_target(PyPackageBuild
COMMAND "${PYTHON_EXECUTABLE}" -m pip wheel .
WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
COMMENT "Building python wheel package"
)
add_dependencies(PyPackageBuild ${TARGET_NAME})
In case you do not want to use pip you have to adjust the PyPackageBuld target.
If you want to include some shared library, e.g. written in C++, which is build by other parts of your cmake project you have to copy the shared object file as well to the binary folder
set_target_properties(${TARGET_NAME} PROPERTIES
PREFIX "${PYTHON_MODULE_PREFIX}"
SUFFIX "${PYTHON_MODULE_EXTENSION}"
BUILD_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
LIBRARY_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/${TARGET_NAME}/libs"
BUILD_WITH_INSTALL_RPATH TRUE)
set(TARGET_PYMODULE_NAME "${PYTHON_MODULE_PREFIX}${TARGET_NAME}${PYTHON_MODULE_EXTENSION}")
and add it to the package_data in setup.py
....
package_data={
'': ['libs/YourLib.cpython-39-x86_64-linux-gnu.so']
}
You can find a working example using pybind11 for `C++´ bindings at https://github.com/maximiliank/cmake_python_r_example

Change default dist folder with setuptools

I'm using setuptools 0.6 to package my code. By default when I type python setup.py sdist, the tool creates a folder dist directly in the same folder of my script. What I can do to change this default folder?
Edit: Another question is, if my setup script and my package folder are not in the same folder, what can I do to specify the exact path of the package?
Thanks

Use the --dist-dir=[differentdir] option. From python setup.py sdist --help:
--dist-dir (-d) directory to put the source distribution archive(s) in
[default: dist]
You can specify the top-level package directory with the package_dir keyword argument to setup():
package_dir = {'': 'src'},
I can recommend the Python Packaging User Guide for a good tutorial on how to package your python projects.

How to specify the setuptools output directory? [duplicate]

I'm using setuptools 0.6 to package my code. By default when I type python setup.py sdist, the tool creates a folder dist directly in the same folder of my script. What I can do to change this default folder?
Edit: Another question is, if my setup script and my package folder are not in the same folder, what can I do to specify the exact path of the package?
Thanks

Use the --dist-dir=[differentdir] option. From python setup.py sdist --help:
--dist-dir (-d) directory to put the source distribution archive(s) in
[default: dist]
You can specify the top-level package directory with the package_dir keyword argument to setup():
package_dir = {'': 'src'},
I can recommend the Python Packaging User Guide for a good tutorial on how to package your python projects.