We Keep Coding

Possible to create python sdist from different directory?

Want to create python source distribution by running python setup.py sdist from a directory outside of the one I want to package up. Can't seem to find a way to do this. I have a script that generates a setup.py and MANIFEST.in dynamically, and I'd like to tell python to use those files to create an sdist of the source in a different directory "over there".
What I'm doing is creating a script that lets a user create an sdist w/o any setup.py etc. They just say "package up this directory and everything under it". So I generate a setup.py and MANIFEST.in (with recursive-include * to grab all files) in a python tempfile.mkdtemp directory (in an unrelated file path like /tmp/whatever) that I can clean up afterwards...but I can't seem to use those to package their directory. I don't want to create those files in their source dir.

You can use setuptools's, --dist-dir=DIR / -d DIR option to specify where the otherwise default dist/-folder is made. In other words, this changes the output directory.
E.g.:
python setup.py sdist -d /tmp/whatever
If you are using distutils.core: Instead of using from distutils.core import setup you can use from setuptools import setup.
In order to define where the source directories come from, I think you can add the directory to sys.path and then setup() will discover the content files automatically:
import sys
from os import path
# ...
# Add other folders to sys.path
sys.path.append('/tmp/whatever')
sys.path.append(path.join(path.abspath('..'), 'some', 'folder'))

Sort of a hack, but this worked for me. Right before running setup(), use os.chdir() to change the directory to that of the base path where setup.py would normally run. To specify where the distribution packages go, I use the arguments to setup.py, specifically:
python setup.py sdist --formats=gztar -d 'directory_for_the_distribution' egg_info --egg-base 'directory_for_the_egg_info'
Thus you can run setuptools from a directory other than at the base of the package directories and the distribution and temp egg directories go wherever you want.

Build a python package with setup.py in CMake

EDIT: The question is a bit too long. Here is my real question: How can I build and install a python package with setuptools (setup.py) inside CMake? The detail of my code is shown below (but with an out-of-source build method, the method with the source is working).
I have a project where I need to distribute my own python package. I made a setup.py script but I would like to build & install it with CMake.
I followed Using CMake with setup.py but it only works with one CMakeLists.txt alongside the setup.py and the python folder and without executing cmake from a build directory.
With this layout :
Project/
--build/
--lib/
----python/
------folder1/
------folder2/
------data/
------...
------__init__.py
----setup.py
----CMakeLists.txt
--CMakeLists.txt
and with CMakeLists.txt:
cmake_minimum_required(VERSION 2.8.8 FATAL_ERROR)
add_subdirectory(lib)
(..)
and with lib/CMakeLists.txt:
find_program(PYTHON "python")
if (PYTHON)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(DEPS "${CMAKE_CURRENT_SOURCE_DIR}/python/__init__.py")
set(OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build")
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(OUTPUT ${OUTPUT}
COMMAND ${PYTHON}
ARGS setup.py build
DEPENDS ${DEPS})
add_custom_target(target ALL DEPENDS ${OUTPUT})
install(CODE "execute_process(COMMAND ${PYTHON} ${SETUP_PY} install)")
endif()
and with setup.py:
from setuptools import setup, find_packages
setup(name="python",
version="xx",
author="xx",
packages = find_packages(),
package_data = {'': ['*.txt']},
description="Python lib for xx")
When I run CMake from build directory and then make, the target is built but with nothing. It is as if no packages were found. The installation installs the python package without .py files.

setuptools doesn't know about the out of source build and therefore doesn't find any python source files (because you do not copy them to the binary dir, only the setup.py file seems to exist there). In order to fix this, you would have to copy the python source tree into the CMAKE_CURRENT_BINARY_DIR.

https://bloerg.net/2012/11/10/cmake-and-distutils.html suggests setting package_dir to ${CMAKE_CURRENT_SOURCE_DIR} in setup.py.

As pointed out previously you can copy your python files to the build folder, e.g. something like this
set(TARGET_NAME YourLib)
file(GLOB_RECURSE pyfiles python/*.py)
foreach (filename ${pyfiles})
get_filename_component(target "${filename}" NAME)
message(STATUS "Copying ${filename} to ${TARGET_NAME}/${target}")
configure_file("${filename}"
"${CMAKE_CURRENT_BINARY_DIR}/${TARGET_NAME}/${target}" COPYONLY)
endforeach (filename)
and then have a build target like this
add_custom_target(PyPackageBuild
COMMAND "${PYTHON_EXECUTABLE}" -m pip wheel .
WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
COMMENT "Building python wheel package"
)
add_dependencies(PyPackageBuild ${TARGET_NAME})
In case you do not want to use pip you have to adjust the PyPackageBuld target.
If you want to include some shared library, e.g. written in C++, which is build by other parts of your cmake project you have to copy the shared object file as well to the binary folder
set_target_properties(${TARGET_NAME} PROPERTIES
PREFIX "${PYTHON_MODULE_PREFIX}"
SUFFIX "${PYTHON_MODULE_EXTENSION}"
BUILD_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
LIBRARY_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/${TARGET_NAME}/libs"
BUILD_WITH_INSTALL_RPATH TRUE)
set(TARGET_PYMODULE_NAME "${PYTHON_MODULE_PREFIX}${TARGET_NAME}${PYTHON_MODULE_EXTENSION}")
and add it to the package_data in setup.py
....
package_data={
'': ['libs/YourLib.cpython-39-x86_64-linux-gnu.so']
}
You can find a working example using pybind11 for `C++´ bindings at https://github.com/maximiliank/cmake_python_r_example

Force `sdist` to create .zip archive even on Linux

I know it is possible to force sdist to produce .zip from command line:
python setup.py sdist --formats=zip
But how to make this option a default for my setup.py?
I'd like to get consistency for running setup.py sdist on both Windows and Linux, and I choose .zip format, because I can turn .zip into executable.

Found it myself from distutils docs here and here, and from distutils sources:
# Override sdist to always produce .zip archive
from distutils.command.sdist import sdist as _sdist
class sdistzip(_sdist):
def initialize_options(self):
_sdist.initialize_options(self)
self.formats = 'zip'
setup(
...
cmdclass={'sdist': sdistzip},
)

Change default dist folder with setuptools

I'm using setuptools 0.6 to package my code. By default when I type python setup.py sdist, the tool creates a folder dist directly in the same folder of my script. What I can do to change this default folder?
Edit: Another question is, if my setup script and my package folder are not in the same folder, what can I do to specify the exact path of the package?
Thanks

Use the --dist-dir=[differentdir] option. From python setup.py sdist --help:
--dist-dir (-d) directory to put the source distribution archive(s) in
[default: dist]
You can specify the top-level package directory with the package_dir keyword argument to setup():
package_dir = {'': 'src'},
I can recommend the Python Packaging User Guide for a good tutorial on how to package your python projects.

setup.py not installing data files

I have a Python library that, in addition to regular Python modules, has some data files that need to go in /usr/local/lib/python2.7/dist-package/mylibrary.
Unfortunately, I have been unable to convince setup.py to actually install the data files there. Note that this behaviour is under install - not sdist.
Here is a slightly redacted version of setup.py
module_list = list_of_files
setup(name ='Modules',
version ='1.33.7',
description ='My Sweet Module',
author ='PN',
author_email ='email',
url ='url',
packages = ['my_module'],
# I tried this. It got installed in /usr/my_module. Not ok.
# data_files = [ ("my_module", ["my_module/data1",
# "my_module/data2"])]
# This doesn't install it at all.
package_data = {"my_module" : ["my_module/data1",
"my_module/data2"] }
)
This is in Python 2.7 (will have to run in 2.6 eventually), and will have to run on some Ubuntu between 10.04 and 12+. Developing it right now on 12.04.

UPD:
package_data accepts dict in format {'package': ['list', 'of?', 'globs*']}, so to make it work, one should specify shell globs relative to package dir, not the file paths relative to the distribution root.
data_files has a different meaning, and, in general, one should avoid using this parameter.
With setuptools you only need include_package_data=True, but data files should be under version control system, known to setuptools (by default it recognizes only CVS and SVN, install setuptools-git or setuptools-hg if you use git or hg...)
with setuptools you can:
- in MANIFEST.im:
include my_module/data*
- in setup.py:
setup(
...
include_package_data = True,
...
)

http://docs.python.org/distutils/setupscript.html#installing-additional-files
If directory is a relative path, it is interpreted relative to the
installation prefix (Python’s sys.prefix for pure-Python packages,
sys.exec_prefix for packages that contain extension modules).
This will probably do it:
data_files = [ ("my_module", ["local/lib/python2.7/dist-package/my_module/data1",
"local/lib/python2.7/dist-package/my_module/data2"])]
Or just use join to add the prefix:
data_dir = os.path.join(sys.prefix, "local/lib/python2.7/dist-package/my_module")
data_files = [ ("my_module", [os.path.join(data_dir, "data1"),
os.path.join(data_dir, "data2")])]

The following solution worked fine for me.
You should have MANIFEST.in file where setup.py is located.
Add the following code to the manifest file
recursive-include mypackage *.json *.md # can be extended with more extensions or file names.
Another solution is adding the following code to the MANIFEST.in file.
graft mypackage # will copy the entire package including non-python files.
global-exclude __pyache__ *.txt # list files you dont want to include here.
Now, when you do pip install all the necessary files will be included.
Hope this helps.
UPDATE:
Make sure that you also have include_package_data=True in the setup file