We Keep Coding

Installing test files with pyproject.toml and setuptools

I'm migrating an old python project to the new pyproject.toml based system and am having trouble with getting files that are required by tests to install. Inside the pyproject.toml I have:
[tool.setuptools]
package-data = {"my_pkg_name" = ["tests/*.sdf", "tests/*.urdf", "tests/*.xml", "tests/meshes/*.obj"]}
[build-system]
requires = ["setuptools>=43.0.0", "wheel"]
build-backend = "setuptools.build_meta"
The tests that are run with pytest require the files described under package-data. After I build and install the build, the test files are not there. How do I get those files to be installed? How to include package data with setuptools/distutils? may be related, but things have changed, and I would rather not have to create a manifest file.
The project structure looks something like:
.
├── LICENSE.txt
├── pyproject.toml
├── README.md
├── src
│   ├── my_pkg_name
│   │   ├── __init__.py
└── tests
├── ant.xml
├── humanoid.xml
├── __init__.py
├── kuka_iiwa.urdf
├── meshes
│   ├── link_0.obj
│   ├── link_1.obj
│   ├── link_2.obj
│   ├── link_3.obj
│   ├── link_4.obj
│   ├── link_5.obj
│   ├── link_6.obj
│   └── link_7.obj
└── test_transform.py
The pyproject.toml has no specific package discovery related settings.

setuptools sdist has appropriate files, wheel is missing top folder for packages

I'm trying to use the basic example from the docs. I am able to successfully generate an sdist, but I want to be able to install from wheel. However, when installing from wheel instead of from the sdist, I don't have a proj directory that I can import, only pkg1 and pkg2.
Starting from this directory:
├── proj
│   ├── __init__.py
│   ├── additional
│   │   └── __init__.py
│   ├── pkg1
│   │   └── __init__.py
│   └── pkg2
│   └── __init__.py
├── pyproject.toml
└── setup.cfg
I tried using this setup.cfg:
[metadata]
name = p1
version = 0.0.1
[options]
packages =
find:
package_dir =
=proj
[options.packages.find]
where = proj
include = pkg*
exclude = additional
To generate this file structure, which is successfully generated in the source distribution:
├── PKG-INFO
├── README.md
├── proj
│   ├── p1.egg-info
│   │   ├── PKG-INFO
│   │   ├── SOURCES.txt
│   │   ├── dependency_links.txt
│   │   └── top_level.txt
│   ├── pkg1
│   │   └── __init__.py
│   └── pkg2
│   └── __init__.py
├── pyproject.toml
└── setup.cfg
But the wheel has this structure:
├── p1-0.0.1.dist-info
│   ├── METADATA
│   ├── RECORD
│   ├── WHEEL
│   └── top_level.txt
├── pkg1
│   └── __init__.py
└── pkg2
└── __init__.py
This makes it impossible to use:
import proj
As there is no such module, only modules pkg1 and pkg2.
I made a repo for this here: https://github.com/spensmith/setuptools-find.
Steps to recreate:
Clone the base repository here: https://github.com/spensmith/setuptools-find
cd setuptools-find
python -m build
mkdir dist/mywheel
unzip dist/p1-0.0.1-py3-none-any.whl -d dist/mywheel
tree dist/mywheel
I must be missing something basic, any help would be so appreciated!
Thank you, Spencer.

My question was answered directly on the setuptools discussion forum. https://github.com/pypa/setuptools/discussions/3185#discussioncomment-2411330.
Effectively, I shouldn't be using
package_dir =
=proj
Because this means that all of my code is within this folder. Instead, I should remove it entirely, and then change my exclude to look like:
[options.packages.find]
exclude = proj.additional
Finally, yielding this file:
[metadata]
name = p1
version = 0.0.1
[options]
packages =
find:
[options.packages.find]
exclude = proj.additional

Python package-namespace: common test/docs/setup.py or one per namespace - which is the better pattern?

(In the interest of transparency, this is a follow up to a question asked here)
I'm dealing with related files in which a namespace package seems a good fit. I'm following the guide from the packaging authority, which places a setup.py in each namespace package;
mynamespace-subpackage-a/
setup.py
mynamespace/
subpackage_a/
__init__.py
mynamespace-subpackage-b/
setup.py
mynamespace/
subpackage_b/
__init__.py
module_b.py
In my tests, created a similar project. Apart from setup.py, I placed my unit tests, docs, and other stuff per namespace (I left out some of the directories for compactness.). I used pyscaffold to generate the namespaces.
├── namespace-package-test.package1
│ ├── LICENSE.txt
│ ├── README.md
│ ├── setup.cfg
│ ├── setup.py
│ ├── src
│ │ └── pkg1
│ │ ├── cli
│ │ │ ├── __init__.py
│ │ │ └── pkg1_cli.py
│ │ └── __init__.py
│ └── tests
├── namespace-package-test.package2
│ ├── AUTHORS.rst
However, I then noticed that pyscaffold has the option to create namespaces packages in the putup command.
(venv) steve#PRVL10SJACKSON:~/Temp$ putup --force my-package -p pkg1 --namespace namespace1
(venv) steve#PRVL10SJACKSON:~/Temp$ putup --force my-package -p pkg1 --namespace namespace2
This creates a folder structure like this;
├── AUTHORS.rst
├── CHANGELOG.rst
├── LICENSE.txt
├── README.rst
├── requirements.txt
├── setup.cfg
├── setup.py
├── src
│   ├── namespace1
│   │   ├── __init__.py
│   │   └── pkg1
│   │   ├── __init__.py
│   │   └── skeleton.py
│   └── namespace2
│   ├── __init__.py
│   └── pkg1
│   ├── __init__.py
│   └── skeleton.py
└── tests
├── conftest.py
└── test_skeleton.py
So I'm conflicted; I trust the team at pyscaffold, but it goes against the example from the packaging authority.
Are both approaches valid?
Is there a reason to choose one approach over the other?

The idea behind the namespace option in PyScaffold is to share/reuse namespaces across projects (in opposite of having more than one namespace inside a single project). Or in other words, to split a larger project in independently maintained/developed projects.
To my best understanding, having an structure like the one you showed in the 4th code block will not work. Using putup --force twice with 2 different namespaces for the same root folder is not the intended/supported usage.
The approach of PyScaffold is the same as the package authority, the only difference is that PyScaffold will assume you have only one package contained in a single project and git repository (PyScaffold also uses a src directory for the reasons explained in Ionel's blog post)
The reason behind adopting one setup.py per namespace+package is that it is required for building separated distribution files (i.e. you need one setup.py per *.whl).

Setuptools how to package just some modules and files

I'm packaging a little python package. I'm a complete newbie to python packaging, my directory structure is as follows (up to second level nesting):
.
├── data
│   ├── images
│   ├── patches
│   └── train.csv
├── docker
│   ├── check_gpu.py
│   ├── Dockerfile.gcloud_base
│   ├── Dockerfile.gcloud_myproject
├── env.sh
├── gcloud_config_p100.yml
├── legacy
│   ├── __init__.py
│   ├── notebooks
│   └── mypackage
├── notebooks
│   ├── EDA.ipynb
│   ├── Inspect_patches.ipynb
├── README.md
├── requirements.txt
├── scripts
│   ├── create_patches_folds.py
│   └── create_patches.py
├── setup.py
├── mypackage
   ├── data
   ├── img
   ├── __init__.py
   ├── jupyter
   ├── keras_utils
   ├── models
   ├── train.py
   └── util.py
My setup.py:
import os
from setuptools import setup, find_packages
REQUIRED_PACKAGES = [
"h5py==2.9.0",
"numpy==1.16.4",
"opencv-python==4.1.0.25",
"pandas==0.24.2",
"keras==2.2.4",
"albumentations==0.3.1"
]
setup(
name='mypackage',
version='0.1',
install_requires=REQUIRED_PACKAGES,
packages=find_packages(include=["mypackage.*"]),
include_package_data=False
)
The code i want to package corresponds only to the mypackage directory. That's why i passed "mypackage.*" to find_packages and used include_package_data=False.
If i run:
python setup.py sdist
All project structure gets packaged in the resulting tar.gz file.
Anyone knowing how to just package modules inside mypackage/ and the top level README file? I'm not finding this in setuptools docs.

First thing to fix is
packages=find_packages(include=["mypackage"]),
But you also need to understand that sdist is mostly controlled by the files MANIFEST or its template MANIFEST.in, not setup.py. You can compare what is created with sdist and bdist_wheel or bdist_egg; content of bdist_* is controlled by setup.py.
So my advice is to create the following MANIFEST.in:
prune *
include README.txt
recursive-include mypackage *.py

Pip install ignores files in MANIFEST.in - how to structure the project correctly?

I read a lot about this problem but could not find any solution, so I'll ask yet another question about it, since I'm not even sure if I use the correct folder structure for my Python package.
So basically I'm developing an application which uses the Tornado web-server framework and I want to package it, so the users can install it via pip and get access to a basic script to start the web server.
The directory structure is the following:
├── MANIFEST.in
├── README.md
├── config
│   └── default.cfg
├── docs
│   ├── Makefile
│   ├── _build
│   ├── _static
│   ├── _templates
│   ├── conf.py
│   ├── index.rst
├── foopackage
│   ├── __init__.py
│   ├── barmodule.py
│   └── bazmodule.py
├── setup.py
├── static
│   ├── css
│   │   ├── menu.css
│   │   └── main.css
│   ├── img
│   │   └── logo.png
│   ├── js
│   │   ├── ui.js
│   │   └── navigation.js
│   └── lib
│      ├── d3.v3.min.js
│      └── jquery-1.11.0.min.js
└── templates
├── index.html
└── whatever.html
The Python code is as you can see in the package foopackage.
The MANIFEST.in file recursively includes the directories config, static, templates, and docs.
This is my setup.py (only the relevant parts:
from setuptools import setup
setup(name='foo',
version='0.1.0',
packages=['foopackage'],
include_package_data=True,
install_requires=[
'tornado>=3.2.2',
],
entry_points={
'console_scripts': [
'foo=foopackage.barmodule:main',
],
},
)
If I run python setup.py sdist, everything gets packaged nicely, the docs, templates and config files etc. are included. However, if I run pip install ..., only the foopackage gets installed and everything else is ignored.
How do I include those additional files to the install procedure? Is my directory structure OK? I also read about "faking a package", so putting everything in a directory and touch a __init__.py file, but that seems pretty odd to me :-\

I solved the problem by moving the static directory to the actual Python module directory (foopackage). It seems that top level "non-package" folders are ignored otherwise.