I'm migrating an old python project to the new pyproject.toml based system and am having trouble with getting files that are required by tests to install. Inside the pyproject.toml I have:
[tool.setuptools]
package-data = {"my_pkg_name" = ["tests/*.sdf", "tests/*.urdf", "tests/*.xml", "tests/meshes/*.obj"]}
[build-system]
requires = ["setuptools>=43.0.0", "wheel"]
build-backend = "setuptools.build_meta"
The tests that are run with pytest require the files described under package-data. After I build and install the build, the test files are not there. How do I get those files to be installed? How to include package data with setuptools/distutils? may be related, but things have changed, and I would rather not have to create a manifest file.
The project structure looks something like:
.
├── LICENSE.txt
├── pyproject.toml
├── README.md
├── src
│ ├── my_pkg_name
│ │ ├── __init__.py
└── tests
├── ant.xml
├── humanoid.xml
├── __init__.py
├── kuka_iiwa.urdf
├── meshes
│ ├── link_0.obj
│ ├── link_1.obj
│ ├── link_2.obj
│ ├── link_3.obj
│ ├── link_4.obj
│ ├── link_5.obj
│ ├── link_6.obj
│ └── link_7.obj
└── test_transform.py
The pyproject.toml has no specific package discovery related settings.
Related
I'm trying to use the basic example from the docs. I am able to successfully generate an sdist, but I want to be able to install from wheel. However, when installing from wheel instead of from the sdist, I don't have a proj directory that I can import, only pkg1 and pkg2.
Starting from this directory:
├── proj
│ ├── __init__.py
│ ├── additional
│ │ └── __init__.py
│ ├── pkg1
│ │ └── __init__.py
│ └── pkg2
│ └── __init__.py
├── pyproject.toml
└── setup.cfg
I tried using this setup.cfg:
[metadata]
name = p1
version = 0.0.1
[options]
packages =
find:
package_dir =
=proj
[options.packages.find]
where = proj
include = pkg*
exclude = additional
To generate this file structure, which is successfully generated in the source distribution:
├── PKG-INFO
├── README.md
├── proj
│ ├── p1.egg-info
│ │ ├── PKG-INFO
│ │ ├── SOURCES.txt
│ │ ├── dependency_links.txt
│ │ └── top_level.txt
│ ├── pkg1
│ │ └── __init__.py
│ └── pkg2
│ └── __init__.py
├── pyproject.toml
└── setup.cfg
But the wheel has this structure:
├── p1-0.0.1.dist-info
│ ├── METADATA
│ ├── RECORD
│ ├── WHEEL
│ └── top_level.txt
├── pkg1
│ └── __init__.py
└── pkg2
└── __init__.py
This makes it impossible to use:
import proj
As there is no such module, only modules pkg1 and pkg2.
I made a repo for this here: https://github.com/spensmith/setuptools-find.
Steps to recreate:
Clone the base repository here: https://github.com/spensmith/setuptools-find
cd setuptools-find
python -m build
mkdir dist/mywheel
unzip dist/p1-0.0.1-py3-none-any.whl -d dist/mywheel
tree dist/mywheel
I must be missing something basic, any help would be so appreciated!
Thank you, Spencer.
My question was answered directly on the setuptools discussion forum. https://github.com/pypa/setuptools/discussions/3185#discussioncomment-2411330.
Effectively, I shouldn't be using
package_dir =
=proj
Because this means that all of my code is within this folder. Instead, I should remove it entirely, and then change my exclude to look like:
[options.packages.find]
exclude = proj.additional
Finally, yielding this file:
[metadata]
name = p1
version = 0.0.1
[options]
packages =
find:
[options.packages.find]
exclude = proj.additional
I'm packaging a little python package. I'm a complete newbie to python packaging, my directory structure is as follows (up to second level nesting):
.
├── data
│ ├── images
│ ├── patches
│ └── train.csv
├── docker
│ ├── check_gpu.py
│ ├── Dockerfile.gcloud_base
│ ├── Dockerfile.gcloud_myproject
├── env.sh
├── gcloud_config_p100.yml
├── legacy
│ ├── __init__.py
│ ├── notebooks
│ └── mypackage
├── notebooks
│ ├── EDA.ipynb
│ ├── Inspect_patches.ipynb
├── README.md
├── requirements.txt
├── scripts
│ ├── create_patches_folds.py
│ └── create_patches.py
├── setup.py
├── mypackage
├── data
├── img
├── __init__.py
├── jupyter
├── keras_utils
├── models
├── train.py
└── util.py
My setup.py:
import os
from setuptools import setup, find_packages
REQUIRED_PACKAGES = [
"h5py==2.9.0",
"numpy==1.16.4",
"opencv-python==4.1.0.25",
"pandas==0.24.2",
"keras==2.2.4",
"albumentations==0.3.1"
]
setup(
name='mypackage',
version='0.1',
install_requires=REQUIRED_PACKAGES,
packages=find_packages(include=["mypackage.*"]),
include_package_data=False
)
The code i want to package corresponds only to the mypackage directory. That's why i passed "mypackage.*" to find_packages and used include_package_data=False.
If i run:
python setup.py sdist
All project structure gets packaged in the resulting tar.gz file.
Anyone knowing how to just package modules inside mypackage/ and the top level README file? I'm not finding this in setuptools docs.
First thing to fix is
packages=find_packages(include=["mypackage"]),
But you also need to understand that sdist is mostly controlled by the files MANIFEST or its template MANIFEST.in, not setup.py. You can compare what is created with sdist and bdist_wheel or bdist_egg; content of bdist_* is controlled by setup.py.
So my advice is to create the following MANIFEST.in:
prune *
include README.txt
recursive-include mypackage *.py
My python project installs via setup.py. The project structure looks like:
├── Makefile
├── README.rst
├── circle.yml
├── docs
│ ├── Makefile
│ ├── conf.py
│ ├── deps.txt
│ ├── guide_installation.rst
│ ├── guide_model.rst
│ ├── guide_transliteration.rst
│ ├── index.rst
│ ├── make.bat
│ └── module_trans.rst
├── indictrans
│ ├── __init__.py
│ ├── _decode
│ ├── _utils
│ ├── base.py
│ ├── iso_code_transformer.py
│ ├── libindic_
│ ├── mappings
│ ├── models
│ ├── polyglot_tokenizer
│ ├── script_transliterate.py
│ ├── test.py
│ ├── tests
│ ├── transliterator.py
│ ├── trunk
│ └── unicode_marks.py
├── requirements.txt
├── setup.cfg
├── setup.py
├── test-requirements.txt
└── tox.ini
where the subfolder indictrans/models looks like
├── ben-eng
│ ├── classes.npy
│ ├── coef.npy
│ ├── intercept_final.npy
│ ├── intercept_init.npy
│ ├── intercept_trans.npy
│ └── sparse.vec
├── ben-guj
│ ├── classes.npy
│ ├── coef.npy
│ ├── intercept_final.npy
│ ├── intercept_init.npy
│ ├── intercept_trans.npy
│ └── sparse.vec
so I have .npy and .vec files to be included in the project.
In my setup.py I'm trying to explicitly include this folder models via the include_package_data directive like:
setup(
setup_requires=['pbr'],
pbr=True,
packages=find_packages(),
include_package_data=True,
package_data={'models': ['*.npy','*.vec']},
ext_modules=cythonize(extensions)
)
and in the setup.cfg I have
[files]
packages =
indictrans
but running python setup.py install does not copy the models folder to the installation folder /usr/local/lib/python2.7/dist-packages/indictrans/.
If I print the it is the output of the find_packages I get
['indictrans', 'indictrans.tests', 'indictrans.libindic_', 'indictrans._utils', 'indictrans._decode', 'indictrans.polyglot_tokenizer', 'indictrans.models', 'indictrans.trunk', 'indictrans.libindic_.utils', 'indictrans.libindic_.soundex', 'indictrans.libindic_.utils.tests', 'indictrans.libindic_.soundex.utils', 'indictrans.libindic_.soundex.tests', 'indictrans.libindic_.soundex.utils.tests', 'indictrans.polyglot_tokenizer.tests', 'indictrans.trunk.tests']
so I will assume that indictrans/models would be included, but it is not.
Add include_package_data=True to your setup-function (you already did that).
Create a file MANIFEST.in in the same directory as setup.py
MANIFEST.in can look as follows:
include indictrans/models/ben-eng/*
include indictrans/models/ben-guj/*
You don't need setup.cfg for doing this.
Source: This great writeup of python packaging
EDIT about recursive-include:
According to the documentation this should also work:
recursive-include indictrans/models *.npy *.vec
include_package_data=True requires MANIFEST.in.
To include data for the module indictrans.models you have to provide the full name:
package_data={'indictrans.models': ['*.npy','*.vec']},
I have a single, large git repo with many different projects (not submodules). A few of these projects are Python projects for which I'd like to track versioning with Python's versioneer, others may be completely independent projects (say, in Haskell). i.e. The directory structure looks like this:
myrepository
├── .git
├── README.md
├── project1/
│ ├── project1/
│ │ ├── __init__.py
│ │ └── _version.py
│ ├── versioneer.py
│ ├── setup.cfg
│ └── setup.py
├── project2/
├── project3/
└── project4/
├── project4/
│ ├── __init__.py
│ └── _version.py
├── versioneer.py
├── setup.cfg
└── setup.py
This doesn't play well with versioneer because it can't discover the .git directory at the project root level, so I get a version of 0+unknown.
Questions:
Is there a suggested way to use versioneer with a single monolithic repo with multiple projects?
Depending on the answer to the above, is it recommended that my git tags read like: project1-2.1.0, project2-1.3.1, or should I unify the git tags like: 1.2.1, 1.2.2?
I read a lot about this problem but could not find any solution, so I'll ask yet another question about it, since I'm not even sure if I use the correct folder structure for my Python package.
So basically I'm developing an application which uses the Tornado web-server framework and I want to package it, so the users can install it via pip and get access to a basic script to start the web server.
The directory structure is the following:
├── MANIFEST.in
├── README.md
├── config
│ └── default.cfg
├── docs
│ ├── Makefile
│ ├── _build
│ ├── _static
│ ├── _templates
│ ├── conf.py
│ ├── index.rst
├── foopackage
│ ├── __init__.py
│ ├── barmodule.py
│ └── bazmodule.py
├── setup.py
├── static
│ ├── css
│ │ ├── menu.css
│ │ └── main.css
│ ├── img
│ │ └── logo.png
│ ├── js
│ │ ├── ui.js
│ │ └── navigation.js
│ └── lib
│ ├── d3.v3.min.js
│ └── jquery-1.11.0.min.js
└── templates
├── index.html
└── whatever.html
The Python code is as you can see in the package foopackage.
The MANIFEST.in file recursively includes the directories config, static, templates, and docs.
This is my setup.py (only the relevant parts:
from setuptools import setup
setup(name='foo',
version='0.1.0',
packages=['foopackage'],
include_package_data=True,
install_requires=[
'tornado>=3.2.2',
],
entry_points={
'console_scripts': [
'foo=foopackage.barmodule:main',
],
},
)
If I run python setup.py sdist, everything gets packaged nicely, the docs, templates and config files etc. are included. However, if I run pip install ..., only the foopackage gets installed and everything else is ignored.
How do I include those additional files to the install procedure? Is my directory structure OK? I also read about "faking a package", so putting everything in a directory and touch a __init__.py file, but that seems pretty odd to me :-\
I solved the problem by moving the static directory to the actual Python module directory (foopackage). It seems that top level "non-package" folders are ignored otherwise.