I'm packaging a little python package. I'm a complete newbie to python packaging, my directory structure is as follows (up to second level nesting):
.
├── data
│ ├── images
│ ├── patches
│ └── train.csv
├── docker
│ ├── check_gpu.py
│ ├── Dockerfile.gcloud_base
│ ├── Dockerfile.gcloud_myproject
├── env.sh
├── gcloud_config_p100.yml
├── legacy
│ ├── __init__.py
│ ├── notebooks
│ └── mypackage
├── notebooks
│ ├── EDA.ipynb
│ ├── Inspect_patches.ipynb
├── README.md
├── requirements.txt
├── scripts
│ ├── create_patches_folds.py
│ └── create_patches.py
├── setup.py
├── mypackage
├── data
├── img
├── __init__.py
├── jupyter
├── keras_utils
├── models
├── train.py
└── util.py
My setup.py:
import os
from setuptools import setup, find_packages
REQUIRED_PACKAGES = [
"h5py==2.9.0",
"numpy==1.16.4",
"opencv-python==4.1.0.25",
"pandas==0.24.2",
"keras==2.2.4",
"albumentations==0.3.1"
]
setup(
name='mypackage',
version='0.1',
install_requires=REQUIRED_PACKAGES,
packages=find_packages(include=["mypackage.*"]),
include_package_data=False
)
The code i want to package corresponds only to the mypackage directory. That's why i passed "mypackage.*" to find_packages and used include_package_data=False.
If i run:
python setup.py sdist
All project structure gets packaged in the resulting tar.gz file.
Anyone knowing how to just package modules inside mypackage/ and the top level README file? I'm not finding this in setuptools docs.
First thing to fix is
packages=find_packages(include=["mypackage"]),
But you also need to understand that sdist is mostly controlled by the files MANIFEST or its template MANIFEST.in, not setup.py. You can compare what is created with sdist and bdist_wheel or bdist_egg; content of bdist_* is controlled by setup.py.
So my advice is to create the following MANIFEST.in:
prune *
include README.txt
recursive-include mypackage *.py
Related
I'm migrating an old python project to the new pyproject.toml based system and am having trouble with getting files that are required by tests to install. Inside the pyproject.toml I have:
[tool.setuptools]
package-data = {"my_pkg_name" = ["tests/*.sdf", "tests/*.urdf", "tests/*.xml", "tests/meshes/*.obj"]}
[build-system]
requires = ["setuptools>=43.0.0", "wheel"]
build-backend = "setuptools.build_meta"
The tests that are run with pytest require the files described under package-data. After I build and install the build, the test files are not there. How do I get those files to be installed? How to include package data with setuptools/distutils? may be related, but things have changed, and I would rather not have to create a manifest file.
The project structure looks something like:
.
├── LICENSE.txt
├── pyproject.toml
├── README.md
├── src
│ ├── my_pkg_name
│ │ ├── __init__.py
└── tests
├── ant.xml
├── humanoid.xml
├── __init__.py
├── kuka_iiwa.urdf
├── meshes
│ ├── link_0.obj
│ ├── link_1.obj
│ ├── link_2.obj
│ ├── link_3.obj
│ ├── link_4.obj
│ ├── link_5.obj
│ ├── link_6.obj
│ └── link_7.obj
└── test_transform.py
The pyproject.toml has no specific package discovery related settings.
I'm trying to use the basic example from the docs. I am able to successfully generate an sdist, but I want to be able to install from wheel. However, when installing from wheel instead of from the sdist, I don't have a proj directory that I can import, only pkg1 and pkg2.
Starting from this directory:
├── proj
│ ├── __init__.py
│ ├── additional
│ │ └── __init__.py
│ ├── pkg1
│ │ └── __init__.py
│ └── pkg2
│ └── __init__.py
├── pyproject.toml
└── setup.cfg
I tried using this setup.cfg:
[metadata]
name = p1
version = 0.0.1
[options]
packages =
find:
package_dir =
=proj
[options.packages.find]
where = proj
include = pkg*
exclude = additional
To generate this file structure, which is successfully generated in the source distribution:
├── PKG-INFO
├── README.md
├── proj
│ ├── p1.egg-info
│ │ ├── PKG-INFO
│ │ ├── SOURCES.txt
│ │ ├── dependency_links.txt
│ │ └── top_level.txt
│ ├── pkg1
│ │ └── __init__.py
│ └── pkg2
│ └── __init__.py
├── pyproject.toml
└── setup.cfg
But the wheel has this structure:
├── p1-0.0.1.dist-info
│ ├── METADATA
│ ├── RECORD
│ ├── WHEEL
│ └── top_level.txt
├── pkg1
│ └── __init__.py
└── pkg2
└── __init__.py
This makes it impossible to use:
import proj
As there is no such module, only modules pkg1 and pkg2.
I made a repo for this here: https://github.com/spensmith/setuptools-find.
Steps to recreate:
Clone the base repository here: https://github.com/spensmith/setuptools-find
cd setuptools-find
python -m build
mkdir dist/mywheel
unzip dist/p1-0.0.1-py3-none-any.whl -d dist/mywheel
tree dist/mywheel
I must be missing something basic, any help would be so appreciated!
Thank you, Spencer.
My question was answered directly on the setuptools discussion forum. https://github.com/pypa/setuptools/discussions/3185#discussioncomment-2411330.
Effectively, I shouldn't be using
package_dir =
=proj
Because this means that all of my code is within this folder. Instead, I should remove it entirely, and then change my exclude to look like:
[options.packages.find]
exclude = proj.additional
Finally, yielding this file:
[metadata]
name = p1
version = 0.0.1
[options]
packages =
find:
[options.packages.find]
exclude = proj.additional
I have a single, large git repo with many different projects (not submodules). A few of these projects are Python projects for which I'd like to track versioning with Python's versioneer, others may be completely independent projects (say, in Haskell). i.e. The directory structure looks like this:
myrepository
├── .git
├── README.md
├── project1/
│ ├── project1/
│ │ ├── __init__.py
│ │ └── _version.py
│ ├── versioneer.py
│ ├── setup.cfg
│ └── setup.py
├── project2/
├── project3/
└── project4/
├── project4/
│ ├── __init__.py
│ └── _version.py
├── versioneer.py
├── setup.cfg
└── setup.py
This doesn't play well with versioneer because it can't discover the .git directory at the project root level, so I get a version of 0+unknown.
Questions:
Is there a suggested way to use versioneer with a single monolithic repo with multiple projects?
Depending on the answer to the above, is it recommended that my git tags read like: project1-2.1.0, project2-1.3.1, or should I unify the git tags like: 1.2.1, 1.2.2?
I read a lot about this problem but could not find any solution, so I'll ask yet another question about it, since I'm not even sure if I use the correct folder structure for my Python package.
So basically I'm developing an application which uses the Tornado web-server framework and I want to package it, so the users can install it via pip and get access to a basic script to start the web server.
The directory structure is the following:
├── MANIFEST.in
├── README.md
├── config
│ └── default.cfg
├── docs
│ ├── Makefile
│ ├── _build
│ ├── _static
│ ├── _templates
│ ├── conf.py
│ ├── index.rst
├── foopackage
│ ├── __init__.py
│ ├── barmodule.py
│ └── bazmodule.py
├── setup.py
├── static
│ ├── css
│ │ ├── menu.css
│ │ └── main.css
│ ├── img
│ │ └── logo.png
│ ├── js
│ │ ├── ui.js
│ │ └── navigation.js
│ └── lib
│ ├── d3.v3.min.js
│ └── jquery-1.11.0.min.js
└── templates
├── index.html
└── whatever.html
The Python code is as you can see in the package foopackage.
The MANIFEST.in file recursively includes the directories config, static, templates, and docs.
This is my setup.py (only the relevant parts:
from setuptools import setup
setup(name='foo',
version='0.1.0',
packages=['foopackage'],
include_package_data=True,
install_requires=[
'tornado>=3.2.2',
],
entry_points={
'console_scripts': [
'foo=foopackage.barmodule:main',
],
},
)
If I run python setup.py sdist, everything gets packaged nicely, the docs, templates and config files etc. are included. However, if I run pip install ..., only the foopackage gets installed and everything else is ignored.
How do I include those additional files to the install procedure? Is my directory structure OK? I also read about "faking a package", so putting everything in a directory and touch a __init__.py file, but that seems pretty odd to me :-\
I solved the problem by moving the static directory to the actual Python module directory (foopackage). It seems that top level "non-package" folders are ignored otherwise.
How to import my modules streaming_capture, streaming_information, report in in other project ?
I wrote a Python project called long_term_streaming_info_capture,
Then installed it in a new project under virtualenv environment.
If I want to import StreamingCapture I should called from long_term_streaming_info_capture.scripts.streaming_capture import StreamingCapture
Can I just use the long_term_streaming_info_capture.streaming_capture without the folder script in the import path ?
(develop+-)$ tree -L 3 -P "*.py"
.
├── helpers
│ ├── __init__.py
│ └── animation
│ ├── __init__.py
│ ├── animation_helper.py
│ ├── dqa_file_io.py
│ ├── dqa_telnet.py
│ ├── file_io_helper.py
│ ├── shm_controller.py
│ ├── telnet_helper.py
│ ├── test_dqa_file_io.py
│ └── test_dqa_telnet.py
├── log
├── main.py
├── report.py
├── sandbox
├── streaming_capture.py
└── streaming_information.py
Project skeleton
.
├── HACKING.txt
├── MANIFEST.in
├── NEWS.txt
├── README.rst
├── bootstrap.py
├── buildout.cfg
├── setup.py
└── src
├── long_term_streaming_info_capture
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── docs
│ ├── scripts
│ └── tests
└── long_term_streaming_info_capture.egg-info
├── PKG-INFO
├── SOURCES.txt
├── dependency_links.txt
├── entry_points.txt
├── not-zip-safe
└── top_level.txt
Here my setup.py
from setuptools import setup, find_packages
import sys, os
here = os.path.abspath(os.path.dirname(__file__))
README = open(os.path.join(here, 'README.rst')).read()
NEWS = open(os.path.join(here, 'NEWS.txt')).read()
version = '0.1'
install_requires = [
# List your project dependencies here.
# For more details, see:
# http://packages.python.org/distribute/setuptools.html#declaring-dependencies
]
setup(name='long_term_streaming_info_capture',
version=version,
description="for capturing fps framerate",
long_description=README + '\n\n' + NEWS,
classifiers=[
# Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers
],
keywords='',
license='',
packages=find_packages('src'),
package_dir = {'': 'src'},include_package_data=True,
zip_safe=False,
install_requires=install_requires,
entry_points={
'console_scripts':
['long_term_streaming_info_capture=long_term_streaming_info_capture:main']
}
)