We Keep Coding

PyPI - folder inside the package folder not getting uploaded [duplicate]

When using setuptools, I can not get the installer to pull in any package_data files. Everything I've read says that the following is the correct way to do it. Can someone please advise?
setup(
name='myapp',
packages=find_packages(),
package_data={
'myapp': ['data/*.txt'],
},
include_package_data=True,
zip_safe=False,
install_requires=['distribute'],
)
where myapp/data/ is the location of the data files.

I realize that this is an old question, but for people finding their way here via Google: package_data is a low-down, dirty lie. It is only used when building binary packages (python setup.py bdist ...) but not when building source packages (python setup.py sdist ...). This is, of course, ridiculous -- one would expect that building a source distribution would result in a collection of files that could be sent to someone else to built the binary distribution.
In any case, using MANIFEST.in will work both for binary and for source distributions.

I just had this same issue. The solution, was simply to remove include_package_data=True.
After reading here, I realized that include_package_data aims to include files from version control, as opposed to merely "include package data" as the name implies. From the docs:
The data files [of include_package_data] must be under CVS or Subversion control
...
If you want finer-grained control over what files are included (for example, if
you have documentation files in your package directories and want to exclude
them from installation), then you can also use the package_data keyword.
Taking that argument out fixed it, which is coincidentally why it also worked when you switched to distutils, since it doesn't take that argument.

Following #Joe 's recommendation to remove the include_package_data=True line also worked for me.
To elaborate a bit more, I have no MANIFEST.in file. I use Git and not CVS.
Repository takes this kind of shape:
/myrepo
- .git/
- setup.py
- myproject
- __init__.py
- some_mod
- __init__.py
- animals.py
- rocks.py
- config
- __init__.py
- settings.py
- other_settings.special
- cool.huh
- other_settings.xml
- words
- __init__.py
word_set.txt
setup.py:
from setuptools import setup, find_packages
import os.path
setup (
name='myproject',
version = "4.19",
packages = find_packages(),
# package_dir={'mypkg': 'src/mypkg'}, # didnt use this.
package_data = {
# If any package contains *.txt or *.rst files, include them:
'': ['*.txt', '*.xml', '*.special', '*.huh'],
},
#
# Oddly enough, include_package_data=True prevented package_data from working.
# include_package_data=True, # Commented out.
data_files=[
# ('bitmaps', ['bm/b1.gif', 'bm/b2.gif']),
('/opt/local/myproject/etc', ['myproject/config/settings.py', 'myproject/config/other_settings.special']),
('/opt/local/myproject/etc', [os.path.join('myproject/config', 'cool.huh')]),
#
('/opt/local/myproject/etc', [os.path.join('myproject/config', 'other_settings.xml')]),
('/opt/local/myproject/data', [os.path.join('myproject/words', 'word_set.txt')]),
],
install_requires=[ 'jsonschema',
'logging', ],
entry_points = {
'console_scripts': [
# Blah...
], },
)
I run python setup.py sdist for a source distrib (haven't tried binary).
And when inside of a brand new virtual environment, I have a myproject-4.19.tar.gz, file,
and I use
(venv) pip install ~/myproject-4.19.tar.gz
...
And other than everything getting installed to my virtual environment's site-packages, those special data files get installed to /opt/local/myproject/data and /opt/local/myproject/etc.

include_package_data=True worked for me.
If you use git, remember to include setuptools-git in install_requires. Far less boring than having a Manifest or including all path in package_data ( in my case it's a django app with all kind of statics )
( pasted the comment I made, as k3-rnc mentioned it's actually helpful as is )

Using setup.cfg (setuptools ≥ 30.3.0)
Starting with setuptools 30.3.0 (released 2016-12-08), you can keep your setup.py very small and move the configuration to a setup.cfg file. With this approach, you could put your package data in an [options.package_data] section:
[options.package_data]
* = *.txt, *.rst
hello = *.msg
In this case, your setup.py can be as short as:
from setuptools import setup
setup()
For more information, see configuring setup using setup.cfg files.
There is some talk of deprecating setup.cfg in favour of pyproject.toml as proposed in PEP 518, but this is still provisional as of 2020-02-21.

Update: This answer is old and the information is no longer valid. All setup.py configs should use import setuptools. I've added a more complete answer at https://stackoverflow.com/a/49501350/64313
I solved this by switching to distutils. Looks like distribute is deprecated and/or broken.
from distutils.core import setup
setup(
name='myapp',
packages=['myapp'],
package_data={
'myapp': ['data/*.txt'],
},
)

I had the same problem for a couple of days but even this thread wasn't able to help me as everything was confusing. So I did my research and found the following solution:
Basically in this case, you should do:
from setuptools import setup
setup(
name='myapp',
packages=['myapp'],
package_dir={'myapp':'myapp'}, # the one line where all the magic happens
package_data={
'myapp': ['data/*.txt'],
},
)
The full other stackoverflow answer here

I found this post while stuck on the same problem.
My experience contradicts the experiences in the other answers.
include_package_data=True does include the data in the
bdist! The explanation in the setuptools
documentation
lacks context and troubleshooting tips, but
include_package_data works as advertised.
My setup:
Windows / Cygwin
git version 2.21.0
Python 3.8.1 Windows distribution
setuptools v47.3.1
check-manifest v0.42
Here is my how-to guide.
How-to include package data
Here is the file structure for a project I published on PyPI.
(It installs the application in __main__.py).
├── LICENSE.md
├── MANIFEST.in
├── my_package
│ ├── __init__.py
│ ├── __main__.py
│ └── _my_data <---- folder with data
│ ├── consola.ttf <---- data file
│ └── icon.png <---- data file
├── README.md
└── setup.py
Starting point
Here is a generic starting point for the setuptools.setup() in
setup.py.
setuptools.setup(
...
packages=setuptools.find_packages(),
...
)
setuptools.find_packages() includes all of my packages in the
distribution. My only package is my_package.
The sub-folder with my data, _my_data, is not considered a
package by Python because it does not contain an __init__.py,
and so find_packages() does not find it.
A solution often-cited, but incorrect, is to put an empty
__init__.py file in the _my_data folder.
This does make it a package, so it does include the folder
_my_data in the distribution. But the data files inside
_my_data are not included.
So making _my_data into a package does not help.
The solution is:
the sdist already contains the data files
add include_package_data=True to include the data files in the bdist as well
Experiment (how to test the solution)
There are three steps to make this a repeatable experiment:
$ rm -fr build/ dist/ my_package.egg-info/
$ check-manifest
$ python setup.py sdist bdist_wheel
I will break these down step-by-step:
Clean out the old build:
$ rm -fr build/ dist/ my_package.egg-info/
Run check-manifest to be sure MANIFEST.in matches the
Git index of files under version control:
$ check-manifest
If MANIFEST.in does not exist yet, create it from the Git
index of files under version control:
$ check-manifest --create
Here is the MANIFEST.in that is created:
include *.md
recursive-include my_package *.png
recursive-include my_package *.ttf
There is no reason to manually edit this file.
As long as everything that should be under version control is
under version control (i.e., is part of the Git index),
check-manifest --create does the right thing.
Note: files are not part of the Git index if they are either:
ignored in a .gitignore
excluded in a .git/info/exclude
or simply new files that have not been added to the index yet
And if any files are under version control that should not be
under version control, check-manifest issues a warning and
specifies which files it recommends removing from the Git index.
Build:
$ python setup.py sdist bdist_wheel
Now inspect the sdist (source distribution) and bdist_wheel
(build distribution) to see if they include the data files.
Look at the contents of the sdist (only the relevant lines are
shown below):
$ tar --list -f dist/my_package-0.0.1a6.tar.gz
my_package-0.0.1a6/
...
my_package-0.0.1a6/my_package/__init__.py
my_package-0.0.1a6/my_package/__main__.py
my_package-0.0.1a6/my_package/_my_data/
my_package-0.0.1a6/my_package/_my_data/consola.ttf <-- yay!
my_package-0.0.1a6/my_package/_my_data/icon.png <-- yay!
...
So the sdist already includes the data files because they are
listed in MANIFEST.in. There is nothing extra to do to include
the data files in the sdist.
Look at the contents of the bdist (it is a .zip file, parsed
with zipfile.ZipFile):
$ python check-whl.py
my_package/__init__.py
my_package/__main__.py
my_package-0.0.1a6.dist-info/LICENSE.md
my_package-0.0.1a6.dist-info/METADATA
my_package-0.0.1a6.dist-info/WHEEL
my_package-0.0.1a6.dist-info/entry_points.txt
my_package-0.0.1a6.dist-info/top_level.txt
my_package-0.0.1a6.dist-info/RECORD
Note: you need to create your own check-whl.py script to produce the
above output. It is just three lines:
from zipfile import ZipFile
path = "dist/my_package-0.0.1a6-py3-none-any.whl" # <-- CHANGE
print('\n'.join(ZipFile(path).namelist()))
As expected, the bdist is missing the data files.
The _my_data folder is completely missing.
What if I create a _my_data/__init__.py? I repeat the
experiment and I find the data files are still not there! The
_my_data/ folder is included but it does not contain the data
files!
Solution
Contrary to the experience of others, this does work:
setuptools.setup(
...
packages=setuptools.find_packages(),
include_package_data=True, # <-- adds data files to bdist
...
)
With the fix in place, redo the experiment:
$ rm -fr build/ dist/ my_package.egg-info/
$ check-manifest
$ python.exe setup.py sdist bdist_wheel
Make sure the sdist still has the data files:
$ tar --list -f dist/my_package-0.0.1a6.tar.gz
my_package-0.0.1a6/
...
my_package-0.0.1a6/my_package/__init__.py
my_package-0.0.1a6/my_package/__main__.py
my_package-0.0.1a6/my_package/_my_data/
my_package-0.0.1a6/my_package/_my_data/consola.ttf <-- yay!
my_package-0.0.1a6/my_package/_my_data/icon.png <-- yay!
...
Look at the contents of the bdist:
$ python check-whl.py
my_package/__init__.py
my_package/__main__.py
my_package/_my_data/consola.ttf <--- yay!
my_package/_my_data/icon.png <--- yay!
my_package-0.0.1a6.dist-info/LICENSE.md
my_package-0.0.1a6.dist-info/METADATA
my_package-0.0.1a6.dist-info/WHEEL
my_package-0.0.1a6.dist-info/entry_points.txt
my_package-0.0.1a6.dist-info/top_level.txt
my_package-0.0.1a6.dist-info/RECORD
How not to test if data files are included
I recommend troubleshooting/testing using the approach outlined
above to inspect the sdist and bdist.
pip install in editable mode is not a valid test
Note: pip install -e . does not show if data files are
included in the bdist.
The symbolic link causes the installation to behave as if the
data files are included (because they already exist locally on
the developer's computer).
After pip install my_package, the data files are in the
virtual environment's lib/site-packages/my_package/ folder,
using the exact same file structure shown above in the list of
the whl contents.
Publishing to TestPyPI is a slow way to test
Publishing to TestPyPI and then installing and looking in
lib/site-packages/my_packages is a valid test, but it is too
time-consuming.

Ancient question and yet... package management of python really leaves a lot to be desired. So I had the use case of installing using pip locally to a specified directory and was surprised both package_data and data_files paths did not work out. I was not keen on adding yet another file to the repo so I ended up leveraging data_files and setup.py option --install-data; something like this
pip install . --install-option="--install-data=$PWD/package" -t package

Moving the folder containing the package data into to module folder solved the problem for me.
See this question: MANIFEST.in ignored on "python setup.py install" - no data files installed?

Just remove the line:
include_package_data=True,
from your setup script, and it will work fine. (Tested just now with latest setuptools.)

Like others in this thread, I'm more than a little surprised at the combination of longevity and still a lack of clarity, BUT the best answer for me was using check-manifest as recommended in the answer from #mike-gazes
So, using just a setup.cfg and no setup.py and additional text and python files required in the package, what worked for me was keeping this in setup.cfg:
[options]
packages = find:
include_package_data = true
and updating the MANIFEST.in based on the check-manifest output:
include *.in
include *.txt
include *.yml
include LICENSE
include tox.ini
recursive-include mypkg *.py
recursive-include mypkg *.txt

For a directory structure like:
foo/
├── foo
│   ├── __init__.py
│   ├── a.py
│   └── data.txt
└── setup.py
and setup.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from setuptools import setup
NAME = 'foo'
DESCRIPTION = 'Test library to check how setuptools works'
URL = 'https://none.com'
EMAIL = 'gzorp#bzorp.com'
AUTHOR = 'KT'
REQUIRES_PYTHON = '>=3.6.0'
setup(
name=NAME,
version='0.0.0',
description=DESCRIPTION,
author=AUTHOR,
author_email=EMAIL,
python_requires=REQUIRES_PYTHON,
url=URL,
license='MIT',
classifiers=[
'Programming Language :: Python',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6',
],
packages=['foo'],
package_data={'foo': ['data.txt']},
include_package_data=True,
install_requires=[],
extras_require={},
cmdclass={},
)
python setup.py bdist_wheel works.

Starting with Setuptools 62.3.0, you can now use recursive wildcards ("**") to include a (sub)directory recursively. This way you can include whole folders with all their folders and files in it.
For example, when using a pyproject.toml file, this is how you include two folders recursively:
[tool.setuptools.package-data]
"ema_workbench.examples.data" = ["**"]
"ema_workbench.examples.models" = ["**"]
But you can also only include certain file-types, in a folder and all subfolders. If you want to include all markdown (.md) files for example:
[tool.setuptools.package-data]
"ema_workbench.examples.data" = ["**/*.md"]
It should also work when using setup.py or setup.cfg.
See https://github.com/pypa/setuptools/pull/3309 for the details.

Single module turns into a package on PyPi when using Python 3 setuptools

I'm trying to change a python module that I have earlier published on PyPi. I have single python file called gaiaclient.py which has class called Client.
If I install the old version with pip install gaiaclient==0.0.9 everything works as it should. I can call it like this:
>>> import gaiaclient
>>> myclient = gaiaclient.Client("http://localhost:1234")
pip show -f gaiaclient shows that it installed these files:
Files:
__pycache__/gaiaclient.cpython-36.pyc
gaiaclient-0.0.9.dist-info/INSTALLER
gaiaclient-0.0.9.dist-info/METADATA
gaiaclient-0.0.9.dist-info/RECORD
gaiaclient-0.0.9.dist-info/WHEEL
gaiaclient-0.0.9.dist-info/top_level.txt
gaiaclient.py
As you see there is just gaiaclient.py module on top level and no __init__.py file.
Now I would like to publish new version of the gaiaclient but I just can't get it working same way it was! After my trials I'm on version 0.0.15 now.
My setup.py looks like this:
import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="gaiaclient",
version="0.0.15",
license='MIT License',
author="JOT Automation Ltd.",
author_email="rami.rahikkala#jotautomation.com",
description="Client for JOT Automation gaia machines",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/jotautomation/gaiapythonclient",
py_modules=['gaiaclient'],
install_requires=['requests'],
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
)
(The whole project can be found from github)
I have tried to install the package I generate with command python setup.py sdist bdist_wheel locally with command pip install dist/gaiaclient-0.0.15.tar.gz and then it still works as with the version 0.0.9. When the same package is published on PyPi (twine upload dist/*) things changes.
When I install it from pypi pip install gaiaclient==0.0.15 it installs also gaiaclient/init.py and gaiaclient/gaiaclient.py in addition to toplevel gaiaclient.py! I.e. it is a package now. How can I make it stop to include the extra files? Because now I cannot anymore just import gaiaclient but I have to do it like this from gaiaclient import gaiaclient which I would not like to do.
File listing with 0.0.15 version installed from PyPi:
Files:
__pycache__/gaiaclient.cpython-36.pyc
gaiaclient-0.0.15.dist-info/INSTALLER
gaiaclient-0.0.15.dist-info/LICENSE
gaiaclient-0.0.15.dist-info/METADATA
gaiaclient-0.0.15.dist-info/RECORD
gaiaclient-0.0.15.dist-info/WHEEL
gaiaclient-0.0.15.dist-info/top_level.txt
gaiaclient.py
gaiaclient/__init__.py
gaiaclient/__pycache__/__init__.cpython-36.pyc
gaiaclient/__pycache__/gaiaclient.cpython-36.pyc
gaiaclient/gaiaclient.py
Edit 1:
I tried the approach Score_Under suggested i.e. added __init__.py with from gaiaclient.gaiaclient import Client. This doesn't change anything. This approach probably would need also change to setup.py because now the gaiaclient/__init__.py is different than what I specified.
Okay, one more try. This time I added packages=['.'], to setup.py. Now it actually works. But! Look at the file listing now:
Files:
__init__.py
__pycache__/__init__.cpython-36.pyc
__pycache__/example.cpython-36.pyc
__pycache__/example_gcode.cpython-36.pyc
__pycache__/gaiaclient.cpython-36.pyc
example.py
example_gcode.py
gaiaclient-0.0.17.dist-info/INSTALLER
gaiaclient-0.0.17.dist-info/METADATA
gaiaclient-0.0.17.dist-info/RECORD
gaiaclient-0.0.17.dist-info/WHEEL
gaiaclient-0.0.17.dist-info/top_level.txt
gaiaclient.py
__init__.py is now on the root level i.e. on the site-packages directory. This is not good. And still, some how it was working before without __init__.py.
Edit 2:
Now I found out how I got it working before. I did use python 2 to run python setup.py sdist bdist_wheel.
Question still remains that how on earth I can publish single module using python 3 setuptools? Is the behaviour I see because of implicit namespace introduced in python 3.3?

Excluding source files from built rpm distribution with setuptool

I have a typical project structure that looks as follows:
EngineEmulator
src
ship
engine
emulator
mapping
tests
emulator
mapping
utils
common
doc
....
tools
....
setup.py
MANIFEST.in
setup.cfg
README.rst
My setup.py looks as follows:
from setuptools import setup, find_packages
setup(
name='Engine',
version=1.0.0,
description='Engine Project',
package_dir={'': 'src'},
packages=find_packages(
'src',
exclude=["*.tests", "*.tests.*", "tests.*", "tests"]),
install_requires =['pycrypto',
'kombu >=1.1.3'],
author='Demo',
author_email='demo#eliza.net'
license='MIT',
classifiers=[
'Topic :: Demo Engine',
'Development Status:: 3 - Iteration',
'Programming Language :: Python -2.6'
]
)
My setup.cfg looks as follows:
[egg_info]
tag_build = .dev
tag_svn_revision = 1
[rotate]
#keep last 15 eggs, clean up order
match = .egg
keep = 15
And My MANIFEST.in looks as follows:
include README.rst
recursive-include src/ship/Engine
prune src/utils
prune src/ship/tests
prune tools/
When I run python setup.py bdist_egg and python setup.py bdist_rpm I get the egg file and two rpm files generated (noarch.rpm and src.rpm).
In my destination machine when I run easy_install <generated egg file> my eg.info file gets copied over but the source files don't get copied over to /usr/lib/python2.6/site-packages. I was expecting I would have a directory called Engine.
Can anybody point out what I am doing wrong? Thanks in advance.

Try to keep things as simple as possible.
Quick check with sdist
Try this:
$ python setup.py sdist
It shall create source distribution file for your package.
It is in zip format, so unpack it and check, if there are all expected files inside present.
If not, you have to find the reason, why expected files are missing in your distribution.
Checking things step by step (and simplifying)
Do you use .py extension?
May be stupid question, but in your file listing I do not see any py files inside of src tree.
In case you have there just files without .py extension, find_packages will not find anything.
Where do you have your __init__.py files located?
Let us know, where the files are:
$ cd src
$ find . -name "*.py"
If you miss __init__.py, find_packages will not find whole package.
Remove utils package
Why do you have it there?
Better have it installed out of your source code you develop or move it subdirectory in your project
root.
This will render prune src/utils unnecessary in your MANIFEST.in.
Put into MANIFEST.in only what must be there
If you read doc for MANIFEST.in, it states, what files are included automatically (all, what
mentioned in arguments of setup function, so in your case all python source files returned by
find_packages).
For this reason, you shall remove recursive-include src/shop/Engine as it shall be already
included by setup call.
Remove prune lines.
src/utils shall not be in your source tree - it is just messing things up.
tools is not to be included, so there is no need to prune it.
src/ship/tests can be there, it will not harm, if you keep these files in the destribution.
Assert, what packages were found
Make sure, your setup get proper names for packages.
For this purpuse, you can call find_package sooner and assert it containts, what you expect.
(temporarily) remove setup.cfg
Just to keep things simpler.
Proposed project reorganization
You shall have file structure in similar manner as follows:
src/ship/__init__.py
src/ship/engine/__init__.py
src/ship/engine/emulator/__init__.py
src/ship/engine/emulator/module.py
src/ship/engine/emulator/module2.py
src/ship/engine/mapping/other.py
src/ship/engine/mapping/another.py
src/ship/tests/__init__.py
src/ship/tests/emulator/__init__.py
src/ship/tests/emulator/test_module.py
src/ship/tests/emulator/test_module2.py
src/ship/tests/mapping/__init__.py
src/ship/tests/mapping/test_other.py
src/ship/tests/mapping/test_another.py
doc
doc/index.rst
tools
tools/knife.py
setup.py
MANIFEST.in
README.rst
setup.py
from setuptools import setup, find_packages
packages=find_packages("src")
assert "ship.engine" in packages
assert "ship.engine.emulator" in packages
assert "ship.engine.mapping" in packages
#etc
install_requires =['pycrypto', 'kombu>=1.1.3'] #watch the spaces around `>=`, shall not be there
setup(
name="Engine",
package_dir={'': 'src'},
packages=packages,
install_requires=install_requires
)
MANIFEST.in
include README.rst
Conclusions
It might happen, that running
$ python setup.py sdist
would fail on asserts. This is sign, some of expected files are missing. Check that.
After you make your project living in simple way, you might add more details around (and do it step
by step to be sure, you do not break something).

packaging with numpy and test suite

Introduction
Disclaimer: I'm very new to python packaging with distutils. So far I've just stashed everything into modules, and packages manually and developed on top of that. I never wrote a setup.py file before.
I have a Fortran module that I want to use in my python code with numpy. I figured the best way to do that would be f2py, since it is included in numpy. To automate the build process I want to use distutils and the corresponding numpy enhancement, which includes convenience functions for f2py wrappers.
I do not understand how I should organize my files, and how to include my test suite.
What I want is the possibility to use ./setup.py for building, installing, and testing, and developing.
My directory structure looks as follows:
volterra
├── setup.py
└── volterra
├── __init__.py
├── integral.f90
├── test
│   ├── __init__.py
│   └── test_volterra.py
└── volterra.f90
And the setup.py file contains this:
def configuration(parent_package='', top_path=None):
from numpy.distutils.misc_util import Configuration
config = Configuration('volterra', parent_package, top_path)
config.add_extension('_volterra',
sources=['volterra/integral.f90', 'volterra/volterra.f90'])
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(**configuration(top_path='').todict())
After running ./setup.py build I get.
build/lib.linux-x86_64-2.7/
└── volterra
└── _volterra.so
Which includes neither the __init__.py file, nor the tests.
Questions
Is it really necessary to add the path to every single source file of the extension? (I.e. volterra/integral.f90) Can't I give a parameter which says, look for stuff in volterra/? The top_path, and package_dir parameters didn't do the trick.
Currently, the __init__.py file is not included in the build. Why is that?
How can I run my tests in this setup?
What's the best workflow for doing development in such an environment? I don't want to install my package for every single change I do. How do you do development in the source directory when you need to compile some extension modules?

Here is a setup.py that works for me:
# pkg - A fancy software package
# Copyright (C) 2013 author (email)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see http://www.gnu.org/licenses/gpl.html.
"""pkg: a software suite for
Hey look at me I'm a long description
But how long am I?
"""
from __future__ import division, print_function
#ideas for setup/f2py came from:
# -numpy setup.py: https://github.com/numpy/numpy/blob/master/setup.py 2013-11-07
# -winpython setup.py: http://code.google.com/p/winpython/source/browse/setup.py 2013-11-07
# -needing to use
# import setuptools; from numpy.distutils.core import setup, Extension:
# http://comments.gmane.org/gmane.comp.python.f2py.user/707 2013-11-07
# -wrapping FORTRAN code with f2py: http://www2-pcmdi.llnl.gov/cdat/tutorials/f2py-wrapping-fortran-code 2013-11-07
# -numpy disutils: http://docs.scipy.org/doc/numpy/reference/distutils.html 2013-11-07
# -manifest files in disutils:
# 'distutils doesn't properly update MANIFEST. when the contents of directories change.'
# https://github.com/numpy/numpy/blob/master/setup.py
# -if things are not woring try deleting build, sdist, egg directories and try again:
# https://stackoverflow.com/a/9982133/2530083 2013-11-07
# -getting fortran extensions to be installed in their appropriate sub package
# i.e. "my_ext = Extension(name = 'my_pack._fortran', sources = ['my_pack/code.f90'])"
# Note that sources is a list even if one file:
# http://numpy-discussion.10968.n7.nabble.com/f2py-and-setup-py-how-can-I-specify-where-the-so-file-goes-tp34490p34497.html 2013-11-07
# -install fortran source files into their appropriate sub-package
# i.e. "package_data={'': ['*.f95','*.f90']}# Note it's a dict and list":
# https://stackoverflow.com/a/19373744/2530083 2013-11-07
# -Chapter 9 Fortran Programming with NumPy Arrays:
# Langtangen, Hans Petter. 2013. Python Scripting for Computational Science. 3rd edition. Springer.
# -Hitchhikers guide to packaging :
# http://guide.python-distribute.org/
# -Python Packaging: Hate, hate, hate everywhere :
# http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
# -How To Package Your Python Code:
# http://www.scotttorborg.com/python-packaging/
# -install testing requirements:
# https://stackoverflow.com/a/7747140/2530083 2013-11-07
import setuptools
from numpy.distutils.core import setup, Extension
import os
import os.path as osp
def readme(filename='README.rst'):
with open('README.rst') as f:
text=f.read()
f.close()
return text
def get_package_data(name, extlist):
"""Return data files for package *name* with extensions in *extlist*"""
#modified slightly from taken from http://code.google.com/p/winpython/source/browse/setup.py 2013-11-7
flist = []
# Workaround to replace os.path.relpath (not available until Python 2.6):
offset = len(name)+len(os.pathsep)
for dirpath, _dirnames, filenames in os.walk(name):
for fname in filenames:
if not fname.startswith('.') and osp.splitext(fname)[1] in extlist:
# flist.append(osp.join(dirpath, fname[offset:]))
flist.append(osp.join(dirpath, fname))
return flist
DOCLINES = __doc__.split("\n")
CLASSIFIERS = """\
Development Status :: 1 - Planning
License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)
Programming Language :: Python :: 2.7
Topic :: Scientific/Engineering
"""
NAME = 'pkg'
MAINTAINER = "me"
MAINTAINER_EMAIL = "me#me.com"
DESCRIPTION = DOCLINES[0]
LONG_DESCRIPTION = "\n".join(DOCLINES[2:])#readme('readme.rst')
URL = "http://meeeee.mmemem"
DOWNLOAD_URL = "https://github.com/rtrwalker/geotecha.git"
LICENSE = 'GNU General Public License v3 or later (GPLv3+)'
CLASSIFIERS = [_f for _f in CLASSIFIERS.split('\n') if _f]
KEYWORDS=''
AUTHOR = "me"
AUTHOR_EMAIL = "me.com"
PLATFORMS = ["Windows"]#, "Linux", "Solaris", "Mac OS-X", "Unix"]
MAJOR = 0
MINOR = 1
MICRO = 0
ISRELEASED = False
VERSION = '%d.%d.%d' % (MAJOR, MINOR, MICRO)
INSTALL_REQUIRES=[]
ZIP_SAFE=False
TEST_SUITE='nose.collector'
TESTS_REQUIRE=['nose']
DATA_FILES = [(NAME, ['LICENSE.txt','README.rst'])]
PACKAGES=setuptools.find_packages()
PACKAGES.remove('tools')
PACKAGE_DATA={'': ['*.f95','*f90']}
ext_files = get_package_data(NAME,['.f90', '.f95','.F90', '.F95'])
ext_module_names = ['.'.join(osp.splitext(v)[0].split(osp.sep)) for v in ext_files]
EXT_MODULES = [Extension(name=x,sources=[y]) for x, y in zip(ext_module_names, ext_files)]
setup(
name=NAME,
version=VERSION,
maintainer=MAINTAINER,
maintainer_email=MAINTAINER_EMAIL,
description=DESCRIPTION,
long_description=LONG_DESCRIPTION,
url=URL,
download_url=DOWNLOAD_URL,
license=LICENSE,
classifiers=CLASSIFIERS,
author=AUTHOR,
author_email=AUTHOR_EMAIL,
platforms=PLATFORMS,
packages=PACKAGES,
data_files=DATA_FILES,
install_requires=INSTALL_REQUIRES,
zip_safe=ZIP_SAFE,
test_suite=TEST_SUITE,
tests_require=TESTS_REQUIRE,
package_data=PACKAGE_DATA,
ext_modules=EXT_MODULES,
)
To install, at the command line I use:
python setup.py install
python setup.py clean --all
The only issue I seem to have is a minor one. when I look in site-packages for my package it is installed inside the egg folder C:\Python27\Lib\site-packages\pkg-0.1.0-py2.7-win32.egg\pkg. Most other packages I see there have a C:\Python27\Lib\site-packages\pkg folder separate to the egg folder. Does anyone know how to get that separation?
As for testing, after installing, I type the following at the command line:
nosetests package_name -v
Try investigating python setup.py develop (Python setup.py develop vs install) for not having to install the package after every change.
As I commented in the code I found the following useful:
numpy setup.py: https://github.com/numpy/numpy/blob/master/setup.py 2013-11-07
winpython setup.py: http://code.google.com/p/winpython/source/browse/setup.py 2013-11-07
needing to use
import setuptools; from numpy.distutils.core import setup, Extension:
http://comments.gmane.org/gmane.comp.python.f2py.user/707 2013-11-07
wrapping FORTRAN code with f2py: http://www2-pcmdi.llnl.gov/cdat/tutorials/f2py-wrapping-fortran-code 2013-11-07
numpy disutils: http://docs.scipy.org/doc/numpy/reference/distutils.html 2013-11-07
manifest files in disutils:
'distutils doesn't properly update MANIFEST. when the contents of directories change.'
https://github.com/numpy/numpy/blob/master/setup.py
if things are not woring try deleting build, sdist, egg directories and try again:
https://stackoverflow.com/a/9982133/2530083 2013-11-07
getting fortran extensions to be installed in their appropriate sub package
i.e. "my_ext = Extension(name = 'my_pack._fortran', sources = ['my_pack/code.f90'])"
Note that sources is a list even if one file:
http://numpy-discussion.10968.n7.nabble.com/f2py-and-setup-py-how-can-I-specify-where-the-so-file-goes-tp34490p34497.html 2013-11-07
install fortran source files into their appropriate sub-package
i.e. "package_data={'': ['.f95','.f90']}# Note it's a dict and list":
https://stackoverflow.com/a/19373744/2530083 2013-11-07
Chapter 9 Fortran Programming with NumPy Arrays:
Langtangen, Hans Petter. 2013. Python Scripting for Computational Science. 3rd edition. Springer.
Hitchhikers guide to packaging :
http://guide.python-distribute.org/
Python Packaging: Hate, hate, hate everywhere :
http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
How To Package Your Python Code:
http://www.scotttorborg.com/python-packaging/
install testing requirements:
https://stackoverflow.com/a/7747140/2530083 2013-11-07
'python setup.py develop' :
https://stackoverflow.com/a/19048754/2530083

Here is setup.py from a project I made. I have found figuring out setup.py / packaging to be frustrating with no solid answers and definitely not pythonic in the sense of having one and only one obvious way to do something. Hopefully this will help a little.
The points you may find useful are:
find_packages which removes the drudgery of including lots of files or messing around with generating manifest.
package_data which allows you to easily specify non .py files to be included
install_requires / tests_require
You'll need to find the source for distribute_setup.py if you don't have it already.
Is it really necessary to add the path to every single source file of
the extension? (I.e. volterra/integral.f90) Can't I give a parameter
which says, look for stuff in volterra/? The top_path, and package_dir
parameters didn't do the trick.
Currently, the init.py file is not
included in the build. Why is that?
Hopefully find_packages() will solve both of those. I don't have much experience packaging but I haven't had to go back to manual inclusion yet.
How can I run my tests in this
setup?
I think this is probably a different question with many answers depending on how you are doing tests. Maybe you can ask it separately?
As a side note, I am under the impression that the standard is to put your tests directory at the top level. I.e. volterra/volterra and volterra/tests.
What's the best workflow for doing development in such an
environment? I don't want to install my package for every single
change I do. How do you do development in the source directory when
you need to compile some extension modules?
This might be worth another question as well. I don't see why you would need to install your package for every single change. If you are uploading the package, just don't install it on your dev system (except to test installation) and work directly from your development copy. Maybe I'm missing something though since I don't work with compiled extensions.
Here is the example
try:
from setuptools import setup, find_packages
except ImportError:
from distribute_setup import use_setuptools
use_setuptools()
from setuptools import setup, find_packages
setup(
# ... other stuff
py_modules=['distribute_setup'],
packages=find_packages(),
package_data={'': ['*.png']}, # for me to include anything with png
install_requires=['numpy', 'treenode', 'investigators'],
tests_require=['mock', 'numpy', 'treenode', 'investigators'],
)

How to include license file in setup.py script?

I have written a Python extension module in C++.
I plan to distribute the module with setuptools.
There will be binary distributions for 32- and 64-bit Windows (built with setup.py bdist_egg) and a source distribution for UNIX-like platforms (built with setup.py sdist).
I plan to license the module under the BSD license.
In my source tree, the file LICENSE.txt is in the top folder along with setup.py.
How should I include it in the installation package?
I tried the following setup.py script:
from setuptools import setup, Extension
from glob import glob
setup(
name = 'Foo',
version = '0.1.0',
ext_modules = [Extension('Foo', glob('Source/*.cpp'))],
package_data = {'': ['LICENSE.txt']}
)
It did not work, the license file is not included in the installation package.
Maybe because the setup.py file does not define any packages,
only a single extension module.
How do I fix this?

Write a setup.cfg file and in there specify:
[metadata]
license_files = LICENSE.txt
For this to work it seems like wheel is required to be installed. That is:
pip install wheel
If you have wheel already installed and it doesn't work, try to update it:
pip install --upgrade wheel
Then when installing the package via pip install <path> the LICENSE file gets included.

Since setuptools 42.0.0 you can use the license_files key to specify a list of license files to be included into a distribution. Since version 56.0.0 it supports pattern matching and defaults to ('LICEN[CS]E*', 'COPYING*', 'NOTICE*', 'AUTHORS*').
Note that due to implementation details there's actually no need to put this key into setup.cfg file (as another answer suggests). You could supply it as an argument to setup() function instead:
(documentation was unclear on this at the time of writing)
from setuptools import setup
setup(
...
license_files = ('LICENSE.txt',),
...
)
Also note that while these files will be included in both binary (wheel) and source distributions, they won't be installed with your package from setup.py-style source distribution if the user doesn't have a wheel package installed!
To ensure the license files will be installed along with your package you need to make some additional modifications to your setup script:
from setuptools import setup
from setuptools.command.egg_info import egg_info
class egg_info_ex(egg_info):
"""Includes license file into `.egg-info` folder."""
def run(self):
# don't duplicate license into `.egg-info` when building a distribution
if not self.distribution.have_run.get('install', True):
# `install` command is in progress, copy license
self.mkpath(self.egg_info)
self.copy_file('LICENSE.txt', self.egg_info)
egg_info.run(self)
setup(
...
license_files = ('LICENSE.txt',),
cmdclass = {'egg_info': egg_info_ex},
...
)
If your project is a pyproject.toml-style project and you think it will be installed by PEP 517-compatible frontend (e.g. pip>=19), a wheel will be forcibly built from your sources and the license files will be installed into .dist-info folder automatically.
Since version 61.0.0 you could specify project metadata and other configuration options in pyproject.toml file instead.

Using a METADATA.in file, the license can be included both the source package and wheels automatically:
METADATA.in
include README.md
include COPYING
Check out an example here:
https://github.com/node40/smsh

New setuptools (40.x) allows metadata, including license, to be stored in the setup.cfg's "metadata" section. If you use older setuptools you could provide license using the "license" named argument in your setup():
def read_text(file_name: str):
return open(os.path.join(base_path, file_name)).read()
setup(
name = 'Foo',
version = '0.1.0',
ext_modules = [Extension('Foo', glob('Source/*.cpp'))],
# package_data = {'': ['LICENSE.txt']}
license=read_text("LICENSE.txt")
)

You have to move the LICENSE.txt file into the package directory for your project. It cannot reside the top level. Python directories get deployed, not the deployment artifact. If you create a python package, that package actually contains a number of subpackages. Each subpackage must contain ALL the files relevant to deployment.
Do not use data_files as it will actually distribute the files as a separate package. (I've heard package_files works, but I have yet to see a working example to do this).

For example:
setup(
...
license="ZPL",
classifiers=[
...
'License :: OSI Approved :: Zope Public License',
...
],
...)
additionally you can insert your licence text into 'long_description':
setup(
...
long_description="Package description. \nLicense Text",
...)