We Keep Coding

Package only binary compiled .so files of a python library compiled with Cython

I have a package named mypack which inside has a module mymod.py, and
the __init__.py.
For some reason that is not in debate, I need to package this module compiled
(nor .py or .pyc files are allowed). That is, the __init__.py is the only
source file allowed in the distributed compressed file.
The folder structure is:
.
│
├── mypack
│ ├── __init__.py
│ └── mymod.py
├── setup.py
I find that Cython is able to do this, by converting each .py file in a .so library
that can be directly imported with python.
The question is: how the setup.py file must be in order to allow an easy packaging and installation?
The target system has a virtualenv where the package must be installed with
whatever method that allows easy install and uninstall (easy_install, pip, etc are all
welcome).
I tried all that was at my reach. I read setuptools and distutils documentation,
all stackoverflow related questions,
and tried with all kind of commands (sdist, bdist, bdist_egg, etc), with lots
of combinations of setup.cfg and MANIFEST.in file entries.
The closest I got was with the below setup file, that would subclass the bdist_egg
command in order to remove also .pyc files, but that is breaking the installation.
A solution that installs "manually" the files in the venv is
also good, provided that all ancillary files that are included in a proper
installation are covered (I need to run pip freeze in the venv and see
mymod==0.0.1).
Run it with:
python setup.py bdist_egg --exclude-source-files
and (try to) install it with
easy_install mymod-0.0.1-py2.7-linux-x86_64.egg
As you may notice, the target is linux 64 bits with python 2.7.
from Cython.Distutils import build_ext
from setuptools import setup, find_packages
from setuptools.extension import Extension
from setuptools.command import bdist_egg
from setuptools.command.bdist_egg import walk_egg, log
import os
class my_bdist_egg(bdist_egg.bdist_egg):
def zap_pyfiles(self):
log.info("Removing .py files from temporary directory")
for base, dirs, files in walk_egg(self.bdist_dir):
for name in files:
if not name.endswith('__init__.py'):
if name.endswith('.py') or name.endswith('.pyc'):
# original 'if' only has name.endswith('.py')
path = os.path.join(base, name)
log.info("Deleting %s",path)
os.unlink(path)
ext_modules=[
Extension("mypack.mymod", ["mypack/mymod.py"]),
]
setup(
name = 'mypack',
cmdclass = {'build_ext': build_ext,
'bdist_egg': my_bdist_egg },
ext_modules = ext_modules,
version='0.0.1',
description='This is mypack compiled lib',
author='Myself',
packages=['mypack'],
)
UPDATE.
Following #Teyras answer, it was possible to build a wheel as requested in the answer. The setup.py file contents are:
import os
import shutil
from setuptools.extension import Extension
from setuptools import setup
from Cython.Build import cythonize
from Cython.Distutils import build_ext
class MyBuildExt(build_ext):
def run(self):
build_ext.run(self)
build_dir = os.path.realpath(self.build_lib)
root_dir = os.path.dirname(os.path.realpath(__file__))
target_dir = build_dir if not self.inplace else root_dir
self.copy_file('mypack/__init__.py', root_dir, target_dir)
def copy_file(self, path, source_dir, destination_dir):
if os.path.exists(os.path.join(source_dir, path)):
shutil.copyfile(os.path.join(source_dir, path),
os.path.join(destination_dir, path))
setup(
name = 'mypack',
cmdclass = {'build_ext': MyBuildExt},
ext_modules = cythonize([Extension("mypack.*", ["mypack/*.py"])]),
version='0.0.1',
description='This is mypack compiled lib',
author='Myself',
packages=[],
include_package_data=True )
The key point was to set packages=[],. The overwriting of the build_ext class run method was needed to get the __init__.py file inside the wheel.

Unfortunately, the answer suggesting setting packages=[] is wrong and may break a lot of stuff, as can e.g. be seen in this question. Don't use it. Instead of excluding all packages from the dist, you should exclude only the python files that will be cythonized and compiled to shared objects.
Below is a working example; it uses my recipe from the question Exclude single source file from python bdist_egg or bdist_wheel. The example project contains package spam with two modules, spam.eggs and spam.bacon, and a subpackage spam.fizz with one module spam.fizz.buzz:
root
├── setup.py
└── spam
├── __init__.py
├── bacon.py
├── eggs.py
└── fizz
├── __init__.py
└── buzz.py
The module lookup is being done in the build_py command, so it is the one you need to subclass with custom behaviour.
Simple case: compile all source code, make no exceptions
If you are about to compile every .py file (including __init__.pys), it is already sufficient to override build_py.build_packages method, making it a noop. Because build_packages doesn't do anything, no .py file will be collected at all and the dist will include only cythonized extensions:
import fnmatch
from setuptools import find_packages, setup, Extension
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize
extensions = [
# example of extensions with regex
Extension('spam.*', ['spam/*.py']),
# example of extension with single source file
Extension('spam.fizz.buzz', ['spam/fizz/buzz.py']),
]
class build_py(build_py_orig):
def build_packages(self):
pass
setup(
name='...',
version='...',
packages=find_packages(),
ext_modules=cythonize(extensions),
cmdclass={'build_py': build_py},
)
Complex case: mix cythonized extensions with source modules
If you want to compile only selected modules and leave the rest untouched, you will need a bit more complex logic; in this case, you need to override module lookup. In the below example, I still compile spam.bacon, spam.eggs and spam.fizz.buzz to shared objects, but leave __init__.py files untouched, so they will be included as source modules:
import fnmatch
from setuptools import find_packages, setup, Extension
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize
extensions = [
Extension('spam.*', ['spam/*.py']),
Extension('spam.fizz.buzz', ['spam/fizz/buzz.py']),
]
cython_excludes = ['**/__init__.py']
def not_cythonized(tup):
(package, module, filepath) = tup
return any(
fnmatch.fnmatchcase(filepath, pat=pattern) for pattern in cython_excludes
) or not any(
fnmatch.fnmatchcase(filepath, pat=pattern)
for ext in extensions
for pattern in ext.sources
)
class build_py(build_py_orig):
def find_modules(self):
modules = super().find_modules()
return list(filter(not_cythonized, modules))
def find_package_modules(self, package, package_dir):
modules = super().find_package_modules(package, package_dir)
return list(filter(not_cythonized, modules))
setup(
name='...',
version='...',
packages=find_packages(),
ext_modules=cythonize(extensions, exclude=cython_excludes),
cmdclass={'build_py': build_py},
)

While packaging as a wheel is definitely what you want, the original question was about excluding .py source files from the package. This is addressed in Using Cython to protect a Python codebase by #Teyras, but his solution uses a hack: it removes the packages argument from the call to setup(). This prevents the build_py step from running which does, indeed, exclude the .py files but it also excludes any data files you want included in the package. (For example my package has a data file called VERSION which contains the package version number.) A better solution would be replacing the build_py setup command with a custom command which only copies the data files.
You also need the __init__.py file as described above. So the custom build_py command should create the __init_.py file. I found that the compiled __init__.so runs when the package is imported so all that is needed is an empty __init__.py file to tell Python that the directory is a module which is ok to import.
Your custom build_py class would look like:
import os
from setuptools.command.build_py import build_py
class CustomBuildPyCommand(build_py):
def run(self):
# package data files but not .py files
build_py.build_package_data(self)
# create empty __init__.py in target dirs
for pdir in self.packages:
open(os.path.join(self.build_lib, pdir, '__init__.py'), 'a').close()
And configure setup to override the original build_py command:
setup(
...
cmdclass={'build_py': CustomBuildPyCommand},
)

I suggest you use the wheel format (as suggested by fish2000). Then, in your setup.py, set the packages argument to []. Your Cython extension will still build and the resulting .so files will be included in the resulting wheel package.
If your __init__.py is not included in the wheel, you can override the run method of build_ext class shipped by Cython and copy the file from your source tree to the build folder (the path can be found in self.build_lib).

This was exactly the sort of problem the Python wheels format – described in PEP 427 – was developed to address.
Wheels are a replacement for Python eggs (which were/are problematic for a bunch of reasons) – they are supported by pip, can contain architecture-specific private binaries (here is one example of such an arrangement) and are accepted generally by the Python communities who have stakes in these kind of things.
Here is one setup.py snippet from the aforelinked Python on Wheels article, showing how one sets up a binary distribution:
import os
from setuptools import setup
from setuptools.dist import Distribution
class BinaryDistribution(Distribution):
def is_pure(self):
return False
setup(
...,
include_package_data=True,
distclass=BinaryDistribution,
)
… in leu of the older (but probably somehow still canonically supported) setuptools classes you are using. It’s very straightforward to make Wheels for your distribution purposes, as outlined – as I recall from experience, either the wheel modules’ build process is somewhat cognizant of virtualenv, or it’s very easy to use one within the other.
In any case, trading in the setuptools egg-based APIs for wheel-based tooling should save you some serious pain, I should think.

packaging with numpy and test suite

Introduction
Disclaimer: I'm very new to python packaging with distutils. So far I've just stashed everything into modules, and packages manually and developed on top of that. I never wrote a setup.py file before.
I have a Fortran module that I want to use in my python code with numpy. I figured the best way to do that would be f2py, since it is included in numpy. To automate the build process I want to use distutils and the corresponding numpy enhancement, which includes convenience functions for f2py wrappers.
I do not understand how I should organize my files, and how to include my test suite.
What I want is the possibility to use ./setup.py for building, installing, and testing, and developing.
My directory structure looks as follows:
volterra
├── setup.py
└── volterra
├── __init__.py
├── integral.f90
├── test
│   ├── __init__.py
│   └── test_volterra.py
└── volterra.f90
And the setup.py file contains this:
def configuration(parent_package='', top_path=None):
from numpy.distutils.misc_util import Configuration
config = Configuration('volterra', parent_package, top_path)
config.add_extension('_volterra',
sources=['volterra/integral.f90', 'volterra/volterra.f90'])
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(**configuration(top_path='').todict())
After running ./setup.py build I get.
build/lib.linux-x86_64-2.7/
└── volterra
└── _volterra.so
Which includes neither the __init__.py file, nor the tests.
Questions
Is it really necessary to add the path to every single source file of the extension? (I.e. volterra/integral.f90) Can't I give a parameter which says, look for stuff in volterra/? The top_path, and package_dir parameters didn't do the trick.
Currently, the __init__.py file is not included in the build. Why is that?
How can I run my tests in this setup?
What's the best workflow for doing development in such an environment? I don't want to install my package for every single change I do. How do you do development in the source directory when you need to compile some extension modules?

Here is a setup.py that works for me:
# pkg - A fancy software package
# Copyright (C) 2013 author (email)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see http://www.gnu.org/licenses/gpl.html.
"""pkg: a software suite for
Hey look at me I'm a long description
But how long am I?
"""
from __future__ import division, print_function
#ideas for setup/f2py came from:
# -numpy setup.py: https://github.com/numpy/numpy/blob/master/setup.py 2013-11-07
# -winpython setup.py: http://code.google.com/p/winpython/source/browse/setup.py 2013-11-07
# -needing to use
# import setuptools; from numpy.distutils.core import setup, Extension:
# http://comments.gmane.org/gmane.comp.python.f2py.user/707 2013-11-07
# -wrapping FORTRAN code with f2py: http://www2-pcmdi.llnl.gov/cdat/tutorials/f2py-wrapping-fortran-code 2013-11-07
# -numpy disutils: http://docs.scipy.org/doc/numpy/reference/distutils.html 2013-11-07
# -manifest files in disutils:
# 'distutils doesn't properly update MANIFEST. when the contents of directories change.'
# https://github.com/numpy/numpy/blob/master/setup.py
# -if things are not woring try deleting build, sdist, egg directories and try again:
# https://stackoverflow.com/a/9982133/2530083 2013-11-07
# -getting fortran extensions to be installed in their appropriate sub package
# i.e. "my_ext = Extension(name = 'my_pack._fortran', sources = ['my_pack/code.f90'])"
# Note that sources is a list even if one file:
# http://numpy-discussion.10968.n7.nabble.com/f2py-and-setup-py-how-can-I-specify-where-the-so-file-goes-tp34490p34497.html 2013-11-07
# -install fortran source files into their appropriate sub-package
# i.e. "package_data={'': ['*.f95','*.f90']}# Note it's a dict and list":
# https://stackoverflow.com/a/19373744/2530083 2013-11-07
# -Chapter 9 Fortran Programming with NumPy Arrays:
# Langtangen, Hans Petter. 2013. Python Scripting for Computational Science. 3rd edition. Springer.
# -Hitchhikers guide to packaging :
# http://guide.python-distribute.org/
# -Python Packaging: Hate, hate, hate everywhere :
# http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
# -How To Package Your Python Code:
# http://www.scotttorborg.com/python-packaging/
# -install testing requirements:
# https://stackoverflow.com/a/7747140/2530083 2013-11-07
import setuptools
from numpy.distutils.core import setup, Extension
import os
import os.path as osp
def readme(filename='README.rst'):
with open('README.rst') as f:
text=f.read()
f.close()
return text
def get_package_data(name, extlist):
"""Return data files for package *name* with extensions in *extlist*"""
#modified slightly from taken from http://code.google.com/p/winpython/source/browse/setup.py 2013-11-7
flist = []
# Workaround to replace os.path.relpath (not available until Python 2.6):
offset = len(name)+len(os.pathsep)
for dirpath, _dirnames, filenames in os.walk(name):
for fname in filenames:
if not fname.startswith('.') and osp.splitext(fname)[1] in extlist:
# flist.append(osp.join(dirpath, fname[offset:]))
flist.append(osp.join(dirpath, fname))
return flist
DOCLINES = __doc__.split("\n")
CLASSIFIERS = """\
Development Status :: 1 - Planning
License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)
Programming Language :: Python :: 2.7
Topic :: Scientific/Engineering
"""
NAME = 'pkg'
MAINTAINER = "me"
MAINTAINER_EMAIL = "me#me.com"
DESCRIPTION = DOCLINES[0]
LONG_DESCRIPTION = "\n".join(DOCLINES[2:])#readme('readme.rst')
URL = "http://meeeee.mmemem"
DOWNLOAD_URL = "https://github.com/rtrwalker/geotecha.git"
LICENSE = 'GNU General Public License v3 or later (GPLv3+)'
CLASSIFIERS = [_f for _f in CLASSIFIERS.split('\n') if _f]
KEYWORDS=''
AUTHOR = "me"
AUTHOR_EMAIL = "me.com"
PLATFORMS = ["Windows"]#, "Linux", "Solaris", "Mac OS-X", "Unix"]
MAJOR = 0
MINOR = 1
MICRO = 0
ISRELEASED = False
VERSION = '%d.%d.%d' % (MAJOR, MINOR, MICRO)
INSTALL_REQUIRES=[]
ZIP_SAFE=False
TEST_SUITE='nose.collector'
TESTS_REQUIRE=['nose']
DATA_FILES = [(NAME, ['LICENSE.txt','README.rst'])]
PACKAGES=setuptools.find_packages()
PACKAGES.remove('tools')
PACKAGE_DATA={'': ['*.f95','*f90']}
ext_files = get_package_data(NAME,['.f90', '.f95','.F90', '.F95'])
ext_module_names = ['.'.join(osp.splitext(v)[0].split(osp.sep)) for v in ext_files]
EXT_MODULES = [Extension(name=x,sources=[y]) for x, y in zip(ext_module_names, ext_files)]
setup(
name=NAME,
version=VERSION,
maintainer=MAINTAINER,
maintainer_email=MAINTAINER_EMAIL,
description=DESCRIPTION,
long_description=LONG_DESCRIPTION,
url=URL,
download_url=DOWNLOAD_URL,
license=LICENSE,
classifiers=CLASSIFIERS,
author=AUTHOR,
author_email=AUTHOR_EMAIL,
platforms=PLATFORMS,
packages=PACKAGES,
data_files=DATA_FILES,
install_requires=INSTALL_REQUIRES,
zip_safe=ZIP_SAFE,
test_suite=TEST_SUITE,
tests_require=TESTS_REQUIRE,
package_data=PACKAGE_DATA,
ext_modules=EXT_MODULES,
)
To install, at the command line I use:
python setup.py install
python setup.py clean --all
The only issue I seem to have is a minor one. when I look in site-packages for my package it is installed inside the egg folder C:\Python27\Lib\site-packages\pkg-0.1.0-py2.7-win32.egg\pkg. Most other packages I see there have a C:\Python27\Lib\site-packages\pkg folder separate to the egg folder. Does anyone know how to get that separation?
As for testing, after installing, I type the following at the command line:
nosetests package_name -v
Try investigating python setup.py develop (Python setup.py develop vs install) for not having to install the package after every change.
As I commented in the code I found the following useful:
numpy setup.py: https://github.com/numpy/numpy/blob/master/setup.py 2013-11-07
winpython setup.py: http://code.google.com/p/winpython/source/browse/setup.py 2013-11-07
needing to use
import setuptools; from numpy.distutils.core import setup, Extension:
http://comments.gmane.org/gmane.comp.python.f2py.user/707 2013-11-07
wrapping FORTRAN code with f2py: http://www2-pcmdi.llnl.gov/cdat/tutorials/f2py-wrapping-fortran-code 2013-11-07
numpy disutils: http://docs.scipy.org/doc/numpy/reference/distutils.html 2013-11-07
manifest files in disutils:
'distutils doesn't properly update MANIFEST. when the contents of directories change.'
https://github.com/numpy/numpy/blob/master/setup.py
if things are not woring try deleting build, sdist, egg directories and try again:
https://stackoverflow.com/a/9982133/2530083 2013-11-07
getting fortran extensions to be installed in their appropriate sub package
i.e. "my_ext = Extension(name = 'my_pack._fortran', sources = ['my_pack/code.f90'])"
Note that sources is a list even if one file:
http://numpy-discussion.10968.n7.nabble.com/f2py-and-setup-py-how-can-I-specify-where-the-so-file-goes-tp34490p34497.html 2013-11-07
install fortran source files into their appropriate sub-package
i.e. "package_data={'': ['.f95','.f90']}# Note it's a dict and list":
https://stackoverflow.com/a/19373744/2530083 2013-11-07
Chapter 9 Fortran Programming with NumPy Arrays:
Langtangen, Hans Petter. 2013. Python Scripting for Computational Science. 3rd edition. Springer.
Hitchhikers guide to packaging :
http://guide.python-distribute.org/
Python Packaging: Hate, hate, hate everywhere :
http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
How To Package Your Python Code:
http://www.scotttorborg.com/python-packaging/
install testing requirements:
https://stackoverflow.com/a/7747140/2530083 2013-11-07
'python setup.py develop' :
https://stackoverflow.com/a/19048754/2530083

Here is setup.py from a project I made. I have found figuring out setup.py / packaging to be frustrating with no solid answers and definitely not pythonic in the sense of having one and only one obvious way to do something. Hopefully this will help a little.
The points you may find useful are:
find_packages which removes the drudgery of including lots of files or messing around with generating manifest.
package_data which allows you to easily specify non .py files to be included
install_requires / tests_require
You'll need to find the source for distribute_setup.py if you don't have it already.
Is it really necessary to add the path to every single source file of
the extension? (I.e. volterra/integral.f90) Can't I give a parameter
which says, look for stuff in volterra/? The top_path, and package_dir
parameters didn't do the trick.
Currently, the init.py file is not
included in the build. Why is that?
Hopefully find_packages() will solve both of those. I don't have much experience packaging but I haven't had to go back to manual inclusion yet.
How can I run my tests in this
setup?
I think this is probably a different question with many answers depending on how you are doing tests. Maybe you can ask it separately?
As a side note, I am under the impression that the standard is to put your tests directory at the top level. I.e. volterra/volterra and volterra/tests.
What's the best workflow for doing development in such an
environment? I don't want to install my package for every single
change I do. How do you do development in the source directory when
you need to compile some extension modules?
This might be worth another question as well. I don't see why you would need to install your package for every single change. If you are uploading the package, just don't install it on your dev system (except to test installation) and work directly from your development copy. Maybe I'm missing something though since I don't work with compiled extensions.
Here is the example
try:
from setuptools import setup, find_packages
except ImportError:
from distribute_setup import use_setuptools
use_setuptools()
from setuptools import setup, find_packages
setup(
# ... other stuff
py_modules=['distribute_setup'],
packages=find_packages(),
package_data={'': ['*.png']}, # for me to include anything with png
install_requires=['numpy', 'treenode', 'investigators'],
tests_require=['mock', 'numpy', 'treenode', 'investigators'],
)

setup.py not installing data files

I have a Python library that, in addition to regular Python modules, has some data files that need to go in /usr/local/lib/python2.7/dist-package/mylibrary.
Unfortunately, I have been unable to convince setup.py to actually install the data files there. Note that this behaviour is under install - not sdist.
Here is a slightly redacted version of setup.py
module_list = list_of_files
setup(name ='Modules',
version ='1.33.7',
description ='My Sweet Module',
author ='PN',
author_email ='email',
url ='url',
packages = ['my_module'],
# I tried this. It got installed in /usr/my_module. Not ok.
# data_files = [ ("my_module", ["my_module/data1",
# "my_module/data2"])]
# This doesn't install it at all.
package_data = {"my_module" : ["my_module/data1",
"my_module/data2"] }
)
This is in Python 2.7 (will have to run in 2.6 eventually), and will have to run on some Ubuntu between 10.04 and 12+. Developing it right now on 12.04.

UPD:
package_data accepts dict in format {'package': ['list', 'of?', 'globs*']}, so to make it work, one should specify shell globs relative to package dir, not the file paths relative to the distribution root.
data_files has a different meaning, and, in general, one should avoid using this parameter.
With setuptools you only need include_package_data=True, but data files should be under version control system, known to setuptools (by default it recognizes only CVS and SVN, install setuptools-git or setuptools-hg if you use git or hg...)
with setuptools you can:
- in MANIFEST.im:
include my_module/data*
- in setup.py:
setup(
...
include_package_data = True,
...
)

http://docs.python.org/distutils/setupscript.html#installing-additional-files
If directory is a relative path, it is interpreted relative to the
installation prefix (Python’s sys.prefix for pure-Python packages,
sys.exec_prefix for packages that contain extension modules).
This will probably do it:
data_files = [ ("my_module", ["local/lib/python2.7/dist-package/my_module/data1",
"local/lib/python2.7/dist-package/my_module/data2"])]
Or just use join to add the prefix:
data_dir = os.path.join(sys.prefix, "local/lib/python2.7/dist-package/my_module")
data_files = [ ("my_module", [os.path.join(data_dir, "data1"),
os.path.join(data_dir, "data2")])]

The following solution worked fine for me.
You should have MANIFEST.in file where setup.py is located.
Add the following code to the manifest file
recursive-include mypackage *.json *.md # can be extended with more extensions or file names.
Another solution is adding the following code to the MANIFEST.in file.
graft mypackage # will copy the entire package including non-python files.
global-exclude __pyache__ *.txt # list files you dont want to include here.
Now, when you do pip install all the necessary files will be included.
Hope this helps.
UPDATE:
Make sure that you also have include_package_data=True in the setup file

setup.py isn't including modules

setup.py includes modules in the same directory it is in but leaves out modules in the site.packages directory and in a lib folder I have added to the pythonpath.
Don't tell me it doesn't just import files like the python interpreter. It should if you name them.
programdir-->
programdir-->
datadir
program.py
functions.py
setup.py
setup.py
from distutils.core import setup
setup(name = "program",
version = "1.0",
description = "a program",
author = "me",
author_email = "email#email.com",
url = "http://www.url.com",
py_modules = ["program", "functions", "tweepy", "anothermod", "webbrowser","" ],
data_files = [("data", ["data/intros.txt"])],
long_description = """
Descriptttttionnnn
"""
)
What am I doing wrong?

The setup.py is responsible for bundling and installing YOUR sources, not dependencies. However, you can specify requirements with install_requires (read the manual). This will be used by pip or setuptools to install the given dependencies, but it will not bundle them either.

Julius is using distutils, not setuptools. This means that requires shall be passed to the setup() function rather than install_requires. See the docs:
2.4. Relationships between Distributions and Packages

Installing my sdist from PyPI puts the files in unexpected places

My problem is that when I upload my Python package to PyPI, and then install it from there using pip, my app breaks because it installs my files into completely different locations than when I simply install the exact same package from a local sdist.
Installing from the local sdist puts files on my system like this:
/Python27/
Lib/
site-packages/
gloopy-0.1.alpha-py2.7.egg/ (egg and install info files)
data/ (images and shader source)
doc/ (html)
examples/ (.py scripts that use the library)
gloopy/ (source)
This is much as I'd expect, and works fine (e.g. my source can find my data dir, because they lie next to each other, just like they do in development.)
If I upload the same sdist to PyPI and then install it from there, using pip, then things look very different:
/Python27/
data/ (images and shader source)
doc/ (html)
Lib/
site-packages/
gloopy-0.1.alpha-py2.7.egg/ (egg and install info files)
gloopy/ (source files)
examples/ (.py scripts that use the library)
This doesn't work at all - my app can't find its data files, plus obviously it's a mess, polluting the top-level /python27 directory with all my junk.
What am I doing wrong? How do I make the pip install behave like the local sdist install? Is that even what I should be trying to achieve?
Details
I have setuptools installed, and also distribute, and I'm calling distribute_setup.use_setuptools()
WindowsXP, Python2.7.
My development directory looks like this:
/gloopy
/data (image files and GLSL shader souce read at runtime)
/doc (html files)
/examples (some scripts to show off the library)
/gloopy (the library itself)
My MANIFEST.in mentions all the files I want to be included in the sdist, including everything in the data, examples and doc directories:
recursive-include data *.*
recursive-include examples *.py
recursive-include doc/html *.html *.css *.js *.png
include LICENSE.txt
include TODO.txt
My setup.py is quite verbose, but I guess the best thing is to include it here, right? I also includes duplicate references to the same data / doc / examples directories as are mentioned in the MANIFEST.in, because I understand this is required in order for these files to be copied from the sdist to the system during install.
NAME = 'gloopy'
VERSION= __import__(NAME).VERSION
RELEASE = __import__(NAME).RELEASE
SCRIPT = None
CONSOLE = False
def main():
import sys
from pprint import pprint
from setup_utils import distribute_setup
from setup_utils.sdist_setup import get_sdist_config
distribute_setup.use_setuptools()
from setuptools import setup
description, long_description = read_description()
config = dict(
name=name,
version=version,
description=description,
long_description=long_description,
keywords='',
packages=find_packages(),
data_files=[
('examples', glob('examples/*.py')),
('data/shaders', glob('data/shaders/*.*')),
('doc', glob('doc/html/*.*')),
('doc/_images', glob('doc/html/_images/*.*')),
('doc/_modules', glob('doc/html/_modules/*.*')),
('doc/_modules/gloopy', glob('doc/html/_modules/gloopy/*.*')),
('doc/_modules/gloopy/geom', glob('doc/html/_modules/gloopy/geom/*.*')),
('doc/_modules/gloopy/move', glob('doc/html/_modules/gloopy/move/*.*')),
('doc/_modules/gloopy/shapes', glob('doc/html/_modules/gloopy/shapes/*.*')),
('doc/_modules/gloopy/util', glob('doc/html/_modules/gloopy/util/*.*')),
('doc/_modules/gloopy/view', glob('doc/html/_modules/gloopy/view/*.*')),
('doc/_static', glob('doc/html/_static/*.*')),
('doc/_api', glob('doc/html/_api/*.*')),
],
classifiers=[
'Development Status :: 1 - Planning',
'Intended Audience :: Developers',
'License :: OSI Approved :: BSD License',
'Operating System :: Microsoft :: Windows',
'Programming Language :: Python :: 2.7',
],
# see classifiers http://pypi.python.org/pypi?:action=list_classifiers
)
config.update(dict(
author='Jonathan Hartley',
author_email='tartley#tartley.com',
url='http://bitbucket.org/tartley/gloopy',
license='New BSD',
) )
if '--verbose' in sys.argv:
pprint(config)
setup(**config)
if __name__ == '__main__':
main()

The data_files parameter is for data files who isn't a part of the package. You should probably use package_data instead.
See https://docs.python.org/3/distutils/setupscript.html#installing-package-data
That wouldn't install the data in site-packages/data, but in my opinion that's not where is should be installed anyway. You won't know which package it's a part of. It should be installed in site-packages//gloopy-0.1.alpha-py2.7.egg/[data|doc|examples] IMO.
If you really do think the data is not package data, then you should use data_files and in that case pip installs it correctly, while I'd claim setup.py install installs it in the wrong place. But in my opinion, in this case, it is package_data, as it's related to the package, and not used by other software.

You can load package data with pkgutil.get_data(), it will find where exactly package data is installed.
Here is a nice blog post about including data files in packages: Including data files into Python packages