Related
I am attempting to build a python wheel using setuptools. The package needs to include two files:
mymodule.py - a python module in the same directory as setup.py
myjar.jar - a java .jar file that exists outside of my package directory
I am building my package using python3 setup.py bdist_wheel.
If I call setup() like so:
setup(
name="mypkg",
py_modules=["mymodule"],
data_files=[('jars', ['../target/scala-2.11/myjar.jar'])]
)
then myjar.jar does successfully get included in the .whl (good so far) however when I pip install mypkg it places the jar at /usr/local/myjar.jar (this kinda explains why) which isn't what I want at all, I want it to exist in the same place as mymodule.py, i.e. /usr/local/lib/python3.7/site-packages/
If I change setup.py to
setup(
name="mypkg",
py_modules=["mymodule"],
package_data={'jars': '../target/scala-2.11/myjar.jar'}
)
or
setup(
name="mypkg",
py_modules=["mymodule"],
package_data={'jars': ['../target/scala-2.11/myjar.jar']}
)
then myjar.jar simply doesn't get included in the .whl. I tried copying myjar.jar into the same directory and changing setup.py to:
setup(
name="mypkg",
py_modules=["mymodule"],
package_data={'jars': 'myjar.jar'}
)
or
setup(
name="mypkg",
py_modules=["mymodule"],
package_data={'jars': ['myjar.jar']}
)
but still myjar.jar does not get included in the .whl.
I've been tearing my hair out over this for hours, hence why I'm here.
I've read a myriad of SO posts on this:
How to include package data with setuptools/distribute?
MANIFEST.in ignored on "python setup.py install" - no data files installed?
How do you add additional files to a wheel?
setuptools: adding additional files outside package
which suggest different combinations of data_files, package_data, include_package_data=True and/or use of a Manifest.in file but still I can't get this working as I would like, so I'm here hoping someone can advise what I'm doing wrong.
The data files (in that case myjar.jar) should really be package data files, and as such they should be part of a Python package. So having such files in parent directories makes things much more complicated, but probably not impossible. So let's start with a simpler example. I believe something like the following should work...
Project directory structure:
MyProject
├ MANIFEST.in
├ mymodule.py
├ setup.py
└ myjars
├ __init__.py
└ myjar.jar
MANIFEST.in:
recursive-include myjars *.jar
setup.py:
#!/usr/bin/env python3
import setuptools
setuptools.setup(
name='MyProject',
version='1.2.3',
#
include_package_data=True,
packages=['myjars'],
py_modules=["mymodule"],
)
myjars/__init__.py might not be strictly necessary, but I believe it's better to have it. And as always, an empty __init__.py file is perfectly good enough.
(This assumes the myjars/myjar.jar file exists before the source distribution sdist is built.)
As to dealing with the data files in parent directories, my recommendation would be to simply copy (or symlink) those files before calling setup.py, maybe as part of a shell script or anything like that. There are probably ways to do the copy as part of a custom setuptools command in setup.py, but it's not worth the effort in my opinion, and really it's not part of setup.py's job.
I have a typical project structure that looks as follows:
EngineEmulator
src
ship
engine
emulator
mapping
tests
emulator
mapping
utils
common
doc
....
tools
....
setup.py
MANIFEST.in
setup.cfg
README.rst
My setup.py looks as follows:
from setuptools import setup, find_packages
setup(
name='Engine',
version=1.0.0,
description='Engine Project',
package_dir={'': 'src'},
packages=find_packages(
'src',
exclude=["*.tests", "*.tests.*", "tests.*", "tests"]),
install_requires =['pycrypto',
'kombu >=1.1.3'],
author='Demo',
author_email='demo#eliza.net'
license='MIT',
classifiers=[
'Topic :: Demo Engine',
'Development Status:: 3 - Iteration',
'Programming Language :: Python -2.6'
]
)
My setup.cfg looks as follows:
[egg_info]
tag_build = .dev
tag_svn_revision = 1
[rotate]
#keep last 15 eggs, clean up order
match = .egg
keep = 15
And My MANIFEST.in looks as follows:
include README.rst
recursive-include src/ship/Engine
prune src/utils
prune src/ship/tests
prune tools/
When I run python setup.py bdist_egg and python setup.py bdist_rpm I get the egg file and two rpm files generated (noarch.rpm and src.rpm).
In my destination machine when I run easy_install <generated egg file> my eg.info file gets copied over but the source files don't get copied over to /usr/lib/python2.6/site-packages. I was expecting I would have a directory called Engine.
Can anybody point out what I am doing wrong? Thanks in advance.
Try to keep things as simple as possible.
Quick check with sdist
Try this:
$ python setup.py sdist
It shall create source distribution file for your package.
It is in zip format, so unpack it and check, if there are all expected files inside present.
If not, you have to find the reason, why expected files are missing in your distribution.
Checking things step by step (and simplifying)
Do you use .py extension?
May be stupid question, but in your file listing I do not see any py files inside of src tree.
In case you have there just files without .py extension, find_packages will not find anything.
Where do you have your __init__.py files located?
Let us know, where the files are:
$ cd src
$ find . -name "*.py"
If you miss __init__.py, find_packages will not find whole package.
Remove utils package
Why do you have it there?
Better have it installed out of your source code you develop or move it subdirectory in your project
root.
This will render prune src/utils unnecessary in your MANIFEST.in.
Put into MANIFEST.in only what must be there
If you read doc for MANIFEST.in, it states, what files are included automatically (all, what
mentioned in arguments of setup function, so in your case all python source files returned by
find_packages).
For this reason, you shall remove recursive-include src/shop/Engine as it shall be already
included by setup call.
Remove prune lines.
src/utils shall not be in your source tree - it is just messing things up.
tools is not to be included, so there is no need to prune it.
src/ship/tests can be there, it will not harm, if you keep these files in the destribution.
Assert, what packages were found
Make sure, your setup get proper names for packages.
For this purpuse, you can call find_package sooner and assert it containts, what you expect.
(temporarily) remove setup.cfg
Just to keep things simpler.
Proposed project reorganization
You shall have file structure in similar manner as follows:
src/ship/__init__.py
src/ship/engine/__init__.py
src/ship/engine/emulator/__init__.py
src/ship/engine/emulator/module.py
src/ship/engine/emulator/module2.py
src/ship/engine/mapping/other.py
src/ship/engine/mapping/another.py
src/ship/tests/__init__.py
src/ship/tests/emulator/__init__.py
src/ship/tests/emulator/test_module.py
src/ship/tests/emulator/test_module2.py
src/ship/tests/mapping/__init__.py
src/ship/tests/mapping/test_other.py
src/ship/tests/mapping/test_another.py
doc
doc/index.rst
tools
tools/knife.py
setup.py
MANIFEST.in
README.rst
setup.py
from setuptools import setup, find_packages
packages=find_packages("src")
assert "ship.engine" in packages
assert "ship.engine.emulator" in packages
assert "ship.engine.mapping" in packages
#etc
install_requires =['pycrypto', 'kombu>=1.1.3'] #watch the spaces around `>=`, shall not be there
setup(
name="Engine",
package_dir={'': 'src'},
packages=packages,
install_requires=install_requires
)
MANIFEST.in
include README.rst
Conclusions
It might happen, that running
$ python setup.py sdist
would fail on asserts. This is sign, some of expected files are missing. Check that.
After you make your project living in simple way, you might add more details around (and do it step
by step to be sure, you do not break something).
How do I make setup.py include a file that isn't part of the code? (Specifically, it's a license file, but it could be any other thing.)
I want to be able to control the location of the file. In the original source folder, the file is in the root of the package. (i.e. on the same level as the topmost __init__.py.) I want it to stay exactly there when the package is installed, regardless of operating system. How do I do that?
Probably the best way to do this is to use the setuptools package_data directive. This does mean using setuptools (or distribute) instead of distutils, but this is a very seamless "upgrade".
Here's a full (but untested) example:
from setuptools import setup, find_packages
setup(
name='your_project_name',
version='0.1',
description='A description.',
packages=find_packages(exclude=['ez_setup', 'tests', 'tests.*']),
package_data={'': ['license.txt']},
include_package_data=True,
install_requires=[],
)
Note the specific lines that are critical here:
package_data={'': ['license.txt']},
include_package_data=True,
package_data is a dict of package names (empty = all packages) to a list of patterns (can include globs). For example, if you want to only specify files within your package, you can do that too:
package_data={'yourpackage': ['*.txt', 'path/to/resources/*.txt']}
The solution here is definitely not to rename your non-py files with a .py extension.
See Ian Bicking's presentation for more info.
UPDATE: Another [Better] Approach
Another approach that works well if you just want to control the contents of the source distribution (sdist) and have files outside of the package (e.g. top-level directory) is to add a MANIFEST.in file. See the Python documentation for the format of this file.
Since writing this response, I have found that using MANIFEST.in is typically a less frustrating approach to just make sure your source distribution (tar.gz) has the files you need.
For example, if you wanted to include the requirements.txt from top-level, recursively include the top-level "data" directory:
include requirements.txt
recursive-include data *
Nevertheless, in order for these files to be copied at install time to the package’s folder inside site-packages, you’ll need to supply include_package_data=True to the setup() function. See Adding Non-Code Files for more information.
To accomplish what you're describing will take two steps...
The file needs to be added to the source tarball
setup.py needs to be modified to install the data file to the source path
Step 1: To add the file to the source tarball, include it in the MANIFEST
Create a MANIFEST template in the folder that contains setup.py
The MANIFEST is basically a text file with a list of all the files that will be included in the source tarball.
Here's what the MANIFEST for my project look like:
CHANGELOG.txt
INSTALL.txt
LICENSE.txt
pypreprocessor.py
README.txt
setup.py
test.py
TODO.txt
Note: While sdist does add some files automatically, I prefer to explicitly specify them to be sure instead of predicting what it does and doesn't.
Step 2: To install the data file to the source folder, modify setup.py
Since you're looking to add a data file (LICENSE.txt) to the source install folder you need to modify the data install path to match the source install path. This is necessary because, by default, data files are installed to a different location than source files.
To modify the data install dir to match the source install dir...
Pull the install dir info from distutils with:
from distutils.command.install import INSTALL_SCHEMES
Modify the data install dir to match the source install dir:
for scheme in INSTALL_SCHEMES.values():
scheme['data'] = scheme['purelib']
And, add the data file and location to setup():
data_files=[('', ['LICENSE.txt'])]
Note: The steps above should accomplish exactly what you described in a standard manner without requiring any extension libraries.
It is 2019, and here is what is working -
despite advice here and there, what I found on the internet halfway documented is using setuptools_scm, passed as options to setuptools.setup. This will include any data files that are versioned on your VCS, be it git or any other, to the wheel package, and will make "pip install" from the git repository to bring those files along.
So, I just added these two lines to the setup call on "setup.py". No extra installs or import required:
setup_requires=['setuptools_scm'],
include_package_data=True,
No need to manually list package_data, or in a MANIFEST.in file - if it is versioned, it is included in the package. The docs on "setuptools_scm" put emphasis on creating a version number from the commit position, and disregard the really important part of adding the data files. (I can't care less if my intermediate wheel file is named "*0.2.2.dev45+g3495a1f" or will use the hardcoded version number "0.3.0dev0" I've typed in - but leaving crucial files for the program to work behind is somewhat important)
create MANIFEST.in in the project root with recursive-include to the required directory or include with the file name.
include LICENSE
include README.rst
recursive-include package/static *
recursive-include package/templates *
documentation can be found here
Step 1: create a MANIFEST.in file in the same folder with setup.py
Step 2: include the relative path to the files you want to add in MANIFEST.in
include README.rst
include docs/*.txt
include funniest/data.json
Step 3: set include_package_data=True in the setup() function to copy these files to site-package
Reference is here.
I wanted to post a comment to one of the questions but I don't enough reputation to do that >.>
Here's what worked for me (came up with it after referring the docs):
package_data={
'mypkg': ['../*.txt']
},
include_package_data: False
The last line was, strangely enough, also crucial for me (you can also omit this keyword argument - it works the same).
What this does is it copies all text files in your top-level or root directory (one level up from the package mypkg you want to distribute).
None of the above really worked for me. What saved me was this answer.
Apparently, in order for these data files to be extracted during installation, I had to do a couple of things:
Like already mentioned - Add a MANIFEST.in to the project and specify the folder/files you want to be included. In my case: recursive-include folder_with_extra_stuff *
Again, like already mentioned - Add include_package_data=True to your setup.py. This is crucial, because without it only the files that match *.py will be brought.
This is what was missing! - Add an empty __init__.py to your data folder. For me I had to add this file to my folder-with-extra-stuff.
Extra - Not sure if this is a requirement, but with my own python modules I saw that they're zipped inside the .egg file in site-packages. So I had to add zip_safe=False to my setup.py file.
Final Directory Structure
my-app/
├─ app/
│ ├─ __init__.py
│ ├─ __main__.py
├─ folder-with-extra-stuff/
│ ├─ __init__.py
│ ├─ data_file.json
├─ setup.py
├─ MANIFEST.in
This works in 2020!
As others said create "MANIFEST.in" where your setup.py is located.
Next in manifest include/exclude all the necessary things. Be careful here regarding the syntax.
Ex: lets say we have template folder to be included in the source package.
in manifest file do this :
recursive-include template *
Make sure you leave space between dir-name and pattern for files/dirs like above.
Dont do like this like we do in .gitignore
recursive-include template/* [this won't work]
Other option is to use include. There are bunch of options. Look up here at their docs for Manifest.in
And the final important step, include this param in your setup.py and you are good to go!
setup(
...
include_package_data=True,
......
)
Hope that helps! Happy Coding!
In setup.py under setup( :
setup(
name = 'foo library'
...
package_data={
'foolibrary.folderA': ['*'], # All files from folder A
'foolibrary.folderB': ['*.txt'] #All text files from folder B
},
Here is a simpler answer that worked for me.
First, per a Python Dev's comment above, setuptools is not required:
package_data is also available to pure distutils setup scripts
since 2.3. – Éric Araujo
That's great because putting a setuptools requirement on your package means you will have to install it also. In short:
from distutils.core import setup
setup(
# ...snip...
packages = ['pkgname'],
package_data = {'pkgname': ['license.txt']},
)
I just wanted to follow up on something I found working with Python 2.7 on Centos 6. Adding the package_data or data_files as mentioned above did not work for me. I added a MANIFEST.IN with the files I wanted which put the non-python files into the tarball, but did not install them on the target machine via RPM.
In the end, I was able to get the files into my solution using the "options" in the setup/setuptools. The option files let you modify various sections of the spec file from setup.py. As follows.
from setuptools import setup
setup(
name='theProjectName',
version='1',
packages=['thePackage'],
url='',
license='',
author='me',
author_email='me#email.com',
description='',
options={'bdist_rpm': {'install_script': 'filewithinstallcommands'}},
)
file - MANIFEST.in:
include license.txt
file - filewithinstallcommands:
mkdir -p $RPM_BUILD_ROOT/pathtoinstall/
#this line installs your python files
python setup.py install -O1 --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
#install license.txt into /pathtoinstall folder
install -m 700 license.txt $RPM_BUILD_ROOT/pathtoinstall/
echo /pathtoinstall/license.txt >> INSTALLED_FILES
None of the answers worked for me because my files were at the top level, outside the package. I used a custom build command instead.
import os
import setuptools
from setuptools.command.build_py import build_py
from shutil import copyfile
HERE = os.path.abspath(os.path.dirname(__file__))
NAME = "thepackage"
class BuildCommand(build_py):
def run(self):
build_py.run(self)
if not self.dry_run:
target_dir = os.path.join(self.build_lib, NAME)
for fn in ["VERSION", "LICENSE.txt"]:
copyfile(os.path.join(HERE, fn), os.path.join(target_dir,fn))
setuptools.setup(
name=NAME,
cmdclass={"build_py": BuildCommand},
description=DESCRIPTION,
...
)
For non-python files to be included in an installation, they must be within one of the installed package directories. If you specify non-python files outside of your package directories in MANIFEST.in, they will be included in your distribution, but they will not be installed. The "documented" ways of installing arbitrary files outside of your package directories do not work reliably (as everyone has noticed by now).
The above answer from Julian Mann copies the files to your package directory in the build directory, so it does work, but not if you are installing in editable/develop mode (pip install -e or python setup.py develop). Based on this answer to a related question (and Julian's answer), below is an example that copies files to your installed package location either way after all the other install/develop tasks are done. The assumption here is that files file1 and file2 in your root-level data directory will be copied to your installed package directory (my_package), and that they will be accessible from python modules in your package using os.path.join(os.path.dirname(__file__), 'file1'), etc.
Remember to also do the MANIFEST.in stuff described above so that these files are also included in your distribution. Why setuptools would include files in your distribution and then silently never install them, is beyond my ken. Though installing them outside of your package directory is probably even more dubious.
import os
from setuptools import setup
from setuptools.command.develop import develop
from setuptools.command.install import install
from shutil import copyfile
HERE = os.path.abspath(os.path.dirname(__file__))
NAME = 'my_package'
def copy_files (target_path):
source_path = os.path.join(HERE, 'data')
for fn in ["file1", "file2"]:
copyfile(os.path.join(source_path, fn), os.path.join(target_path,fn))
class PostDevelopCommand(develop):
"""Post-installation for development mode."""
def run(self):
develop.run(self)
copy_files (os.path.abspath(NAME))
class PostInstallCommand(install):
"""Post-installation for installation mode."""
def run(self):
install.run(self)
copy_files (os.path.abspath(os.path.join(self.install_lib, NAME)))
setup(
name=NAME,
cmdclass={
'develop': PostDevelopCommand,
'install': PostInstallCommand,
},
version='0.1.0',
packages=[NAME],
include_package_data=True,
setup_requires=['setuptools_scm'],
)
Figured out a workaround: I renamed my lgpl2.1_license.txt to lgpl2.1_license.txt.py, and put some triple quotes around the text. Now I don't need to use the data_files option nor to specify any absolute paths. Making it a Python module is ugly, I know, but I consider it less ugly than specifying absolute paths.
I have a Python library that, in addition to regular Python modules, has some data files that need to go in /usr/local/lib/python2.7/dist-package/mylibrary.
Unfortunately, I have been unable to convince setup.py to actually install the data files there. Note that this behaviour is under install - not sdist.
Here is a slightly redacted version of setup.py
module_list = list_of_files
setup(name ='Modules',
version ='1.33.7',
description ='My Sweet Module',
author ='PN',
author_email ='email',
url ='url',
packages = ['my_module'],
# I tried this. It got installed in /usr/my_module. Not ok.
# data_files = [ ("my_module", ["my_module/data1",
# "my_module/data2"])]
# This doesn't install it at all.
package_data = {"my_module" : ["my_module/data1",
"my_module/data2"] }
)
This is in Python 2.7 (will have to run in 2.6 eventually), and will have to run on some Ubuntu between 10.04 and 12+. Developing it right now on 12.04.
UPD:
package_data accepts dict in format {'package': ['list', 'of?', 'globs*']}, so to make it work, one should specify shell globs relative to package dir, not the file paths relative to the distribution root.
data_files has a different meaning, and, in general, one should avoid using this parameter.
With setuptools you only need include_package_data=True, but data files should be under version control system, known to setuptools (by default it recognizes only CVS and SVN, install setuptools-git or setuptools-hg if you use git or hg...)
with setuptools you can:
- in MANIFEST.im:
include my_module/data*
- in setup.py:
setup(
...
include_package_data = True,
...
)
http://docs.python.org/distutils/setupscript.html#installing-additional-files
If directory is a relative path, it is interpreted relative to the
installation prefix (Python’s sys.prefix for pure-Python packages,
sys.exec_prefix for packages that contain extension modules).
This will probably do it:
data_files = [ ("my_module", ["local/lib/python2.7/dist-package/my_module/data1",
"local/lib/python2.7/dist-package/my_module/data2"])]
Or just use join to add the prefix:
data_dir = os.path.join(sys.prefix, "local/lib/python2.7/dist-package/my_module")
data_files = [ ("my_module", [os.path.join(data_dir, "data1"),
os.path.join(data_dir, "data2")])]
The following solution worked fine for me.
You should have MANIFEST.in file where setup.py is located.
Add the following code to the manifest file
recursive-include mypackage *.json *.md # can be extended with more extensions or file names.
Another solution is adding the following code to the MANIFEST.in file.
graft mypackage # will copy the entire package including non-python files.
global-exclude __pyache__ *.txt # list files you dont want to include here.
Now, when you do pip install all the necessary files will be included.
Hope this helps.
UPDATE:
Make sure that you also have include_package_data=True in the setup file
How can I write setup.py so that:
The binary egg distribution (bdist_egg) includes a sample configuration file and
Upon installation puts it into the {prefix}/etc directory?
A sample project source directory looks like this:
bin/
myapp
etc/
myapp.cfg
myapp/
__init__.py
[...]
setup.py
The setup.py looks like this:
from distutils.command.install_data import install_data
packages = ['myapp', ]
scripts = ['bin/myapp',]
cmdclasses = {'install_data': install_data}
data_files = [('etc', ['etc/myapp.cfg'])]
setup_args = {
'name': 'MyApp',
'version': '0.1',
'packages': packages,
'cmdclass': cmdclasses,
'data_files': data_files,
'scripts': scripts,
# 'include_package_data': True,
'test_suite': 'nose.collector'
}
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
setup(**setup_args)
setuptools are installed in both the build environment and in the installation environment.
The 'include_package_data' commented out or not does not help.
I was doing some research on this issue and I think the answer is in the setuptools documentation: http://peak.telecommunity.com/DevCenter/setuptools#non-package-data-files
Next, I quote the extract that I think has the answer:
Non-Package Data Files
The distutils normally install general "data files" to a
platform-specific location (e.g. /usr/share). This feature intended to
be used for things like documentation, example configuration files,
and the like. setuptools does not install these data files in a
separate location, however. They are bundled inside the egg file or
directory, alongside the Python modules and packages. The data files
can also be accessed using the Resource Management API [...]
Note, by the way, that this encapsulation of data files means that you
can't actually install data files to some arbitrary location on a
user's machine; this is a feature, not a bug. You can always include a
script in your distribution that extracts and copies your the
documentation or data files to a user-specified location, at their
discretion. If you put related data files in a single directory, you
can use resource_filename() with the directory name to get a
filesystem directory that then can be copied with the shutil module.
[...]