We Keep Coding

How to correctly install data files with setup.py?

Using python 3.8 I have the following structure in a test library:
testrepo
setup.py
Manifest.in
util/
mycode.py
data/
mydata.txt
The file setup.py looks like
setup(
name='testrepo',
version="0.1.0",
packages=['util'],
author='Tester',
description='Testing repo',
include_package_data=True,
package_data={
"util.data": ["*"]
},
)
and using the following Manifest.in:
include util/data/mycode.txt
and when I install this package I do not see any hint of the data folder in venv/lib/python3.8/site-packages/util (when installing the repo into a python virtual environment).
How to do it correctly, so I can read the content from the file util/data/mydata.txt using
from util import data
import importlib.resources as import_resources
text = import_resources.read_text(data, "mydata.txt")
or whatever...
Where can I find this completely documented, with examples etc.?

I guess what you have to do is to create the following basic structure of the repo:
myrepo
setup.py
Manifest.in
mypackage/
__init__.py
mycode.py
data/
__init__.py
mydata.txt
Just make sure to keep in mind 6 additional steps:
You need to put the data folder inside your package folder
You need to add __init__.py inside your data folder.
In setup.py you have to use packages=find_packages(), to find your packages.
In setup.py, you have to set include_package_data=True,
In setup.py, you have to specify the path to your data files:
package_data={
"mypackage.data": ["*"]
},
You also have to define a second file names Manifest.in containing again your data files as follows (using a placeholder here - you can also add each file in a separate line):
include util/data/*
It you are lucky, then you can include/use your data file like
from mypackage import data
import importlib.resources as import_resources
text = import_resources.read_text(data, "mydata.txt")
or
with import_resources.path(data, "mydata.txt") as filename:
myfilename = filename
to get the path to the data file.
Not sure this is documented anywhere.

Building Python Package with source in different directory

When building a python package where the source tree looks like this:
src -\
+- module -\
<stuff>
+- setup.py
is pretty clear.
Is it possible to build a package where module source doesn't reside in the same location as the setup.py? For more specific use case the code for the module is either partially or full autogenerated in a location other then src
E.g.
src -\
+- setup.py
generated -\
module -\
<module code>

You can control the directory where packages reside by using the package_dir argument to setup(...)
and while it does appear to build a proper source distribution when package_dir is a relative path starting with .., it appears that pip will refuse to install it -- I'd suggest instead nesting your generated code inside that src directory instead and then using package_dir to select that.
Here's an example which moves all modules inside a generated subdir:
setup(
name='mypkg',
package_dir={'': 'generated'},
packages=find_packages('generated'),
)
Using a setup like:
$ tree .
.
├── generated
│   ├── mod1
│   │   └── __init__.py
│   └── mod2
│   └── __init__.py
└── setup.py
This would make the following succeed after install: import mod1; import mod2
If you wanted to make those modules available under a different prefix, you would do:
setup(
name='mypkg',
package_dir={'hello': 'generated'},
packages=[f'hello.{mod}' for mod in find_packages('generated')],
)
This would make import hello.mod1; import hello.mod2 succeed after installation

You can use relative paths in package lookup configuration. Examples:
all distributed sources are in generated
from setuptools import setup, find_packages
setup(
...
package_dir={'': '../generated'},
packages=find_packages(where='../generated'),
)
selected packages should be included from generated
In this example, only packages spam and eggs from generated will be included:
import pathlib
from setuptools import setup, find_packages
setup(
name='so',
package_dir={'spam': '../generated/spam', 'eggs': '../generated/eggs'},
packages=find_packages(where='../generated'), # or just ['spam', 'eggs']
)
Or implement a dynamic lookup like e.g.
package_dir={p.name: p.resolve() for p in pathlib.Path('..', 'generated').iterdir()}
better implementation by resolving all paths relative to the setup.py file
Resolving all paths relative to the setup.py script allows you to run the script from any other directory than src, e.g. you can run python src/setup.py bdist_wheel etc. You may or may not need it, depending on your use case. Nevertheless, the recipe is as usual: resolve all paths to __file__, e.g.
import pathlib
from setuptools import setup, find_packages
src_base = pathlib.Path(__file__, '..', '..', 'generated').resolve()
setup(
...
package_dir={'': str(src_base)},
packages=find_packages(where=src_base),
)

entry points from setup.py not recognized

I am trying to get an entry point to run my flask application.
I think its due to the directory structure:
my_app
- __init__.py
- app.py
- setup.py
- etc..
My setup.py file:
from setuptools import setup, find_packages
import os.path
def read_requirements(pathname):
with open(pathname) as f:
return [line for line in (x.strip() for x in f) if not line.startswith('#')]
def project_path(*names):
return os.path.join(os.path.dirname(__file__), *names)
setup(
name='my_app',
version='0.1.0',
install_requires=read_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt')),
test_suite='nose.collector',
entry_points={
'console_scripts': [
'START_ME=app:run',
],
},
classifiers=["Programming Language :: Python :: 2.7"],
description=__doc__,
long_description='\n\n'.join(open(project_path(name)).read() for name in (
'README.md',
)),
zip_safe=False,
include_package_data=True,
packages=find_packages(),
)
I think the find_packages() method is not picking up the fact that its in the package, maybe its looking in lower level directories for packages? I've tried find_packages('.') to try get it to search in the project root directory but this did not work.
Can I get this to work without changing my directory structure?
Here is the actual project.
Also, I noticed that when I run setup.py install I get a top_level.txt file in my egg.info folder, it says that the top level is actually a package that exists inside of the root/main package, like:
/ main_package
- __init__.py
- app.py
/ sub_package
- __init__.py
- sub.py
in the top_level.txt file, sub_package is written.

I just ended up putting all the flask app files into a sub directory inside the project root directory. fixed it nicely.
/project
- setup.py
/flask_app
- __init__.py
- app.py

Python setup.py include .json files in the egg

I want to package .json files as well in the python egg file.
For example: boto package has endpoints.json file. But when I run python setup.py bdist_egg it does not include the json file in the egg. How do I include the Json file in the egg?
How do I include *.json file in the egg?
Below is the setup.py code
from setuptools import setup, find_packages, Extension
setup(
name='X-py-backend',
version='tip',
description='X Python backend tools',
author='meme',
packages=find_packages('python'),
package_dir={'': 'python'},
data_files=[('boto', ['python/boto/endpoints.json'])],
namespace_packages = ['br'],
zip_safe=True,
)
setup(
name='X-py-backend',
version='tip',
packages=find_packages('protobuf/target/python'),
package_dir={'': 'protobuf/target/python'},
namespace_packages = ['br'],
zip_safe=True,
)

You only need to list the file on data_files parameter. Here is the example.
setup(
name='X-py-backend',
version='tip',
description='XXX Python backend tools',
author='meme',
packages=find_packages('python'),
package_dir={'': 'python'},
data_files=[('boto', ['boto/*.json'])]
namespace_packages = ['br'],
zip_safe=True
)
You can see the details here. https://docs.python.org/2/distutils/setupscript.html#installing-additional-files
Another way to do this is to use MANIFEST.in files. you need to create a MANIFEST.in file in your project root. Here is the example.
include python/boto/endpoints.json
Please visit here for more information.https://docs.python.org/2/distutils/sourcedist.html#manifest-template

Well this works for me.
setup.py:
from setuptools import setup, find_packages
setup(
name="clean",
version="0.1",
description="Clean package",
packages=find_packages() + ['config'],
include_package_data=True
)
MANIFEST.in:
recursive-include config *
where there is a config file under the project root directory which contains a whole bunch of json files.
Hope this helps.

Include non python files in RPM with setuptools

I have some fixture directories that contain xml files that I would like included with my python project when building the RPM with bdist_rpm. I thought I could do this by having MANIFEST.in do a recursive-include * *, however, it does not include anything other than *.py files. Is there anyway to have bdist_rpm include non python files in the package or specifically include *.xml files as well?

Where are you trying to install them? If you put them inside a package directory, like this...
myproject/
mypackage/
__init__.py
resources/
file1.xml
file2.xml
...you can use the package_data option in your setup.py file, like this:
from setuptools import setup, find_packages
setup(
name='myproject',
version='0.1',
description='A description.',
packages=find_packages(),
include_package_data=True,
package_data = { '': [ '*.xml' ] },
install_requires=[],
)
This will recursively include any *.xml files inside of any packages. They'll get installed with the rest of your package(s) somewhere inside of the Python library path. You can do the same thing with a MANIFEST.in that looks like this:
recursive-include * *.xml
If you're trying to install them into specific filesystem locations outside of the Python library, I'm not sure if you can do that via setup.py.

You can use data_files parameter of setup to do what you need. Something like this:
setup(
...
package_data = { '/usr/share/yourapp/xmls': [ 'xmls/1.xml', 'xmls/2.xml' ] },
...
)
This would install following files:
/usr/share/yourapp/xmls/1.xml
/usr/share/yourapp/xmls/2.xml
I generally create the list of files in a function like this:
def get_xmls():
xmlfiles = []
for filename in os.listdir('xmls/'):
if filename.endswith('.xml'):
xmlfiles.append('xmls/%s' % filename)
return xmlfiles