We Keep Coding

pyproject.toml/setuptools duplicates files into root site-packages directory

I have a problem with how pip/setuptools is installing my package. When installing from the project directory (i.e. pip install .) my project's sub-packages are duplicated and placed in the root site-packages directory. The configuration is set entirely within pyproject.toml (with a minimal setup.py for compiling a single extension).
If my package is named mypackage which contains 3 sub-packages and depends on 3 dependencies, this is the expected directory structure under site-packages in the venv:
site-packages
- dependency1
- dependency2
- dependency3
- myproject
- subpackage1
- subpackage2
- subpackage3
Yet below is what I end up with, it looks like any folder containing any .py files are copied to root site-packages (i.e. including the venv itself and docs since they contain py files:
site-packages
- dependency1
- dependency2
- dependency3
- mypackage
- subpackage1
- subpackage2
- subpackage3
- subpackage1
- subpackage2
- subpackage3
- docs
- venv
What can I do to avoid duplicating sub-packages into the top-level site-packages directory/install correctly?
Here is my project structure:
myproject/
- pyproject.toml
- setup.py
- docs/
- myproject/
- __init__.py
- subpackage1/
- subpackage2/
- subpackage3/
- venv/
The reduced contents of pyproject.toml
[project]
name = "myproject"
requires-python = ">= 3.7"
dependencies = [
"dependency1",
"dependency2",
"dependency3",
]
[tool.setuptools]
packages = [
"myproject",
"myproject.subpackage1",
"myproject.subpackage2",
"myproject.subpackage3",
]
[build-system]
requires = ["setuptools >= 61.0.0", "cython"]
build-backend = "setuptools.build_meta"
The contents of setup.py:
from setuptools import Extension, setup
from Cython.Build import cythonize
ext_modules = [
Extension(
"subpackage1.func",
["..."],
extra_compile_args=['-fopenmp'],
extra_link_args=['-fopenmp'],
)
]
setup(ext_modules=cythonize(ext_modules))

I've just ran into the same issue.
In my case, the build directory used by pip was polluted with the "subfolders", probably because of a previous run where my package discovery settings were erroneous.
Because of this, although my configuration was (now) correct, these orphaned directories were copied to my site-packages as well.
In my case the build directory was in the folder where I called pip install . from.
If you want to find the build directory, or just check whether this is the problem, log pip's output to a file with pip install . --log foo.txt, and search for copying inside.
You should see a line like:
Arguments: ('copying', '<build directory>\\lib\\subpackage1\\bar.py', ...
Hope this helps!

How to correctly install data files with setup.py?

Using python 3.8 I have the following structure in a test library:
testrepo
setup.py
Manifest.in
util/
mycode.py
data/
mydata.txt
The file setup.py looks like
setup(
name='testrepo',
version="0.1.0",
packages=['util'],
author='Tester',
description='Testing repo',
include_package_data=True,
package_data={
"util.data": ["*"]
},
)
and using the following Manifest.in:
include util/data/mycode.txt
and when I install this package I do not see any hint of the data folder in venv/lib/python3.8/site-packages/util (when installing the repo into a python virtual environment).
How to do it correctly, so I can read the content from the file util/data/mydata.txt using
from util import data
import importlib.resources as import_resources
text = import_resources.read_text(data, "mydata.txt")
or whatever...
Where can I find this completely documented, with examples etc.?

I guess what you have to do is to create the following basic structure of the repo:
myrepo
setup.py
Manifest.in
mypackage/
__init__.py
mycode.py
data/
__init__.py
mydata.txt
Just make sure to keep in mind 6 additional steps:
You need to put the data folder inside your package folder
You need to add __init__.py inside your data folder.
In setup.py you have to use packages=find_packages(), to find your packages.
In setup.py, you have to set include_package_data=True,
In setup.py, you have to specify the path to your data files:
package_data={
"mypackage.data": ["*"]
},
You also have to define a second file names Manifest.in containing again your data files as follows (using a placeholder here - you can also add each file in a separate line):
include util/data/*
It you are lucky, then you can include/use your data file like
from mypackage import data
import importlib.resources as import_resources
text = import_resources.read_text(data, "mydata.txt")
or
with import_resources.path(data, "mydata.txt") as filename:
myfilename = filename
to get the path to the data file.
Not sure this is documented anywhere.

entry points from setup.py not recognized

I am trying to get an entry point to run my flask application.
I think its due to the directory structure:
my_app
- __init__.py
- app.py
- setup.py
- etc..
My setup.py file:
from setuptools import setup, find_packages
import os.path
def read_requirements(pathname):
with open(pathname) as f:
return [line for line in (x.strip() for x in f) if not line.startswith('#')]
def project_path(*names):
return os.path.join(os.path.dirname(__file__), *names)
setup(
name='my_app',
version='0.1.0',
install_requires=read_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt')),
test_suite='nose.collector',
entry_points={
'console_scripts': [
'START_ME=app:run',
],
},
classifiers=["Programming Language :: Python :: 2.7"],
description=__doc__,
long_description='\n\n'.join(open(project_path(name)).read() for name in (
'README.md',
)),
zip_safe=False,
include_package_data=True,
packages=find_packages(),
)
I think the find_packages() method is not picking up the fact that its in the package, maybe its looking in lower level directories for packages? I've tried find_packages('.') to try get it to search in the project root directory but this did not work.
Can I get this to work without changing my directory structure?
Here is the actual project.
Also, I noticed that when I run setup.py install I get a top_level.txt file in my egg.info folder, it says that the top level is actually a package that exists inside of the root/main package, like:
/ main_package
- __init__.py
- app.py
/ sub_package
- __init__.py
- sub.py
in the top_level.txt file, sub_package is written.

I just ended up putting all the flask app files into a sub directory inside the project root directory. fixed it nicely.
/project
- setup.py
/flask_app
- __init__.py
- app.py

Paths from .egg file not working (os.path.dirname)

I have a Scrapy spider that is using gettext to translate some strings. The localization files is stored in /locale/.
When I load the translation I do it with the following code:
t = gettext.translation('sv', localedir=LOCALE_DIR, languages=['sv'])
LOCALE_DIR is set in settings.py with the following code:
import os
BASE_DIR = os.path.dirname(os.path.dirname(__file__))
LOCALE_DIR = os.path.join(BASE_DIR, 'locale')
This works great when I run the code locally with scrapy crawl spider. But when I deploy it to scrapyd, it generates an .egg file and the localization files can no longer be found. When I print the LOCALE_DIR from the deployed version. It gives me /tmp/condobot-1428391146-4QuH3E.egg/locale.
I guess this is the reason why the files can not be found. The path is a subfolder of a file, which does not make sense. What I expected was that the .egg file would be extracted into a folder, and the path would point to /tmp/condobot-1428391146-4quh3e/locale.
Is there another, better way of setting the path to LOCALE_DIR than the way that I currently do it? I have also tried setting it to locale without any better results.
EDIT: I do use a setup.py file with the following code:
from setuptools import setup, find_packages
setup(
name = 'project',
version = '1.0',
packages = find_packages(),
entry_points = {'scrapy': ['settings = condobot.settings']},
package_data = {
# If any package contains *.txt or *.rst files, include them:
'': ['*.mo', '*.po', '*.txt'],
},
)
I also tried using the following setup.py with a MANIFEST.in file:
from setuptools import setup, find_packages
setup(
name = 'project',
install_requires = ['distribute'],
version = '1.0',
packages = find_packages(),
entry_points = {'scrapy': ['settings = condobot.settings']},
include_package_data = True,
zip_safe = False,
)
MANIFEST.in
recursive-include locale *
recursive-include condobot/locale *
My file structure looks like this:
- condobot
- locale
- sv
- LC_MESSAGES
sv.mo
pipelines.py
settings.py
- locale
- sv
- LC_MESSAGES
sv.mo
MANIFEST.in
requirements.txt
scrapy.cfg
setup.py
(I have placed the /locale/ folder both in / and in /condobot/ just to make sure that the path is not wrong.)
I have extracted the .egg file and I can confirm that it does include the /locale/ folder, and in the /locale/ folder there is /locale/sv/LC_MESSAGES/sv.mo and /locale/sv/LC_MESSAGES/sv.po.
So the issue does not seem to be that the setup.py file is not including the files in the .egg file. It seems to be that the path /......./file.egg/locale/ does not work.

One way to be sure that any other non-source file is accessible after the python module is packaged as an egg is to specify zip_safe as False to have the package fully extracted when installed, i.e.:
setup(
name = 'project',
version = '1.0',
...
zip_safe = False,
)

Making a python package for PIP with some data_files

I'm doing a project with this layout:
project/
bin/
my_bin.py
CHANGES.txt
docs/
LICENSE.txt
README.txt
MANIFEST.in
setup.py
project/
__init__.py
some_thing.py
default_data.json
other_datas/
default/
other_default_datas.json
And the problem is that when I install this using pip, it puts the "default_data.json" and the "other_datas" folder not in the same place as the rest of the app.
How am I supposed to do to make them be in the same place?
They end up on "/home/user/.virtualenvs/proj-env/project"
instead of "/home/user/.virtualenvs/proj-env/lib/python2.6/site-packages/project"
In the setup.py I'm doing it like this:
inside_dir = 'project'
data_folder= os.path.join(inside_dir,'other_datas')
data_files = [(inside_dir, [os.path.join(inside_dir,'default_data.json')])]
for dirpath, dirnames, filenames in os.walk(data_folder):
data_files.append([dirpath, [os.path.join(dirpath, f) for f in filenames]])

From https://docs.python.org/3.4/distutils/setupscript.html#installing-additional-files:
If directory is a relative path, it is interpreted relative to the installation prefix (Python’s sys.prefix for pure-Python packages, sys.exec_prefix for packages that contain extension modules).
Each file name in files is interpreted relative to the setup.py script at the top of the package source distribution.
So the described behavior is simply how data_files work.
If you want to include the data files within your package you need to use package_data instead:
package_data={'project': ['default_data.json', 'other_datas/default/*.json']}

Take a look at this package https://pypi.python.org/pypi/datafolder. It makes it easy to install and use (data files: *.conf, *.ini *.db, ...) by your package and by the user.

Change your MANIFEST.in to include those .json.
It is probably gonna work:
recursive-include project/ *.json