If I have a tree that looks like:
├── project
│ ├── package
│ │ ├── __init__.py
│ │ ├── setup.py
├── env
└── setup.py
Is there a way to include the nested setup.py in the install for the top setup.py? I want to avoid this:
pip install -e . ; cd project/package ; pip install -e .
The solution is to have two separate projects: a main project (usually an application) and a sub-project (usually a library). The main application has a dependency to the library.
Tree structure and setup.py
The main project can have the following structure:
your_app/
|-- setup.py
ˋ-- src/
ˋ-- your_app/
|-- __init__.py
|-- module1.py
ˋ-- ...
The setup.py of your application can be:
from setuptools import find_packages
from setuptools import setup
setup(
name='Your-App',
version='0.1.0',
install_requires=['Your-Library'],
packages=find_packages('src'),
package_dir={'': 'src'},
url='https://github.com/your-name/your_app',
license='MIT',
author='Your NAME',
author_email='your#email.com',
description='Your main project'
)
You can notice that:
The name of your application can be slightly different to the name of your package;
This package has a dependency to "Your-Library", defined below;
You can put your source in the src directory, but it is optional. A lot of project have none.
The sub-project can have the following structure:
your_library/
|-- setup.py
ˋ-- src/
ˋ-- your_library/
|-- __init__.py
|-- lib1.py
ˋ-- ...
The setup of you library can be:
from setuptools import find_packages
from setuptools import setup
setup(
name='Your-Library',
version='0.1.0',
packages=find_packages('src'),
package_dir={'': 'src'},
url='https://github.com/your-name/your_library',
license='MIT',
author='Your NAME',
author_email='your#email.com',
description='Your sub-project'
)
Putting all things together
Create a virtualenv for your application and activate it
Go in the your_library/ directory and run:
pip install -e .
Then, go in your_app/ directory and run:
pip install -e .
You are now ready to code. Have fun!
See the Hitchhiker's Guide to Python: “Structuring Your Project”.
Related
I have a customized built module, lets call it abc, and pip install /local_path/abc-0.1-py3-none-any.whl. Installation is correct,
>>pip install dist/abc-0.1-py3-none-any.whl
Processing ./dist/abc-0.1-py3-none-any.whl
Successfully installed abc-0.1
but I could not import the module.
After I ran ppip freeze list and found out the name of module in list is abc # file:///local_path/abc-0.1-py3-none-any.whl.
my question is how could import the module? Thank you
.
├── requirements.txt
├── setup.py
├── src
│ ├── bin
│ │ ├── __init__.py
│ │ ├── xyz1.py
│ │ ├── xyz2.py
│ │ └── xyz3.py
here is my setup.py
with open("requirements.txt") as f:
install_requires = f.read()
setup(
name="abc",
version="0.1",
author="galaxyan",
author_email="galaxyan#123.com",
description="test whell framework",
packages=find_packages(include=["src"]),
zip_safe=False,
install_requires=install_requires,
)
############ update ############
it does not work even change setup.py
with open("requirements.txt") as f:
install_requires = f.read()
setup(
name="abc",
version="0.1",
author="galaxyan",
author_email="galaxyan#123.com",
description="test whell framework",
packages=find_packages(where="src"),
package_dir={"": "src"},
zip_safe=False,
install_requires=install_requires,
)
The setup.py is wrong, which means you're building a wheel with no packages actually inside.
Instead of
setup(
...
packages=find_packages(include=["src"]),
...
)
Try this:
setup(
...
packages=find_packages(where="src"),
package_dir={"": "src"},
...
)
See Testing & Packaging for more info.
I used to be able to run the command runner.py from my module my-runner while using python setup.py develop. However, ever since that I reinstalled it using python setup.py install, I now get a pkg_resources.ResolutionError when calling runner.py.
This is the mini tree structure
.
├── bin
│ ├── some_other_file.py
│ ├── runner.py
├── setup.py
here is my setup.py
from setuptools import setup, find_packages
setup(
name='my-runner',
version='1.0.0',
license='private',
author='MyName',
author_email='myname#myemail.com',
description='My Runner',
packages=find_packages(),
scripts=['bin/runner.py', 'bin/some_other_file.py']
)
Running command runner.py returns the error
pkg_resources.ResolutionError: Script 'scripts/runner.py' not found in metadata at '/home/myname/module/my-runner.egg-info
I guess I have no clue why install would break it? I am guessing it has to do with the fact that develop does not read the egg-info dir but would like a solution to this problem.
A possible way is to remove the package(pip3 uninstall my-runner) and reinstall it (python setup.py install)
I have the following in my setup.py
entry_points={
'console_scripts': [
'my-app=main:run',
],
and my file structure is
▶ tree -L 2
.
├── Dockerfile
├── README.md
├── redis.conf
├── setup.cfg
├── setup.py
├── src
│ ├── main.py
│ ├── settings.py
│ ├── bfile.py
│ └── afile.py
and in my main.py
def run():
actual_run()
Locally, in a virtualenv when performing
pip install -e . && my-app
it runs smoothly.
However in an image created with the following Dockerfile
FROM python:3.9-slim-buster
WORKDIR /app
COPY . .
RUN pip3 install .
CMD [ "my-app"]
I get
Traceback (most recent call last):
File "/usr/local/bin/my-app", line 5, in <module>
from main import run
ModuleNotFoundError: No module named 'main'
What I noticed, is that in my image, in /usr/local/lib/python3.9/site-packages/ there only the a file named my-app-0.0.1.dist-info and not a dir with the source files.
Why is that?
You didn't give a minimal workable setup.py, but next could works, FYI:
Dockerfile:
FROM python:3.9-slim-buster
WORKDIR /app
COPY . .
RUN pip3 install .
CMD [ "my-app"]
setup.py:
from setuptools import setup, find_packages
setup(
name="my-app",
version="0.0.1",
packages=['main'],
package_dir={'main':'src'},
entry_points={
'console_scripts': [
'my-app=main.main:run',
],
}
)
src/main.py:
def run():
print("hello world")
Execution:
# docker build -t abc:1 .
# docker run --rm abc:1
hello world
Explain:
In above, src folder will map to main module, and packages=['main'] specify the python package name which will be installed into site-packages.
In final image, the package will be located as next:
root#7b5ce63d8226:/usr/local/lib/python3.9/site-packages/main# ls
__pycache__ main.py
So, you need to use main.main:run to call your function.
I have this project architecture and I want to make my_package pip installable. The project doesn't only contain stuff to package but also simple scripts (the quick and dirty kind) and over things that are important in my project but not for the package (external data for example).
my_project
├── code
│ ├── data #<-- I don't want to package this
│ │ └── make_dataset.py
│ ├── script #<-- I don't want to package this
│ │ └── make_experiment.py
│ └── my_package #<-- This is the module I want to package
│ ├── core.py
│ ├── utils.py
│ └── __init__.py
├── data
│ └── some_data.txt
├── references
│ └── reference_paper.pdf
├── reports
│ └── report.tex
├── LICENSE
├── README.md
├── requirements.txt
└── setup.py
I would like the setup.py file to be in the top-level directory so that people can do the usual
git clone gitinstance.com/my_project
cd my_project
pip install .
and get my_package module installed in their environment so they can already do python -c import my_package; print(my_package.__version__) and it works.
The question is: How can I make my_package pip-installable without putting setup.py inside the code directory?
Usually, the setup.py would look like this:
from setuptools import find_packages, setup
setup(
name='my_package',
packages=find_packages(),
version='0.1.0',
description='Research project',
author='Name',
license='MIT',
)
But it wouldn't work here because setup.py can't find my_package.
I found an example in the documentation of setuptools that more or less fit my use-case.
The solution is in the packages and package_dir arguments of the setup function that allows to specify where to find the packages to install. This is usually hidden because it defaults to the current working directory.
In my simple case, the setup.py transforms to:
from setuptools import find_packages, setup
setup(
name='my_package',
packages=find_packages(where="code"),
package_dir={'': "code"},
version='0.1.0',
description='Research project',
author='Name',
license='MIT',
)
The reusable app docs (https://docs.djangoproject.com/en/1.9/intro/reusable-apps/) tells you to list template and static files in MANIFEST.in, but it doesn't look like python setup.py bdist_wheel looks at that file at all.
I've seen references to data_files but those files go in directories relative to the python installation (sys.prefix) and not the package installation (and sys.prefix isn't uniformly related to site-packages across systems).
Am I correct in assuming that myapp/templates/myapp/foo.html should end up in .../site-packages/myapp/templates/myapp/foo.html and similarly for static files, and that the user needs to run a manage.py collectstatic after pip install myapp?
Update (example):
The following structure:
(build2) go|c:\srv\tmp\myapp> tree
.
|-- MANIFEST.in
|-- myapp
| |-- static
| | `-- myapp
| | `-- foo.css
| |-- templates
| | `-- myapp
| | `-- foo.html
| |-- urls.py
| `-- views.py
`-- setup.py
5 directories, 6 files
setup.py
import setuptools
from distutils.core import setup
setup(
name='myapp',
version='0.1.0',
packages=['myapp']
)
MANIFEST.in
recursive-include myapp/templates *
recursive-include myapp/static *
running python setup.py sdist and python setup.py bdist_wheel creates the following files bin myapp/dist:
2016-06-18 13:47 2,073 myapp-0.1.0-py2-none-any.whl
2016-06-18 13:46 2,493 myapp-0.1.0.zip
if you look inside the .zip file, you'll find the templates and static folders, if you rename the .whl file to .zip and look inside it, the directories are not included.
Update 2 (solution):
Changing the MANIFEST.in file to
recursive-include myapp *
and setup.py to
from setuptools import find_packages, setup
setup(
name='myapp',
version='0.1.0',
include_package_data=True,
packages=['myapp'],
zip_safe=False,
)
then running python setup.py bdist_wheel will create a .whl file that installs myapp/templates and myapp/static in the expected places.
The MANIFEST.in file needs to be changed to:
recursive-include myapp *
This includes everything under myapp/myapp with the correct paths. In particular, this includes myapp/myapp/templates, which is necessary.
The declaration above also includes myapp/myapp/static which could be useful if you plan to run manage.py collectstatic after installing the .whl file.
In setup.py, the setup function needs to be imported from setuptools (and not distutils), i.e.:
from setuptools import find_packages, setup
setup(
name='myapp',
version='0.1.0',
include_package_data=True,
packages=['myapp'],
zip_safe=False,
)
When you now run python setup.py bdist_wheel it will create a .whl file that installs myapp/templates and myapp/static in the expected places.