I have a Git repository cloned into myproject, with an __init__.py at the root of the repository, making the whole thing an importable Python package.
I'm trying to write a setuptools setup.py for the package, which will also sit in the root of the repository, next to the __init__.py file. I want setup.py to install the directory it resides in as a package. It's fine if setup.py itself comes along as part of the installation, but it would be better if it didn't. Ideally this should work also in editable mode (pip install -e .)
Is this configuration at all supported? I can kind of make it work by having a package_dir= {"": ".."}, argument to setup(), telling it to look for myproject in the directory above the current one. However, this requires the package to always be installed from a directory named myproject, which does not appear to be the case if, say, it's being installed through pip, or if someone is working out of a Git clone named myproject-dev, or in any number of other cases.
Another hack I'm contemplating is a symlink to . named mypackage inside of the repository. That ought to work, but I wanted to check if there was a better way first.
See also Create editable package setup.py in the same root folder as __init__.py
As far as I know this should work:
myproject-dev/
├── __init__.py
├── setup.py
└── submodule
└── __init__.py
#!/usr/bin/env python3
import setuptools
setuptools.setup(
name='MyProject',
version='0.0.0.dev0',
packages=['myproject', 'myproject.submodule'],
package_dir={
'myproject': '.',
},
)
One way to make this work for editable or develop installations is to manually modify the easy-install.pth file.
Assuming:
the project lives in: /home/user/workspace/empty/project;
a virtual environment .venv is used;
the project is installed with python3 -m pip install -e . or python3 setup.py develop;
the Python version is 3.6.
Then:
the file is found at a location such as /home/user/workspace/empty/project/.venv/lib/python3.6/site-packages/easy-install.pth;
its content is: /home/user/workspace/empty/project.
In order to let the imports work as expected one can edit this line to read the following:
/home/user/workspace/empty
Note:
Everything in /home/user/workspace/empty that looks like a Python package is then susceptible to be imported, that is why it is a good idea to place the project in its own directory, in this case the directory empty contains nothing else but the directory project.
The module project.setup is also importable.
Related
I have a small team of people working on a small new Python project.
I created a small package called Environment.
This has two files:
Setup.py
from setuptools import setup, find_packages
setup(name='Environment', version='1.0', packages=find_packages())
root.py
ROOT_DIR = "STATIC_VALUE_15"
When I use pip install -e . and then run the project within the anaconda enviroment I use for this project, I can run files that use this Environment package. However when other people use the same branch and do the same steps, they get the error:
Exception has occurred: ModuleNotFoundError
They are also using their conda environment and running the project the same way in vsc.
When anyone uses pip list, for this package our paths are all correct.
Are there any suggestions for things that I could try and test to get it to work on their system, or why it works on mine and not theirs?
Setup.py is deprecated, use pyproject.toml instead.
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "environment"
version = "1.0.0"
You also need to follow this file hierarchy:
package_project/
└── src/
└── environment/
├── __init__.py
└── root.py
You may read Packaging Python Projects. There is plenty of room to make mistake, I struggled a lot in the past because I forgot a file, or didn't follow the file hierarchy, and if the build fail, sometime you can still run or import the code because of your python environment.
If you want to test if it works for someone else without having someone else, you can build the package and install it in an other environment where the sources are not easily accessible for python, using the .whl.
I have a python package with the following standard directory structure:
package_name/
setup.py
package_name/
module_1.py
module_2.py
...
tests/
docs/
I've installed this package with pip3 install -e .. I've noticed an inconsistent importing issue. (Please read to the end!) If I restart terminal and run the following (1) within the interpreter:
>>> from package_name import module_1
I get an import error. If I instead run this (2):
>>> from package_name.package_name import module_1
it imports fine. If I then navigate to the directory and rerun pip3 install -e ., I can import in the standard way (following (1)). What on earth is causing this? To make things stranger, I can import in the standard way (1) in Jupyter and my IDE without reinstalling the package. This issue only comes up when I open/restart terminal.
This should be fixed by adding the main project folder package_name/ to your PATH
Also, try renaming your project folders with different names, in order to avoid confusion for yourself, people working with you and also to help python to find the right module location
You should also create the __init__.py files on each module folders, even if those are empty files. This also helps python to find the modules locations.
I've reviewed the information here: How can I make setuptools (or distribute) install a package from the local file system
It looks like that question was posted a very long time ago and I'm hoping that in the seven years since it was posted that there have been some new developments to managing dependencies in the Python world.
Specifically, in our case we are working on a repository of related GCP packages:
src/
airflow/
__init__.py
dags/
__init__.py
requirements.txt
dag1.py
libs/
__init__.py
utils.py
tests/
dags/
test_dag1.py
plugins/
dataflow/
__init__.py
setup.py
dataflow1.py
libs/
__init__.py
utils.py
cli_helper/
__init__.py
cli_command.py
libs/
__init__.py
util.py
shared_utils/
util1.py
I have found myself repeating the same bits of helper functions within the context of each package and would like to put those helper functions in one place and then have a linked copy of the shared_utils files either in a shared_utils folder under each package or even to just have a copy of util1.py placed under the existing libs directory for each package.
What is most "pythonic" way to accomplish this?
Right now it seems that my only options would be to:
Use requirements.txt as listed above where I can and use a custom command in setup.py where requirements.txt can't be used.
Create a os level link from the shared_utils directory into each package such that it appears that the directory exists natively in each of my packages.
Package my shared_utils and then install directly from git. Though this option again requires requirements.txt and in some of my deployment environments, I can't rely on requirements.txt, I have to run everything through setup.py
If you can't use requirements.txt, maybe you can use setuptool's install_requires and go the shared_utils package root. Would that solve your issue?
I have created a custom python package following this guide, so I have the following structure:
mypackage/ <-- VCS root
mypackage/
submodule1/
submodule2/
setup.py
And setup.py contains exactly the same information as in the guide:
from setuptools import setup, find_packages
setup(name='mypackage',
version='0.1',
description='desc',
url='vcs_url',
author='Hodossy, Szabolcs',
author_email='myemail#example.com',
license='MIT',
packages=find_packages(),
install_requires=[
# deps
],
zip_safe=False)
I have noticed if I go into the folder where setup.py is, and then call python setup.py install in a virtual environment, in site-packages the following structure is installed:
.../site-packages/mypackage-0.1-py3.6.egg/mypackage/
submodule1/
submodule2/
but if I call it from one folder up like python mypackage/setup.py install, then the structure is the following:
.../site-packages/mypackage-0.1-py3.6.egg/mypackage/
mypackage/
submodule1/
submodule2/
This later one ruins all imports from my module, as the path is different for the submodules.
Could you explain what is happening here and how to prevent that kind of behaviour?
This is experienced with Python 3.6 on both Windows and Linux.
Your setup.py does not contain any paths, but seems to only find the files via find_packages. So of course it depends from where you run it. The setup.py isn't strictly tied to its location. Of course you could do things like chdir to the basename of the setup file path in sys.argv[0], but that's rather ugly.
The question is, WHY do you want to build it that way? It looks more like you would want a structure like
mypackage-source
mypackage
submodule1
submodule2
setup.py
And then execute setup.py from the work directory. If you want to be able to run it from anywhere, the better workaround would be to put a shellscript next to it, like
#!/bin/sh
cd ``basename $0``
python setup.py $#
which separates the task of changing to the right directory (here I assume the directory with setup.py in the workdir) from running setup.py
I have a repository I inherited used by a lot of teams, lots of scripts call it, and it seems like its going to be a real headache to make any structural changes to it. I would like to make this repo installable somehow. It is structured like this:
my_repo/
scripts.py
If it was my repository, I would change the structure like so and make it installable, and run python setup.py install:
my_repo/
setup.py
my_repo/
__init__.py
scripts.py
If this is not feasible (and it sounds like it might not be), can I somehow do something like:
my_repo/
setup.py
__init__.py
scripts.py
And add something to setup.py to let it know that the repo is structured funny like this, so that I can install it?
You can do what you suggest.
my_repo/
setup.py
__init__.py
scripts.py
The only thing is you will need to import modules in your package via their name if they are in the base level. So for example if your structure looked like this:
my_repo/
setup.py
__init__.py
scripts.py
lib.py
pkg/
__init__.py
pkgmodule.py
Then your imports in scripts.py might look like
from lib import func1, func2
from pkg.pkgmodule import stuff1, stuff2
So in your base directory imports are essentially by module name not by package. This could screw up some of your other packages namespaces if you're not careful, like if there is another dependency with a package named lib. So it would be best if you have these scripts running in a virtualenv and if you test to ensure namespacing doesn't get messed up
There is a directive in setup.py file to set the name of a package to install and from where it should get it's modules for installation. That would let you use the desired directory structure. For instance with a given directory structure as :
my_repo/
setup.py
__init__.py
scripts.py
You could write a setup.py such as:
setup(
# -- Package structure ----
packages=['my_repo'],
package_dir={'my_repo': '.'})
Thus anyone installing the contents of my_repo with the command "./setup.py install" or "pip install ." would end up with an installed copy of my_repo 's modules.
As a side note; relative imports work differently in python 2 and python 3. In the latter, any relative imports need to explicitly specify the will to do so. This method of installing my_repo will work in python 3 when calling in an absolute import fashion:
from my_repo import scripts