I'm pip-installing my module like so:
cd my_working_dir
pip install -e .
When I later import the module from within Python, can I somehow detect if the module is installed in this editable mode?
Right now, I'm just checking if there's a .git folder in os.path.dirname(mymodule.__file__)) which, well, only works if there's actually a .git folder there. Is there a more reliable way?
Another workaround:
Place an "not to install" file into your package. This can be a README.md, or a not_to_install.txt file. Use any non-pythonic extension, to prevent that file installation. Then check if that file exists in your package.
The suggested source structure:
my_repo
|-- setup.py
`-- awesome_package
|-- __init__.py
|-- not_to_install.txt
`-- awesome_module.py
setup.py:
# setup.py
from setuptools import setup, find_packages
setup(
name='awesome_package',
version='1.0.0',
# find_packages() will ignore non-python files.
packages=find_packages(),
)
The __init__.py or the awesome_module.py:
import os
# The current directory
__here__ = os.path.dirname(os.path.realpath(__file__))
# Place the following function into __init__.py or into awesome_module.py
def check_editable_installation():
'''
Returns true if the package was installed with the editable flag.
'''
not_to_install_exists = os.path.isfile(os.path.join(__here__, 'not_to_install.txt'))
return not_to_install_exists
Run pip list, and there is a column called "Editable project location". If that column has a value, specifically the directory from which you installed it, then the package is pip installed in editable mode.
I don't know of a way to detect this directly (e.g. ask setuptools).
You could try to detect that you package can not be reached through the paths in sys.path. But that's tedious. It's also not bullet proof -- what if it can be reached through sys.path but it's also installed as en editable package?
The best option is to look at the artifacts an editable install leaves in your site-packages folder. There's a file called my_package.egg-link there.
from pathlib import Path
# get site packages folder through some other magic
# assuming this current file is located in the root of your package
current_package_root = str(Path(__file__).parent.parent)
installed_as_editable = False
egg_link_file = Path(site_packages_folder) / "my_package.egg-link"
try:
linked_folder = egg_link_file.read_text()
installed_as_editable = current_package_root in linked_folder
except FileNotFoundError:
installed_as_editable = False
Note: to make this a bit more bullet-proof, read only the first line of the egg-link file and parse it using Path() as well to account for proper slashes etc.
Recently I had to test if various packages were installed in editable mode across different machines. Running pip show <package name> reveals not only the version, but other information about it, which includes the location of the source code. If the package was not installed in editable mode, this location will point to site-packages, so for my case it was sufficient with checking the output of such command:
import subprocess
def check_if_editable(name_of_the_package:str) -> bool:
out = subprocess.check_output(["pip", "show", f"{name_of_the_package}"]).decode()
return "site-packages" in out
Haven't found a definite source for this, but if the output of
$ pip show my-package -f
is something like this:
..
..
Files:
__editable__.my-package-0.0.5.pth
my-package-0.0.5.dist-info/INSTALLER
my-package-0.0.5.dist-info/METADATA
my-package-0.0.5.dist-info/RECORD
my-package-0.0.5.dist-info/REQUESTED
my-package-0.0.5.dist-info/WHEEL
my-package-0.0.5.dist-info/direct_url.json
my-package-0.0.5.dist-info/top_level.txt
then it's probably editable.
Related
I am using setuptools to create a python package. This is the directory structure:
mypackage
.git
__init__.py
pyproject.toml
setup.cfg
module1.py
module2.py
I have this structured in a flat hierarchy so I can clone this repository/copy this directory into a parent project/repository and directly write from mypackage import something, instead of having to pip install or play around with PYTHONPATH.
What do I specify in the setup.cfg file, such that this is installed as a single package, given the file structure?
# setup.cfg
[options]
package_dir = # What is here?
I tried out the custom discovery options in setuptools, which seems to work. I have to manually specify the root directory for packages. However setuptools fails when it finds multiple top-level modules/packages. So I have to use the find option to explicitly include discovered modules into a single package.
[options]
# Use the find function to find modules/packages
packages = find:
# The root directory is the current directory, relative to setup.cfg
packages_dir =
.
[options.packages.find]
# Options for the find command. Include is implicitly looking for all .py files,
# where the directory is "."
where = .
I have a Git repository cloned into myproject, with an __init__.py at the root of the repository, making the whole thing an importable Python package.
I'm trying to write a setuptools setup.py for the package, which will also sit in the root of the repository, next to the __init__.py file. I want setup.py to install the directory it resides in as a package. It's fine if setup.py itself comes along as part of the installation, but it would be better if it didn't. Ideally this should work also in editable mode (pip install -e .)
Is this configuration at all supported? I can kind of make it work by having a package_dir= {"": ".."}, argument to setup(), telling it to look for myproject in the directory above the current one. However, this requires the package to always be installed from a directory named myproject, which does not appear to be the case if, say, it's being installed through pip, or if someone is working out of a Git clone named myproject-dev, or in any number of other cases.
Another hack I'm contemplating is a symlink to . named mypackage inside of the repository. That ought to work, but I wanted to check if there was a better way first.
See also Create editable package setup.py in the same root folder as __init__.py
As far as I know this should work:
myproject-dev/
├── __init__.py
├── setup.py
└── submodule
└── __init__.py
#!/usr/bin/env python3
import setuptools
setuptools.setup(
name='MyProject',
version='0.0.0.dev0',
packages=['myproject', 'myproject.submodule'],
package_dir={
'myproject': '.',
},
)
One way to make this work for editable or develop installations is to manually modify the easy-install.pth file.
Assuming:
the project lives in: /home/user/workspace/empty/project;
a virtual environment .venv is used;
the project is installed with python3 -m pip install -e . or python3 setup.py develop;
the Python version is 3.6.
Then:
the file is found at a location such as /home/user/workspace/empty/project/.venv/lib/python3.6/site-packages/easy-install.pth;
its content is: /home/user/workspace/empty/project.
In order to let the imports work as expected one can edit this line to read the following:
/home/user/workspace/empty
Note:
Everything in /home/user/workspace/empty that looks like a Python package is then susceptible to be imported, that is why it is a good idea to place the project in its own directory, in this case the directory empty contains nothing else but the directory project.
The module project.setup is also importable.
I have this git repo structure:
.gitignore
JSONs/subdirA/some.json
JSONs/subdirB/other.json
MyPackage/__init__.py
MyPackage/myModule.py
How do I properly pack the JSONs folder into MyPackage/JSONs, without moving it there permanently (mostly because customers use this git repo directly for non-python usage, and the folder at the top of the repo is easy/intuitive... But now I also want to release this same dir into my PyPi package)?
I've tried adding it to the MANIFEST.in and then playing with data_files in setup.py as well as package_data... But to no avail. Maybe some .pyc or cached build files got the best of me... But I haven't figured it out from all the other (not quite) duplicate questions as they don't specifically call out their directory structure and desired final location.
I've tried resorting to os.walk and shutil.copy before the call to setup and then deleting that directory after setup... While it seems to work locally, when pushing to our local devpi package server, something goes wrong. Is my goal totally off the radar for setuptools/pip ideals??? Or am I just missing some key understanding? Please enlighten me!
Something like the following could help:
First we need to make sure that the json files are added to the source distribution.
MANIFEST.in:
recursive-include JSONs *.json
Then in the actual setup script, the list of packages has to be modified on the fly to take into account the target package structure.
setup.py:
#!/usr/bin/env python3
import setuptools
PACKAGES = (
setuptools.find_packages(exclude=['JSONs*'])
+
[
f'MyPackage.{package}'
for package
in setuptools.find_namespace_packages(include=['JSONs*'])
]
)
setuptools.setup(
packages=PACKAGES,
package_dir={
'MyPackage.JSONs': 'JSONs',
},
include_package_data=True,
#
name='Something',
version='1.2.3',
)
JSONs/subdirA/some.json :
{"Marco": "Polo"}
Such package data can be read like so:
MyPackage/myModule.py:
import pkgutil
print(pkgutil.get_data('MyPackage', 'JSONs/subdirA/some.json').decode())
And use it like in the following:
$ python -m pip install .
$ # Move to another directory to prevent that the current working directory
$ # ... overshadows the installed project
$ cd ..
$ python -m MyPackage.myModule
{"Marco": "Polo"}
I created project and helper modules for it. But some of modules are using each other like worker 1 uses helper1 and helper2 also uses helper1. So I completle confused how I need to import all those modules so so can work standalone (for example I want to debug helper2 out of main script) and they still will be functional. Summarizing - how to correctly import modules so maint_script will work and other modules when using out of main_script. Sorry for my English.
main program dir/
main_script.py
-classes/
|
|--helper1.py
|--helper2.py
-worker_classes/
|
|--worker1.py
At the moment I'am using this constructions in the begging of each script, but I feel that this approach isn't appropriate for python
import os
import sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), 'shell_modules')))
The way I deal with imports inside a project is to install the project in editable mode. This way, all files will be able to locate each other, always starting from your project root directory.
In order to do this, follow these steps:
1) write a setup.py file and add it to your project root folder - it doesn't need much info at all:
# setup.py
from setuptools import setup, find_packages
setup(name='MyPackageName', version='1.0.0', packages=find_packages())
2) install your package in editable mode (ideally from a virtual environment). From a terminal in your project folder, write
$ pip install -e .
Note the dot - this means "install the package from the current directory in editable mode".
3) your files are now able to locate each other, always starting from the project root. To import helper1.py, for example, you write:
from classes import helper1
or alternatively:
from classes.helper1 import foo, bar
This will be true to import helper1.py for any file, no matter where it is located in the project structure.
Like I said, you should use a virtual environment for this, so that pip does not install your package to your main Python installation (which could be messy if your project has many dependencies).
Currently my favorite tool for this is pipenv. When using it, replace the terminal command with
$ pipenv install -e .
So that your project gets added to the Pipfile.
I have a file structure like this:
project_folder/
notebooks/
notebook01.ipynb
notebook02.ipynb
...
notebookXY.ipynb
module01.py
module02.py
module03.py
In .ipynb files inside notebook/ folder I want to import classes and functions from module01.py, module02.py and module03.py.
I have found answer in this question that it is possible using following lines of code inside every notebook and run those lines as first cell every time:
import os
import sys
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
sys.path.append(module_path)
Is there please a better way for this? What if I have A LOT of .ipynb files inside notebooks/ folder, do I have to paste those lines of code at the beginning of every single one? Is there a better, more minimalist or cleaner way?
Try adding the project_folder to your PYTHONPATH environment variable. This will allow you to tell python to search that directory for imports.
You would do this in your user profile settings, or in your startup script - not in python. It's something that has to be set before python ever gets run.
Another solution is to move all your Python modules (.py files) into a folder and make them an installable package. If you pip install it into your current environment, you can then import the package into any notebook in that environment, regardless of folder structure.
So in your situation you could have:
project_folder/
notebooks/
notebook01.ipynb
notebook02.ipynb
...
notebookXY.ipynb
my_package/
__init__.py
module01.py
module02.py
module03.py
setup.py
__init__.py can just be an empty file, and tells Python "everything in this folder is part of a package"
For an explanation of what goes in setup.py see here.
A basic setup.py can be as simple as this:
import setuptools
setuptools.setup(
name="my_package",
version="0.0.1",
description="A small example package",
packages=setuptools.find_packages(),
python_requires='>=3.7',
)
Install it:
cd project_folder
pip install [-e] .
Including the optional -e flag will install my_package in "editable" mode, meaning that instead of copying the files into your virtual environment, a symlink will be created to the files where they are.
Now in any notebook you can do:
import my_package
Or
from my_package.module01 import <some object>