We Keep Coding

Python setuptools installs console scripts even if optional dependency are not installed

i am currently working on a package and am confused with setuptools. This package contains many dependencies and with these dependencies, multiple scripts can be executed via cli.
E.G.
> main_pkg
> main_pkg_which_needs_dep1
> main_pkg_which_needs_dep2
> ...
It is not necessary to have all scripts available on a system. Only the relevant ones. So i thought that i could simply modify my setup.py as follows:
...
entry_points=dict(console_scripts=[
'main_pkg = main_pkg.main_pkg:main ',
'main_pkg_which_needs_dep1 = main_pkg.main_pkg:main_dep1 [dep1]',
...
]),
...
extras_require={
"dep1": ["psycopg"],
"dep2": ["apsw"],
"dep3": ["numpy"],
...
},
And assummed if someone executes pip install main_pkg, that only main_pkg would be available in CLI. (Therefore, if executing pip install main_pkg[dep1], then there would be main_pkg and main_pkg_which_needs_dep1 available in CLI)
However, executing pip install main_pkg also makes all other console_scripts available through CLI, failing if executing e.g. main_pkg_which_needs_dep1 due to missing dependencies.
Is this behaviour expected by setuptools?
From the documentation i am reading the following:
It is up to the installer to determine how to handle the situation where PDF was not indicated (e.g. omit the console script, provide a warning when attempting to load the entry point, assume the extras are present and let the implementation fail later).
Also, if looking here, the documentation mentions the following:
In this case, the hello-world script is only viable if the pretty-printer extra is indicated, and so a plugin host might exclude that entry point (i.e. not install a console script) if the relevant extra dependencies are not installed.
Am i understanding the documentation correctly, that the installer (plugin host? --> pip?) has to handle this case, which is currently not working?
Or do i have to further modify the setup.py to achieve such a behaviour?
Thanks in advance!

I ran into this same problem. Based on this thread: https://github.com/pypa/pip/issues/9726, it does not look like you can optionally install console scripts.
However, this comment: https://github.com/pypa/pip/issues/9726#issuecomment-826381705 proposes a solution that may help you. I'll copy-paste it below.
Have myscript with the extra [cli] depends on myscript-cli the package and myscript-cli depends on myscript but contains the entrypoint to the console_script in the main package.
If you install myscript[cli] it requires myscript-cli package which then gets installed and that contains the entrypoint you wanted. This makes myscript[cli] or myscript-cli install both packages, but permits a myscript install that will not require the -cli package and thus will not provide the entrypoint.

Change .egg-info directory with pip install --editable

Is there a way to modify the location of the .egg-info directory that is generated upon:
pip install --editable .
I'm asking because I store my source code on (locally synchronized) cloud storage, and I want to install the package in editable mode on independent computers. So, ideally, the package directory would not be polluted with anything related to a given installation of the package.
I have tried using the --src option but this did not work; I don't understand what this option is meant to do.

You can achieve this by adding the egg_base option to setup.cfg:
[egg_info]
egg_base = relative/path/to/egg_info_folder
I have used this successfully in pip 19.3.1.
In my environment, the actual files that this altered are:
/anaconda/envs/my_env/lib/python3.6/site-packages/easy-install.pth
/anaconda/envs/my_env/lib/python3.6/site-packages/package_name.egg-link
Note, pip install raises an error if the egg_base base is not a relative path. But directly altering the files appears to work:
/anaconda/envs/my_env/lib/python3.6/site-packages/easy-install.pth:
/path/to/repository/folder
/anaconda/envs/my_env/lib/python3.6/site-packages/package_name.egg-link:
/path/to/egg_info/folder
/path/to/repository/folder/

Not sure if still relevant, but here is a setup.py based solution: https://jbhannah.net/articles/python-docker-disappearing-egg-info

Installing python module to python-version neutral location

I've built a debian package containing a python module.
The problem is that
dpkg-deb -c python-mymodule_1.0_i386.deb
show that all the files will be installed under
/usr/lib/python2.6/dist-packages/mymodule*
This means that the end-user who installs my deb package will need to be using exactly the same version of python as I am - yet I know that my module works fine on later versions too.
In my Makefile I have the following target:
install:
python setup.py install --root $(DESTDIR) $(COMPILE) --install-layout=deb
Where setup.py is
from distutils.core import setup
setup(name='mymodule',
version='1.0',
description='does my stuff',
author='Me',
author_email='myemail#localhost',
url='http://myurl/',
packages=['mymodule'],
)
Is there some way I can edit either the setup.py file or the Makefile so that the resulting module is installed in a python-version neutral directory instead of /usr/lib/python2.6?
Thanks,
Alex

I think I have found the answer now (thanks to the pointers from Tshepang):
In debian/rules you need to invoke dh_pysupport.
This grabs all the files installed by setup.py on the build machine in
$(DESTDIR)/usr/lib/pythonX.Y/dist-packages/mymodule*
and puts them into a non python-version specific location in the .deb file, namely
/usr/share/pyshared/mymodule*
finally, it adds a call to update-python-modules to the postinst script, which makes sure that the module is available on every version of python present on the target machine.

How do I write a setup.py for a twistd/twisted plugin that works with setuptools, distribute, etc?

The Twisted Plugin System is the preferred way to write extensible twisted applications.
However, due to the way the plugin system is structured (plugins go into a twisted/plugins directory which should not be a Python package), writing a proper setup.py for installing those plugins appears to be non-trivial.
I've seen some attempts that add 'twisted.plugins' to the 'packages' key of the distutils setup command, but since it is not really a package, bad things happen (for example, an __init__.py is helpfully added by some tools).
Other attempts seem to use 'package_data' instead (eg, http://bazaar.launchpad.net/~glyph/divmod.org/trunk/view/head:/Epsilon/epsilon/setuphelper.py), but that can also fail in weird ways.
The question is: has anyone successfully written a setup.py for installing twisted plugins which works in all cases?

I document a setup.py below that is needed only if you have users with pip < 1.2 (e.g. on Ubuntu 12.04). If everyone has pip 1.2 or newer, the only thing you need is packages=[..., 'twisted.plugins'].
By preventing pip from writing the line "twisted" to .egg-info/top_level.txt, you can keep using packages=[..., 'twisted.plugins'] and have a working pip uninstall that doesn't remove all of twisted/. This involves monkeypatching setuptools/distribute near the top of your setup.py. Here is a sample setup.py:
from distutils.core import setup
# When pip installs anything from packages, py_modules, or ext_modules that
# includes a twistd plugin (which are installed to twisted/plugins/),
# setuptools/distribute writes a Package.egg-info/top_level.txt that includes
# "twisted". If you later uninstall Package with `pip uninstall Package`,
# pip <1.2 removes all of twisted/ instead of just Package's twistd plugins.
# See https://github.com/pypa/pip/issues/355 (now fixed)
#
# To work around this problem, we monkeypatch
# setuptools.command.egg_info.write_toplevel_names to not write the line
# "twisted". This fixes the behavior of `pip uninstall Package`. Note that
# even with this workaround, `pip uninstall Package` still correctly uninstalls
# Package's twistd plugins from twisted/plugins/, since pip also uses
# Package.egg-info/installed-files.txt to determine what to uninstall,
# and the paths to the plugin files are indeed listed in installed-files.txt.
try:
from setuptools.command import egg_info
egg_info.write_toplevel_names
except (ImportError, AttributeError):
pass
else:
def _top_level_package(name):
return name.split('.', 1)[0]
def _hacked_write_toplevel_names(cmd, basename, filename):
pkgs = dict.fromkeys(
[_top_level_package(k)
for k in cmd.distribution.iter_distribution_names()
if _top_level_package(k) != "twisted"
]
)
cmd.write_file("top-level names", filename, '\n'.join(pkgs) + '\n')
egg_info.write_toplevel_names = _hacked_write_toplevel_names
setup(
name='MyPackage',
version='1.0',
description="You can do anything with MyPackage, anything at all.",
url="http://example.com/",
author="John Doe",
author_email="jdoe#example.com",
packages=['mypackage', 'twisted.plugins'],
# You may want more options here, including install_requires=,
# package_data=, and classifiers=
)
# Make Twisted regenerate the dropin.cache, if possible. This is necessary
# because in a site-wide install, dropin.cache cannot be rewritten by
# normal users.
try:
from twisted.plugin import IPlugin, getPlugins
except ImportError:
pass
else:
list(getPlugins(IPlugin))
I've tested this with pip install, pip install --user, and easy_install. With any install method, the above monkeypatch and pip uninstall work fine.
You might be wondering: do I need to clear the monkeypatch to avoid messing up the next install? (e.g. pip install --no-deps MyPackage Twisted; you wouldn't want to affect Twisted's top_level.txt.) The answer is no; the monkeypatch does not affect another install because pip spawns a new python for each install.
Related: keep in mind that in your project, you must not have a file twisted/plugins/__init__.py. If you see this warning during installation:
package init file 'twisted/plugins/__init__.py' not found (or not a regular file)
it is completely normal and you should not try to fix it by adding an __init__.py.

Here is a blog entry which describes doing it with 'package_data':
http://chrismiles.livejournal.com/23399.html
In what weird ways can that fail? It could fail if the installation of the package doesn't put the package data into a directory which is on the sys.path. In that case the Twisted plugin loader wouldn't find it. However, all installations of Python packages that I know of will put it into the same directory where they are installing the Python modules or packages themselves, so that won't be a problem.

Maybe you could adapt the package_data idea to use data_files instead: it wouldn’t require you to list twisted.plugins as package, as it uses absolute paths. It would still be a kludge, though.
My tests with pure distutils have told me that its is possible to overwrite files from another distribution. I wanted to test poor man’s namespace packages using pkgutil.extend_path and distutils, and it turns out that I can install spam/ham/__init__.py with spam.ham/setup.py and spam/eggs/__init__.py with spam.eggs/setup.py. Directories are not a problem, but files will be happily overwritten. I think this is actually undefined behavior in distutils which trickles up to setuptools and pip, so pip could IMO close as wontfix.
What is the usual way to install Twisted plugins? Drop-it-here by hand?

I use this approach:
Put '.py' and '.pyc' versions of your file to "twisted/plugins/" folder inside your package.
Note that '.pyc' file can be empty, it just should exist.
In setup.py specify copying both files to a library folder (make sure that you will not overwrite existing plugins!). For example:
# setup.py
from distutils import sysconfig
LIB_PATH = sysconfig.get_python_lib()
# ...
plugin_name = '<your_package>/twisted/plugins/<plugin_name>'
# '.pyc' extension is necessary for correct plugins removing
data_files = [
(os.path.join(LIB_PATH, 'twisted', 'plugins'),
[''.join((plugin_name, extension)) for extension in ('.py', '.pyc')])
]
setup(
# ...
data_files=data_files
)

How to build debian package with CPack to execute setup.py?

Until now, my project had only .cpp files that were compiled into different binaries and I managed to configure CPack to build a proper debian package without any problems.
Recently I wrote a couple of python applications and added them to the project, as well as some custom modules that I would also like to incorporate to the package.
After writing a setup.py script, I'm wondering how to add these files to the CPack configuration in a way that setup.py get's executed automatically when the user installs the package on the system with dpkg -i package.deb.
I'm struggling to find relevant information on how to configure CPack to install custom python applications/modules. Has anyone tried this?

I figured out a way to do it but it's not very simple. I'll do my best to explain the procedure so please be patient.
The idea of this approach is to use postinst and prerm to install and remove the python application from the system.
In the CMakeLists.txt that defines the project, you need to state that CPACK is going to be used to generate a .deb package. There's some variables that need to be filled with info related to the package itself, but one named CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA is very important because it's used to specify the location of postinst and prerm, which are standard scripts of the debian packaging system that are automatically executed by dpkg when the package is installed/removed.
At some point of your main CMakeLists.txt you should have something like this:
add_subdirectory(name_of_python_app)
set(CPACK_COMPONENTS_ALL_IN_ONE_PACKAGE 1)
set(CPACK_PACKAGE_NAME "fake-package")
set(CPACK_PACKAGE_VENDOR "ACME")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "fake-package - brought to you by ACME")
set(CPACK_PACKAGE_VERSION "1.0.2")
set(CPACK_PACKAGE_VERSION_MAJOR "1")
set(CPACK_PACKAGE_VERSION_MINOR "0")
set(CPACK_PACKAGE_VERSION_PATCH "2")
SET(CPACK_SYSTEM_NAME "i386")
set(CPACK_GENERATOR "DEB")
set(CPACK_DEBIAN_PACKAGE_MAINTAINER "ACME Technology")
set(CPACK_DEBIAN_PACKAGE_DEPENDS "libc6 (>= 2.3.1-6), libgcc1 (>= 1:3.4.2-12), python2.6, libboost-program-options1.40.0 (>= 1.40.0)")
set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_SOURCE_DIR}/name_of_python_app/postinst;${CMAKE_SOURCE_DIR}/name_of_python_app/prerm;")
set(CPACK_SET_DESTDIR "ON")
include(CPack)
Some of these variables are optional, but I'm filling them with info for educational purposes.
Now, let's take a look at the scripts:
postinst:
#!/bin/sh
# postinst script for fake_python_app
set -e
cd /usr/share/pyshared/fake_package
sudo python setup.py install
prerm:
#!/bin/sh
# prerm script
#
# Removes all files installed by: ./setup.py install
sudo rm -rf /usr/share/pyshared/fake_package
sudo rm /usr/local/bin/fake_python_app
If you noticed, script postinst enters at /usr/share/pyshared/fake_package and executes the setup.py that is laying there to install the app on the system. Where does this file come from and how it ends up there? This file is created by you and will be copied to that location when your package is installed on the system. This action is configured in name_of_python_app/CMakeLists.txt:
install(FILES setup.py
DESTINATION "/usr/share/pyshared/fake_package"
)
install(FILES __init__.py
DESTINATION "/usr/share/pyshared/fake_package/fake_package"
)
install(FILES fake_python_app
DESTINATION "/usr/share/pyshared/fake_package/fake_package"
)
install(FILES fake_module_1.py
DESTINATION "/usr/share/pyshared/fake_package/fake_package"
)
install(FILES fake_module_2.py
DESTINATION "/usr/share/pyshared/fake_package/fake_package"
)
As you can probably tell, besides the python application I want to install there's also 2 custom python modules that I wrote that also need to be installed. Below I describe the contents of the most important files:
setup.py:
#!/usr/bin/env python
from distutils.core import setup
setup(name='fake_package',
version='1.0.5',
description='Python modules used by fake-package',
py_modules=['fake_package.fake_module_1', 'fake_package.fake_module_2'],
scripts=['fake_package/fake_python_app']
)
_init_.py: is an empty file.
fake_python_app : your python application that will be installed in /usr/local/bin
And that's pretty much it!

A setup.py file is the equivalent of the configure && make && make install dance for a standard unix source distribution and as such is inappropriate to run as a part of a distributions package install process. See this discussion of the different ways to include Python modules in a .deb package.