We Keep Coding

Question about Packaging in Python with pip

I saw this nice explanation video (link) of packaging using pip and I got two questions:
The first one is:
I write a code which I want to share with my colleagues, but I do not aim to share it via pypi. Thus, I want to share it internally, so everyone can install it within his/ her environment.
I actually needn't to create a wheel file with python setup.py bdist_wheel, right? I create the setup.py file and I can install it with the command pip install -e . (for editable use), and everyone else can do it so as well, after cloning the repository. Is this right?
My second question is more technical:
I create the setup.py file:
from setuptools import setup
setup(
name = 'helloandbyemate',
version = '0.0.1',
description="Say hello in slang",
py_modules=['hellomate'],
package_dir={"": "src"}
)
To test it, I write a file hellomate.py which contains a function printing hello, mate!. I put this function in src/. In the setup.py file I put only this module in the list py_modules. In src/ is another module called byemate.py. When I install the whole module, it installs the module byemate.py as well, although I only put hellomate in the list of py_modules. Has anyone an explanation for this behaviour?

I actually needn't to create a wheel file ... everyone else can do it so as well, after cloning the repository. Is this right?
This is correct. However, the installation from source is slower, so you may want to publish wheels to an index anyway if you would like faster installs.
When I install the whole module, it installs the module byemate.py as well, although I only put hellomate in the list of py_modules. Has anyone an explanation for this behaviour?
Yes, this is an artifact of the "editable" installation mode. It works by putting the src directory onto the sys.path, via a line in the path configuration file .../lib/pythonX.Y/site-packages/easy-install.pth. This means that the entire source directory is exposed and everything in there is available to import, whether it is packaged up into a release file or not.
The benefit is that source code is "editable" without reinstallation (adding/removing/modifying files in src will be reflected in the package immediately)
The drawback is that the editable installation is not exactly the same as a "real" installation, where only the files specified in the setup.py will be copied into site-packages directly
If you don't want other files such as byemate.py to be available to import, use a regular install pip install . without the -e option. However, local changes to hellomate.py won't be reflected until the installation step is repeated.
Strict editable installs
It is possible to get a mode of installation where byemate.py is not exposed at all, but live modifications to hellomate.py are still possible. This is the "strict" editable mode of setuptools. However, it is not possible using setup.py, you have to use a modern build system declaration in pyproject.toml:
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "helloandbyemate"
version = "0.0.1"
description = "Say hello in slang"
[tool.setuptools]
py-modules = ["hellomate"]
include-package-data = false
[tool.setuptools.package-dir]
"" = "src"
Now you can perform a strict install with:
pip install -e . --config-settings editable_mode=strict

Difference between installation of pip git+https and python setup.py

I am aware of this popular topic, however I am running into a different outcome when installing a python app using pip with git+https and python setup.py
I am building a docker image. I am trying to install in an image containing several other python apps, this custom webhook.
Using git+https
RUN /venv/bin/pip install git+https://github.com/alerta/alerta-contrib.git#subdirectory=webhooks/sentry
This seems to install the webhook the right way, as the relevant endpoint is l8r discoverable.
What is more, when I exec into the running container and doing a search for relevant files, I see the following
./venv/lib/python3.7/site-packages/sentry_sdk
./venv/lib/python3.7/site-packages/__pycache__/alerta_sentry.cpython-37.pyc
./venv/lib/python3.7/site-packages/sentry_sdk-0.15.1.dist-info
./venv/lib/python3.7/site-packages/alerta_sentry.py
./venv/lib/python3.7/site-packages/alerta_sentry-5.0.0-py3.7.egg-info
In my second approach I just copy this directory locally and in my Dockerfile I do
COPY sentry /app/sentry
RUN /venv/bin/python /app/sentry/setup.py install
This does not install the webhook appropriately and what is more, in the respective container I see a different file layout
./venv/lib/python3.7/site-packages/sentry_sdk
./venv/lib/python3.7/site-packages/sentry_sdk-0.15.1.dist-info
./venv/lib/python3.7/site-packages/alerta_sentry-5.0.0-py3.7.egg
./alerta_sentry.egg-info
./dist/alerta_sentry-5.0.0-py3.7.egg
(the sentry_sdk - related files must be irrelevant)
Why does the second approach fail to install the webhook appropriately?
Should these two option yield the same result?

What finally worked is the following
RUN /venv/bin/pip install /app/sentry/
I don't know the subtle differences between these two installation modes
I did notice however that /venv/bin/python /app/sentry/setup.py install did not produce an alerta_sentry.py but only the .egg file, i.e. ./venv/lib/python3.7/site-packages/alerta_sentry-5.0.0-py3.7.egg
On the other hand, /venv/bin/pip install /app/sentry/ unpacked (?) the .egg creating the ./venv/lib/python3.7/site-packages/alerta_sentry.py
I don't also know why the second installation option (i.e. the one creating the .egg file) was not working run time.

Can't find submodule within package

I feel a bit dumb asking this since there are a lot of similar questions, but I've honestly searched around a lot and could not find a solution for this. Here goes:
I have a Python package (on TestPyPi, here is the source code, note that it uses and needs python3.8) with the following structure:
paillier/
setup.py
test/
paillier/
__init__.py
keygen.py
util/
__init__.py
math_shortcuts.py
My use case is: in keygen.py, I want to use util/math_shortcuts.py.
So, in keygen.py, I have the following import:
from paillier.util.math_shortcuts import generate_coprime, lcm, get_mu.
However, when I try to use my package (by doing from paillier.keygen import generate_keys), I am greeted with the Error ModuleNotFoundError: No module named 'paillier.util'
This ModuleNotFoundError is always present when I install it using pip from TestPyPi, but it doesn't happen when I build the package locally: when I run pip install -e . in the paillier/ directory (where setup.py lives), I can run from paillier.keygen import generate_keys, even when my working directory is somewhere else.
I've tried to do from .util.math_shortcuts ..., or from util.math_shortcuts ..., or from paillier.paillier.util.math_shortcuts ..., but all to no avail.
In short, when doing pip install --index-url <TestPyPi> rens-paillier my files can't seem to find the submodules.
However, when doing pip install -e . in the outer paillier/ dir, it seems to work.

I ran into the same problem. (my -e installation worked, normal installation didn't) My solution is, to actually name the subpackages in the setup.py.
packages=['paillier', 'paillier.util']
While this works, I am not sure why :D

I found that removing the packages parameter from the setup function call in the setup.py file resolved a similar issue I had with my module not resolving submodules.

looking around for the same question (credits to this post), the easiest is probably to use
setuptools.find_packages()
some optional arguments:
where="src" for package structures where the files are in src/
exclude=["*-old"] to exclude some packages (in my case, the my_package-old that I want to keep until my refactoring is done)

How to compare requirement file and actually installed Python modules?

Given requirements.txt and a virtualenv environment, what is the best way to check from a script whether requirements are met and possibly provide details in case of mismatch?
Pip changes it's internal API with major releases, so I seen advices not to use it's parse_requirements method.
There is a way of pkg_resources.require(dependencies), but then how to parse requirements file with all it's fanciness, like github links, etc.?
This should be something pretty simple, but can't find any pointers.
UPDATE: programmatic solution is needed.

You can save your virtualenv's current installed packages with pip freeze to a file, say current.txt
pip freeze > current.txt
Then you can compare this to requirements.txt with difflib using a script like this:
import difflib
req = open('requirements.txt')
current = open('current.txt')
diff = difflib.ndiff(req.readlines(), current.readlines())
delta = ''.join([x for x in diff if x.startswith('-')])
print(delta)
This should display only the packages that are in 'requirements.txt' that aren't in 'current.txt'.

Got tired of the discrepancies between requirements.txt and the actually installed packages (e.g. when deploying to Heroku, I'd often get ModuleNotFoundError for forgetting to add a module to requirements.)
This helps:
Use compare-requirements (GitHub)
(you'll need to pip install pipdeptree to use it.)
It's then as simple as...
cmpreqs --pipdeptree
...to show you (in "Input 2") which modules are installed, but missing from requirements.txt.
You can then examine the list and see which ones should in fact be added to requirements.txt.

How do I install modules on qpython3 (Android port of python)

I found this great module on within and downloaded it as a zip file. Once I extracted the zip file, i put the two modules inside the file(setup and the main one) on the module folder including an extra read me file I needed to run. I tried installing the setup file but I couldn't install it because the console couldn't find it. So I did some research and I tried using pip to install it as well, but that didn't work. So I was wondering if any of you could give me the steps to install it manually and with pip (keep in mind that the setup.py file needs to be installed in order for the main module to work).
Thanks!

The cleanest and simplest way I have found is to use pip from within QPython console as in This Answer
import pip
pip.main(['install', 'networkx'])

Step1: Install QPython.
Step2: Install AIPY for QPython.
Step3: Then go to QPython-->QPYPI-->AIPY and install from there Numpy, SciPy, Matplotlib, openCV etc.

Extract the zip file to the site-packages folder.
Find the qpyplus folder in that Lib/python3.2/site-packages extract here that's it.Now you can directly use your module from REPL terminal by importing it.