We Keep Coding

Python package raises ModuleNotFoundError on calling an entry-point except if installed in editable mode

I have a Python package that at first appears to install just fine, but when calling on one of the entry points raises a ModuleNotFoundException. The module is otherwise found just fine with both import package from the interactive interpreter as well as with python -m package.etc. But if I try to call on the entry-point directly (flike python -m package.etc.main) it will raise an AttributeError saying that the module has no attribute __path__.
I can see the package if I do pip list.
The project is currently set up with the "template" pyproject.toml and only setup.cfg, but the behaviour is essentially the same (the traceback looks slightly different but the error is the same) when using setup.py over pyproject.toml, both with pip but also if I invoke setup.py directly. The structure of the project is:
package
├── __init__.py
├── cli
│   ├── __init__.py
│   ├── entry.py
├── file.py
I get the same behaviour if doing this in a virtual environment as when I do it with a userspace (--user) install.
Modifying the environment variable ${PYTHONPATH} fixes the issue, and installing the package in editable mode works just fine.

Turns out that the issue was that I had something like:
[options]
packages = find:
[options.packages.find]
include =
README.md
in my setup.cfg, and it appears as if the declaration of include was exclusive which led to the package not being included in the installation, which still worked when installed in editable mode (presumably because editable mode only sets up some sort of links or appends the source code directories to some path).

How to integrate Sphinx Makefile into existing project Makefile?

I have an existing project with the following directory structure:
docs/
conf.py
Makefile
index.rst
documentation-foo.rst
documentation-bar.rst
src/
...code...
common.mk
Makefile
README.rst
In the root of the repository, the Makefile contains various useful commands related to making, testing, packaging, and deploying the code. Likewise, common.mk includes a check to make sure the user has several utility programs available on their system (e.g., jq for manipulating/displaying JSON) that cannot be installed with pip. I can issue commands like make lint or make test and those are defined in that Makefile. (The top of the Makefile also has an include common.mk, see below for Makefile contents.)
In the docs/ folder, the Makefile is automatically generated by Sphinx. This Makefile is standalone and is totally independent of the Makefile or common.mk files in the root of the repository. I can issue commands like make html and it will use Sphinx to generate the documentation in HTML format.
Contents of (root) /Makefile:
include common.mk
clean:
...
lint:
...
test:
....
Contents of (root) /common.mk:
ifeq ($(shell which jq),)
$(error Please install jq using "apt-get install jq" or "brew install jq")
endif
Contents of (sphinx) /docs/Makefile:
# Minimal makefile for Sphinx documentation
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
...
My primary question is this: how can I incorporate the rules from the sphinx Makefile at /docs/Makefile so that they are available from the root level /Makefile?
For example, to make HTML documentation, I currently have to cd docs && make html from the root of the repo, but I would like to be able to run make html from the root of the repo and have the Makefile in the root of the repo resolve the target to the html target defined in docs/Makefile and use the rule defined in that Makefile.
Second, related question: if I have commands in docs/Makefile that have relative paths, must the paths be updated to be relative to the location where the make command is being run (the root of the repo instead of docs/), or does make resolve relative links to be relative to the Makefile's location in docs/?

You can do this with recursive make:
html:
$(MAKE) -C docs html
Some people strongly dislike recursive make. You can read the arguments against it, but I don't believe they apply to your use of make, since you're not trying to express dependencies.

Ansible - generate .rst file from module's DOCUMENTATION string

I have written custom Ansible module and documented it using standard Ansible convention, i.e. by writing DOCUMENTATION and EXAMPLES global strings in module file.
I already have some of the documentation generated using Sphinx 1.8.3 and hosted locally. I would like to have Ansible documenation included in Sphinx generated pages. My directory structure is fairly simple:
./ansible/docs
├── conf.py
├── index.rst
├── _static
└── _templates
./ansible/library/
├── __init__.py
└── module.py
Now, I could write documentation as function docstrings and then include it using Sphinx .. automodule:: directive. This works, but uses different format than Ansible DOCUMENTATION string.
Although Ansible module documentation goes on in depth, how the docstrings should be formatted, it does not seem to provide any information how to generate docs locally.
What is the correct way to convert Ansible module documentation to .rst file, so that it could be included by Sphinx?

By using the provided Makefile in the docs/docsite directory (you can also run make webdocs from the top-level). You'll want to ensure you have loaded the docsite requirements into your virtualenv, in addition to pip install -e $PWD or its equivalent because the docsite sphinx uses some of ansible's own libraries to do its work.

How does one handle multiple modules/packages in python?

I've spent hours researching this problem, and I'm still baffled. Please find my ignorance charming.
I'm building a python program that will allow me to pit two AIs against each other in a game of battleship.
Here's my directory structure:
.
├── ais_play_battleship
│   ├── game.py
│   ├── __init__.py
│   ├── player.py
│   └── ship.py
├── LICENSE
├── README.md
└── tests
└── ship_test.py
2 directories, 7 files
Currently, I'm trying to write ship_test.py, but I cannot seem to import ais_play_battleship.ship. I get the dreaded "ModuleNotFoundError"
Here's what my research has taught me about my problem:
If you want to import python code from another directory, you should make that directory a package instead of a module. Therefore, I've placed an __init__.py file in the root of ais_play_battleship.
Python will only search the directory python is launched from as well as the directory of the script you're running. Therefore, I've been trying to launch my tests by running python3 tests/ship_tests.py from the root directory.
Here are my specific questions:
Why is the error a "ModuleNotFound" error? Shouldn't it be "PackageNotFound"?
Am I correct to make ais_play_battleship a package?
How can I keep my tests in a separate directory and still use the code in ais_play_battleship?
Please forgive me, as I'm not very good at asking questions on StackOverflow. Please tell me how I can improve.

I am answering my own question, as I haven't yet received a satisfactory answer. The best resource I've found is available here. In summary:
Python does NOT search the directory you run python from for modules. Furthermore, adding an __init__.py file to make a directory a package is not enough to make it visible to an instance of python running in another folder. You must also install that package. Therefore, the only two ways to access a module in another directory are:
Install the packaged module in site-packages (this requires the creation of a setup.py file and installation using pip install . More information is available here.
Modify path to resolve the module
I ended up settling with the second option, for reasons discussed below.
The first option requires one to reinstall the package at every change to a package. This is difficult on a constantly-changing codebase, but can be made easier by using build automation. However, I'd like to avoid this added complexity.
I shied away from the second option for a long time, because it seemed that modifying the path would require hard-coding the absolute path to my module, which is obviously unacceptable, as every developer would have to edit that path to fit their environment. However, this guide provides a solution to this problem. Create a ./tests/context.py file with the following contents:
import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
Then, in my ship_test.py module, I imported the context and the module I needed:
import context
import ais_play_battleship.ship
# (I include the submodule ship because ais_play_battleship itself does not have
# any attributes or methods, and the submodule ship is the only one I am testing
# in ship_test.py)
This fits my project better, because it works as expected without having to worry about installing my package (or the method by which my package was installed).

To solve this problem without relying on hacking about your sys.path, create a setup.py file and as a build step for your test runner, have it run pip install . first. You might want to use a tool like tox.
In the top level directory:
setup.py
from setuptools import setup
setup(name='ais_play_battleship')
tox.ini
[tox]
envlist = py36, py37
[testenv]
deps=pytest
commands=
pip install . --quiet
py.test -q
then run your tests (in this example we use tox to do this so that we can also configure how the test environment can be configured) : tox

Run tests/ship_test.py as a module
python -m tests.ship_test

Python PEX loading

I've been trying to wrap my head around the python pex utility (https://pex.readthedocs.org/en/latest/) for bundling some applications into .pex files for deployment.
My app isn't large enough to require twitters pants build tool, as well as I have some build requirements which pants doesn't address. I did however, try the pants tools build system using python_binary which resulted in pex files with the sources loaded into the pex files root. The BUILD files in pants accept a sources property for python_binary which can be a glob for files within the directory the build is running in, however, pants is using the pex programmatic API and not the command-line utility.
The problem is when I use the pex command-line utility on it's own, it seems to want distributions (i.e. folders set up with a setup.py, etc...) and wants to install my code into the .deps folder in the pex file rather than just copying the python files into the root of the pex file like pants seems to do.
Is copying the files over (not installing a package) not possible through the command-line pex tool?

As of pex 1.0.0 there is no facility to glob up files and directories directly, you must have a setup.py as you suggest, or use pants (no longer Twitter's by the way - independent).
So you have 3 paths forward (#1 you already know, but spelling it out for others):
Create a setup.py and point the pex tool at its dir
$ tree -h
.
├── [4.0K] lib
│   ├── [ 0] __init__.py
│   ├── [ 38] lib.py
│   └── [ 68] main.py
└── [ 76] setup.py
1 directory, 4 files
$ cat lib/lib.py
def func():
return 'func in lib'
$ cat lib/main.py
from .lib import func
if __name__ == '__main__':
print(func())
$ cat setup.py
from setuptools import setup
setup(
name='lib',
packages=['lib']
)
$ pex . -e lib.main -o lib.pex
$ ./lib.pex
func in lib
NB: The . in the pex command line is the bit pointing pex at this dir's setup.py
File an issue against pex to support a set of files in place of a requirement / setup.py.
You can do that here.
File issues against pants to support the requirements you have that it does not address.
You can do that here
As a pants committer I can say that we're working towards making pants easier and easier to use such that no project is too small. You should be able to pip install pantsbuild.pants.backend.python && touch pants.ini and be up and running in a python-only repo but we're not there today.