We Keep Coding

Sphinx index.rst inside directory

The path to the master doc of a Sphinx build can be specified in conf.py. However, this directory path is reflected in the generated HTML, and shows for example in Read The Docs as a missing index. I'd like to use this alternative path to structure my project cleanly - to have configuration at the top level and documentation inside src, but have the build essentially get rid of it.
So here's what I had previously:
docs
conf.py
index.rst
things
doc1.rst
doc2.rst
This works, but when lots of files are added to the top level, it gets messy with Sphinx's makefiles etc. I'd like to have this instead:
docs
conf.py
src
index.rst
things
doc1.rst
doc2.rst
Which builds, but index.html is in build/html/src instead of build/html. I was surprised to find no information on this, other than the fact that master_doc in conf.py controls the location and name of the main file. How could I get my documentation to be built to build/html?

There is an option for specifying the location of the configuration file: -c.
# conf.py
master_doc = 'index'
# structure
docs
conf.py
src
index.rst
...
Then run sphinx-build -b html -c . src build/html. However, this solution indeed needs control over the build command, which is not available in Read The Docs. And it seems that -c doesn't work in Sphinx's own makefiles either (with -M instead of -b).

Running a pytest test from CMake where the test and sources are in different folders

I have a CMake based project that uses both C++ and python sources. Tests are added via add_test for all C++ tests in a folder structure like:
src
mynamespace/foo.cpp
mypyspace/mypkg/__init__.py
mypyspace/mypkg/bar.py
test
mynamespace/testFoo.cpp
mypyspace/testBar.py
In testBar.py I have a import mypkg and want to run this with add_test(... COMMAND pytest WORKING_DIRECTORY .../test)
Of course it does not find my package, but I also do not want to install it (it is required to generate some stuff during build and is actively developed as part of the other sources).
What is the correct/preferred way of executing those tests?
Notes:
I don't want to move the sources out of src or tests out of test
set_tests_properties may set environment variables.
I think setting PYTHONPATH=.../src/mypyspace should work, but that would also ignore changes to the users PYTHONPATH.

There are 2 options:
Use PYTHONPATH
add_test (NAME python-tests
COMMAND ${PYTHON_EXECUTABLE} -m pytest # Or just COMMAND pytest
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
)
set_tests_properties(python-tests
PROPERTIES ENVIRONMENT "PYTHONPATH=${PROJECT_SOURCE_DIR}/src/mypyspace:$ENV{PYTHONPATH}")
Invoke inside of the src directory
add_test (NAME python-tests
COMMAND ${PYTHON_EXECUTABLE} -m pytest ${CMAKE_CURRENT_SOURCE_DIR}
WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/src/mypyspace
)
The first approach has the disadvantage, that the user might configure and later change its environment so that PYTHONPATH is different (e.g. loading modules in HPC clusters) and find that the tests incorrectly succeed.
The 2nd is the most flexible one. It relies on information from https://docs.pytest.org/en/latest/pythonpath.html. Most important part:
the [python -m pytest] call will add the current directory to sys.path.
So we only have to set the current directory when running the tests to the source directory.
Note: For other alternatives specific to unittest (the module) you may also find something in Running unittest with typical test directory structure

How to run tests without installing package?

I have some Python package and some tests. The files are layed out following http://pytest.org/latest/goodpractices.html#choosing-a-test-layout-import-rules
Putting tests into an extra directory outside your actual application
code, useful if you have many functional tests or for other reasons
want to keep tests separate from actual application code (often a good
idea):
setup.py # your distutils/setuptools Python package metadata
mypkg/
__init__.py
appmodule.py
tests/
test_app.py
My problem is, when I run the tests py.test, I get an error
ImportError: No module named 'mypkg'
I can solve this by installing the package python setup.py install but this means the tests run against the installed package, not the local one, which makes development very tedious. Whenever I make a change and want to run the tests, I need to reinstall, else I am testing the old code.
What can I do?

I know this question has been already closed, but a simple way I often use is to call pytest via python -m, from the root (the parent of the package).
$ python -m pytest tests
This works because -m option adds the current directory to the python path, and hence mypkg is detected as a local package (not as the installed).
See:
https://docs.pytest.org/en/latest/usage.html#calling-pytest-through-python-m-pytest

The normal approach for development is to use a virtualenv and use pip install -e . in the virtualenv (this is almost equivalent to python setup.py develop). Now your source directory is used as installed package on sys.path.
There are of course a bunch of other ways to get your package on sys.path for testing, see Ensuring py.test includes the application directory in sys.path for a question with a more complete answer for this exact same problem.

On my side, while developing, I prefer to run tests from the IDE (using a runner extension) rather than using the command line. However, before pushing my code or prior to a release, I like to use the command line.
Here is a way to deal with this issue, allowing you to run tests from both the test runner used by your IDE and the command line.
My setup:
IDE: Visual Studio Code
Testing: pytest
Extension (test runner): https://marketplace.visualstudio.com/items?itemName=LittleFoxTeam.vscode-python-test-adapter
Work directory structure (my solution should be easily adaptable to your context):
project_folder/
src/
mypkg/
__init__.py
appmodule.py
tests/
mypkg/
appmodule_test.py
pytest.ini <- Use so pytest can locate pkgs from ./src
.env <- Use so VsCode and its extention can locate pkgs from ./src
.env:
PYTHONPATH="${PYTHONPATH};./src;"
pytest.ini (tried with pytest 7.1.2):
[pytest]
pythonpath = . src
./src/mypkg/appmodule.py:
def i_hate_configuring_python():
return "Finally..."
./tests/mypkg/appmodule_test.py:
from mypkg import app_module
def test_demo():
print(app_module.i_hate_configuring_python())
This should do the trick

Import the package using from .. import mypkg. For this to work you will need to add (empty) __init__.py files to the tests directory and the containing directory. py.test should take care of the rest.

How to generate index.rst alone or generate everything excluding conf.py by sphinx-apidocs?

I understand that the following generates rst for all modules, excluding the index.rst
sphinx-apidoc -f -o very_good_docs/ very_good
And the following generates everything, including index.rst, conf.py, Makefile, etc
sphinx-apidoc -F -f -o very_good_docs/ very_good
The problem is that sphinx-apidoc does not generate a correct conf.py, so I have to always manually modify conf.py to include the correct sys.paths. If I modify my python code in "very_good" folder, I should run the sphinx-apidoc command without "-F", so conf.py is preserved. However, if I add a new module under very_good directly, index.rst is not updated without the "-F" option, which means my new module will not be in the doc. I guess the solution is to either someone generates the index.rst file only, or using "-F" option without overriding the conf.py. Is there a way to do it?

This is something I'm experiencing right now. It's really frustrating not finding information to achieve something so common.
My approach is to use a custom config.py and specifying it always I execute sphinx-build with the -F option. This way my config.py is not overwritten even if a new (and useless) config.py is created.
In order to specify a custom config.py you should use the -c option as stated in the documentation:
-c path
Don’t look for the conf.py in the source directory, but use the given configuration directory instead. Note that various other files and paths given by configuration values are expected to be relative to the configuration directory, so they will have to be present at this location too.
New in version 0.3.
The command will look like this:
sphinx-build -b html -c %sphinxConfigDir% %sourceCode% %buildDir%

How do you get Python documentation in Texinfo Info format?

Since Python 2.6, it seems the documentation is in the new reStructuredText format, and it doesn't seem very easy to build a Texinfo Info file out of the box anymore.
I'm an Emacs addict and prefer my documentation installed in Info.
Does anyone have Python 2.6 or later docs in Texinfo format? How did you convert them? Or, is there a maintained build somewhere out there?
I know I can use w3m or haddoc to view the html docs - I really want them in Info.
I've played with Pandoc but after a few small experiments it doesn't seem to deal well with links between documents, and my larger experiment - running it across all docs cat'ed together to see what happens - is still chugging along two days since I started it!
Two good answers
Highlighting two answers below, because SO won't allow me to accept both answers:
#wilfred-hughes: Installing from MELPA is the quickest way to get pre-build info into Emacs
#alioth: Building it yourself looks like it's a lot easier than when I asked this question in 2009

Jon Waltman http://bitbucket.org/jonwaltman/sphinx-info has forked sphinx and written a texinfo builder, it can build the python documentation (I've yet done it). It seems that it will be merged soon into sphinx.
Here's the quick links for the downloads (temporary):
http://dl.dropbox.com/u/1276730/python.info
http://dl.dropbox.com/u/1276730/python.texi
Steps to generate python doc in texinfo format:
Download the python source code
Download and install the sphinx-info package (in a virtualenv)
Enter in the Python/Doc directory from the python sources
Edit the Makefile, to the build target replace $(PYTHON) tools/sphinx-build.py with sphinx-build, then add this target to the makefile, pay attention, the space before echo is a TAB:
texinfo: BUILDER = texinfo
texinfo: build
#echo
#echo "Build finished. The Texinfo files are in _build/texinfo."
#echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
Edit the Python/Doc/conf.py adding:
texinfo_documents = [
('contents', 'python', 'Python Documentation', 'Georg Brandl',
'Python', 'The Python Programming Language', 'Documentation tools',
1),
]
Then run make texinfo and it should produce the texifile in the build/texinfo directory.
To generate the info file run makeinfo python.texi

I've packaged up the Python docs as a texinfo file.
If you're using Emacs with MELPA, you can simply install this with M-x package-install python-info.

With no doubt it would be cool and challenging to generate the Python documentation on your particular Python version by yourself. Just follow EmacsWiki, or feel free to compile it locally (at Debian Jessy for Python3.4.2):
sudo apt-get install python3-sphinx
cd ~/Desktop
wget https://www.python.org/ftp/python/3.4.2/Python-3.4.2rc1.tar.xz
tar -xf Python-3.4.2rc1.tar.xz
cd Python-3.4.2rc1/Doc/
sphinx-build -b texinfo -d build/doctrees . build/texinfo
# extra time to build
cd build/texinfo/
makeinfo python.texi
# extra time for convertation
I got this tree:
.
├── logging_flow.png
├── Makefile
├── pathlib-inheritance.png
├── python.info
├── python.info-1
├── python.info-10
├── python.info-11
├── python.info-12
├── python.info-13
├── python.info-14
├── python.info-15
├── python.info-16
├── python.info-17
├── python.info-18
├── python.info-19
├── python.info-2
├── python.info-20
├── python.info-21
├── python.info-22
├── python.info-23
├── python.info-24
├── python.info-25
├── python.info-26
├── python.info-27
├── python.info-28
├── python.info-29
├── python.info-3
├── python.info-30
├── python.info-31
├── python.info-32
├── python.info-33
├── python.info-34
├── python.info-4
├── python.info-5
├── python.info-6
├── python.info-7
├── python.info-8
├── python.info-9
├── python.texi
├── python-video-icon.png
├── tulip_coro.png
└── turtle-star.png
And now it is possible to review python documentation natively in Emacs by
C-u C-h i python-info RET
python-info is a filename (fourth in the tree above), and even to bookmark some arbitrary nodes for habitual and regular reviewing convenience.

For those following this question in the hope of an answer, I found another rst2texinfo implementation which you might like to try:
http://bitbucket.org/jonwaltman/rst2texinfo/src

Another "workaround" is to execute pydoc as suggested by Nikokrock directly in Emacs:
(defun pydoc (&optional arg)
(interactive)
(when (not (stringp arg))
(setq arg (thing-at-point 'word)))
(setq cmd (concat "pydoc " arg))
(ad-activate-regexp "auto-compile-yes-or-no-p-always-yes")
(shell-command cmd)
(setq pydoc-buf (get-buffer "*Shell Command Output*"))
(switch-to-buffer-other-window pydoc-buf)
(python-mode)
(ad-deactivate-regexp "auto-compile-yes-or-no-p-always-yes")
)

Michael Ernst used to maintain Info formats of Python docs:
http://www.cs.washington.edu/homes/mernst/software/#python-info
You can try using his makefile and html2texi script to generate an updated version. Both are linked at the above URL. I'm not sure how well it works now (the last version was around 2001), but his script is well commented (grep for "python").

Python docs are now generated using Sphynx framework. This framework does not have texinfo output format. Currently it has:
HTML
latex
plain text
Maybe you can get what you want using the Latex output. With the text output you will lost the cross ref.
Personnaly I prefer using pydoc when I want textual output. With Vim I have a shorcut to call pydoc and open a window with the doc for the entity under my cursor...

For Python 3.8.0 and later, pre-built Info files are available at https://www.python.org/ftp/python/doc and/or https://docs.python.org/3/archives/.

The Ubuntu distribution provides packages pythonX.Y-doc (which include the documentation in Info format) at least since 18.04 (bionic); in 19.04 X.Y stands for 2.7, 3.7 and 3.8. The package does not have many dependencies, I assume it is possible to install it in other distributions too.

Believe it or not, the Python project actually provides us a way to do this through various Makefiles. The files utilize the Python Sphinx project to generate a texi file which makeinfo can then convert to info, the format Emacs uses for documentation.
In addition to Python3000, these instructions require GNU Make and Texinfo. These are packaged in most Linux distributions. Different distros may use different naming conventions. Refer to your distro's documentation for the corresponding package names. For Debian based distros:
# install make to utilize the Makefiles provided by the Python project
~/$ sudo apt-get install make
# install texinfo for the `makeinfo` command
~/$ sudo apt-get install texinfo
Package names are usually similar for non-Debian systems. For Windows users, I recommend WSL or creating a virtual machine.
1. Download the documentation
Navigate to https://www.python.org/ftp/python/ and download the tarball for your Python version. It will look like:
https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tar.xz
You can use wget to download the tarball and tar to unpack it. The options x and f are for "extract file":
# download the tarball
~/$ wget https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tar.xz
# extract the tarball
~/$ tar xf Python-3.7.9.tar.xz
2. Run make venv in Python-X.Y.Z/Doc
Sphinx requires more dependencies than are bundled with the basic pip install. Fortunately, the Python project provides a Makefile to create the necessary environment. See the Makefile for precise details.
# Navigate to the Doc/ directory
~/$ cd Python-3.7.9/Doc
# "create a venv with necessary tools"
~/Python-3.7.9/Doc$ make venv
# activate the venv created by make
~/Python-3.7.9/Doc$ source venv/bin/activate
3. Run sphinx-build
Now that the correct environment is set up, we can run Sphinx. This call creates a cache used during generation with the -d option. The documentation files found in the current directory are converted by the texinfo "builder" and output to build/texinfo:
# -b: Use the textinfo builder
# -d: Create "doctree pickles" cache in doctrees/
# Use the current directory as source
# Output to build/texinfo
(venv) ~/Python-3.7.9/Doc$ sphinx-build -b texinfo -d build/doctrees . build/texinfo
4. Use makeinfo to generate the info file
Again, the Python maintainers have given us what we need (even if they haven't documented it well). The previous command created a texi file along with another Makefile. The Makefile calls makeinfo.
# Navigate to the output directory
(venv) ~/Python-3.7.9/Doc$ cd build/texinfo
# Run the generated Makefile
(venv) ~/Python-3.7.9/Doc/build/texinfo$ make
# Hark, unto us an info file is born
(venv) ~/Python-3.7.9/Doc/build/texinfo$ ls
Makefile python-figures python.info python.texi
Like Indiana Jones, you behold the Holy Grail. Many have perished in this journey; you have prevailed. Take a moment to celebrate.
Note: The makeinfo conversion yields errors for me. No matter, I say. The desired info is obtained and I greedily drink from it.
5. Load python.info into Emacs...
Use C-u C-h i to directly open python.info.
To install the info file within the Emacs Help Directory node, first
check C-h v Info-default-directory-list for where info files are stored. Put python.info file there. There may be a file called dir in that directory. The dir file is generated by texinfo and contains the node listing. If no dir file exists, don't worry, that's what we're creating. Note that it's not recommended to edit dir files manually1.
Run update-info-dir in whichever directory you put python.info. This will update (or create) dir with python.info.
For complete details about the texinfo system, see https://www.gnu.org/software/texinfo/manual/texinfo/html_node/Installing-an-Info-File.html.
1Aside from human error, like mistyping a reference, issues may arise due to "malformed" dir files.