I'm trying to auto-generate basic documentation for my codebase using Sphinx. However, I'm having difficulty instructing Sphinx to recursively scan my files.
I have a Python codebase with a folder structure like:
<workspace>
└── src
└── mypackage
├── __init__.py
│
├── subpackageA
│ ├── __init__.py
│ ├── submoduleA1
│ └── submoduleA2
│
└── subpackageB
├── __init__.py
├── submoduleB1
└── submoduleB2
I ran sphinx-quickstart in <workspace>, so now my structure looks like:
<workspace>
├── src
│ └── mypackage
│ ├── __init__.py
│ │
│ ├── subpackageA
│ │ ├── __init__.py
│ │ ├── submoduleA1
│ │ └── submoduleA2
│ │
│ └── subpackageB
│ ├── __init__.py
│ ├── submoduleB1
│ └── ubmoduleB2
│
├── index.rst
├── _build
├── _static
└── _templates
I've read the quickstart tutorial, and although I'm still trying to understand the docs, the way it's worded makes me concerned that Sphinx assumes I'm going to manually create documentation files for every single module/class/function in my codebase.
However, I did notice the "automodule" statement, and I enabled autodoc during quickstart, so I'm hoping most of the documentation can be automatically generated. I modified my conf.py to add my src folder to sys.path and then modified my index.rst to use automodule. So now my index.rst looks like:
Contents:
.. toctree::
:maxdepth: 2
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
.. automodule:: alphabuyer
:members:
I have dozens of classes and functions defined in the subpackages. Yet, when I run:
sphinx-build -b html . ./_build
it reports:
updating environment: 1 added, 0 changed, 0 removed
And this appears to have failed to import anything inside my package. Viewing the generated index.html shows nothing next to "Contents:". The Index page only shows "mypackage (module)", but clicking it shows it also has no contents.
How do you direct Sphinx to recursively parse a package and automatically generate documentation for every class/method/function it encounters, without having to manually list every class yourself?
You can try using sphinx-apidoc.
$ sphinx-apidoc --help
Usage: sphinx-apidoc [options] -o <output_path> <module_path> [exclude_paths, ...]
Look recursively in <module_path> for Python modules and packages and create
one reST file with automodule directives per package in the <output_path>.
You can mix sphinx-apidoc with sphinx-quickstart in order to create the whole doc project like this:
$ sphinx-apidoc -F -o docs project
This call will generate a full project with sphinx-quickstart and Look recursively in <module_path> (project) for Python modules.
Perhaps apigen.py can help: https://github.com/nipy/nipy/tree/master/tools.
This tool is described very briefly here: http://comments.gmane.org/gmane.comp.python.sphinx.devel/2912.
Or better yet, use pdoc.
Update: the sphinx-apidoc utility was added in Sphinx version 1.1.
Note
For Sphinx (actually, the Python interpreter that executes
Sphinx) to find your module, it must be importable. That means that
the module or the package must be in one of the directories on
sys.path – adapt your sys.path in the configuration file accordingly
So, go to your conf.py and add
import an_example_pypi_project.useful_1
import an_example_pypi_project.useful_2
Now your index.rst looks like:
.. toctree::
:glob:
example
an_example_pypi_project/*
and
make html
From Sphinx version 3.1 (June 2020), if you're happy to use sphinx.ext.autosummary to display summary tables, you can use the new :recursive: option to automatically detect every module in your package, however deeply nested, and automatically generate documentation for every attribute, class, function and exception in that module.
See my answer here: https://stackoverflow.com/a/62613202/12014259
Related
i'm experimenting with DDD in Python so i've decided to implement a toy project.
I've created different directories in other to separate shared concepts from specific bounded contexts concepts.
As i try to import these files, i'm facing a No module named error exceptions
For example, with this project structure:
.
└── src/
├── Book/
│ ├── application
│ ├── domain/
│ │ ├── Book.py
│ │ └── __init__.py
│ ├── infrastructure
│ └── __init__.py
└── Shared/
├── application
├── domain/
│ ├── Properties/
│ │ ├── __init__.py
│ │ └── UuidProperty.py
│ ├── ValueObjects/
│ │ ├── __init__.py
│ │ └── BookId.py
│ └── __init__.py
└── infrastructure
On src/Book/domain/Book.py i have:
from Shared.domain.ValueObjects.BookId import BookId
class Book:
bookId: BookId
pages: int
As i've seen in other answer (pretty old ones) it can be fixed by adding these folders to PYTHONPATH or PATH like sys.path.insert(*path to file*) but i'm wondering if there is a more pythonic way to achieve that.
I've also tried to add an __init__.py file to src and import as from src.Shared.domain.ValueObjects.BookId import BookId but none of previous attempts worked for me
On other repos i've saw that they use setuptools to install the src package in order to import it at unit tests (i cant either import them at tests), but i don't know if that is recommended or would work inside package imports
In case someone is facing the same issue as me, i managed to import subpackages and the full package in tests directory.
Just include, in each subpackage, an __init__.py file, and inside the package/subpackages, use relative imports (it looses semantic of imports, when we know where each imports comes from by absolute path from root directory, but works)
from ..Properties import UuidProperty
# inside __init__.py of Properties directory
from .UuidProperty import UuidProperty
And, by including an __init__.py inside src/ we could import them at tests directory like
from src.Book.domain import Book
Hope this helps someone!
I am currently developing a package simultaneously with a few projects that use the package, and I'm struggling to figure out how to structure my directory tree and imports.
Ideally, I want something like this:
main_directory
├── shared_package
│ ├── __init__.py
│ ├── package_file1.py
│ └── package_file2.py
├── project1
│ ├── main.py
│ ├── module1.py
│ └── other_package
│ ├── __init__.py
│ └── other_package_file.py
└── project2
└── ...
I can't figure out how to make the imports work cleanly for importing shared_package from python files in project1. Is there a preferred way to do this?
Any help would be appreciated!
shared_package will eventually be standalone. Other people will import and install it the normal way, and it'll be stored with the rest of the python modules in site-packages or wherever.
To replicate this, I recommend just updating your PYTHONPATH to point to main_directory (or wherever you put shared_package anyway) - that way,
import shared_package
will still work fine for the code if shared_package was installed normally, because it's on the pythonpath either way.
Note that PYTHONPATH is an environment variable, so the means for doing this will vary based on your operating system. Regardless, a quick search for how to modify the variable permanently on your OS should be easy.
I have a question regarding the Sphinx autodoc generation. I feel that what I am trying to do should be very simple, but for some reason, it won't work.
I have a Python project of which the directory is named slotting_tool. This directory is located at C:\Users\Sam\Desktop\picnic-data-shared-tools\standalone\slotting_tool
I set up Sphinx using sphinx-quickstart. Then my directory structure (simplified) is as follows:
slotting_tool/
|_ build/
|_ source/
|___ conf.py
|___ index.rst
|_ main/
|___ run_me.py
Now, I set the root directory of my project to slotting_tool by adding the following to the conf.py file.
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
Next, I update my index.rst file to look like this:
.. toctree::
:maxdepth: 2
:caption: Contents:
.. automodule:: main.run_me
:members:
When trying to build my html using the sphinx-build -b html source .\build command, I get the following output, with the no module named error:
(base) C:\Users\Sam\Desktop\picnic-data-shared-tools\standalone\slotting_tool>sphinx-build -b html source .\build
Running Sphinx v1.8.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [] 0 added, 1 changed, 0 removed
reading sources... [100%] index
WARNING: autodoc: failed to import module 'run_me' from module 'main'; the following exception was raised:
No module named 'standalone'
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
generating indices... genindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 1 warning.
The HTML pages are in build.
There are no HTML pages that refer to run_me.py in build. I have tried setting my root directory to all different kinds of directories and I have tried replacing all dots . with backslashes \ and so forth, but can't seem to find out what I'm doing wrong.
By the way, the statement that standalone is not a module is in fact true, it is just a directory without an __init__.py. Don't know if that might have caused some trouble?
Anyone have an idea?
This is the usual "canonical approach" to "getting started" applied to the case when your source code resides in a src directory like Project/src instead of simply being inside the Project base directory.
Follows these steps:
Create a docs directory in your Project directory (it's from this docs directory the commands in the following steps are executed).
sphinx-quickstart (choose separate source from build. Places .html and .rst files in different folders).
sphinx-apidoc -o ./source ../src
make html
This would yield the following structure (provided you .py source files reside in Project/src):
Project
|
├───docs
│ │ make.bat
│ │ Makefile
│ │
│ ├───build
│ └───source
│ │ conf.py
│ │ index.rst
│ │ modules.rst
│ │ stack.rst
│ │
│ ├───_static
│ └───_templates
└───src
stack.py
In your conf.py you'd add (after step 2):
import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join('..', '..', 'src')))
Also include in conf.py:
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.napoleon']
And in index.rst you'd link modules.rst:
Welcome to Project's documentation!
================================
.. toctree::
:maxdepth: 2
:caption: Contents:
modules
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
Your stack.rst and modules.rst were auto-generated by sphinx-apidoc, no need to change them (at this point). But just so you know this is what they look like:
stack.rst:
stack module
============
.. automodule:: stack
:members:
:undoc-members:
:show-inheritance:
modules.rst:
src
===
.. toctree::
:maxdepth: 4
stack
After `make html` open `Project/docs/build/index.html` in your browser, the results:
and:
Let's take an example with a project: dl4sci-school-2020 on master branch, commit: 6cbcc2c72d5dc74d2defa56bf63706fd628d9892:
├── dl4sci-school-2020
│ ├── LICENSE
│ ├── README.md
│ ├── src
│ │ └── __init__.py
│ └── utility
│ ├── __init__.py
│ └── utils.py
and utility package has a utils.py module:
Follow this process(FYI, I'm using sphinx-build 3.1.2):
create a docs/ directory under you project:
mkdir docs
cd docs
start sphinx within docs/, and just pass your project_name, your_name & version of your choice and rest keep defaults.
sphinx-quickstart
you will get below auto-generated in your docs/ folder
├── docs
│ ├── Makefile
│ ├── build
│ ├── make.bat
│ └── source
│ ├── _static
│ ├── _templates
│ ├── conf.py
│ └── index.rst
Since, we created a separate docs directory so we need sphinx find
where to find build files and python src module.
So, edit the conf.py file, you can use my conf.py file too
import os
import sys
basedir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
sys.path.insert(0, basedir)
Now, to enable access to nested multiple packages & modules if any, you need to edit index.rst file.
.. toctree::
:maxdepth: 2
:caption: Description of my CodeBase:
modules
The modules picks up content from modules.rst file which we will create below:
Make sure you're still in doc/ to run the below command
sphinx-apidoc -o ./source ..
The output you get:
├── docs
│ ├── Makefile
│ ├── build
│ ├── make.bat
│ └── source
│ ├── _static
│ ├── _templates
│ ├── conf.py
│ ├── index.rst
│ ├── modules.rst
│ ├── src.rst
│ └── utility.rst
now run:
make html
Now, go and open in browser of your choice,
file:///<absolute_path_to_your_project>/dl4sci-school-2020/docs/build/html/index.html
have you beautiful documentation ready
https://imgur.com/5t1uguh
FYI, You can switch any theme of your choice, I found sphinx_rtd_theme and extension sphinxcontrib.napoleon super dope!. Thanks to their creators, so I used it.
below does the work!
pip install sphinxcontrib-napoleon
pip install sphinx-rtd-theme
You can host your documentation it on readthedocs
enjoy documenting your code!
sys.path.insert(0, os.path.abspath('../..'))
That's not correct. Steve Piercy's comment is not entirely on point (you don't need to add a __init__.py since you're using a simple module) but they're right that autodoc will try to import the module and then inspect the content.
Hoever assuming your tree is
doc/conf.py
src/stack.py
then you're just adding the folder which contains your repository to the sys.path which is completely useless. What you need to do is add the src folder to sys.path, such that when sphinx tries to import stack it finds your module. So your line should be:
sys.path.insert(0, os.path.abspath('../src')
(the path should be relative to conf.py).
Of note: since you have something which is completely synthetic and should contain no secrets, an accessible repository or a zip file of the entire thing makes it much easier to diagnose issues and provide relevant help: the less has to be inferred, the less can be wrong in the answer.
IMHO running pip install --no-deps -e . in the top project folder (or where ever setup.py is) to get an "editable" install is a better alternative to get your package modules on the PYTHONPATH than altering it in docs/conf.py using sys.path.
For me installing the package via setup.py file and re-running corresponding commands fixed the problem:
$ python setup.py install
I have a project that defines testing support modules, including py.test plugins in a package subdirectory like this:
bokeh/_testing
├── __init__.py
├── plugins
│ ├── __init__.py
│ ├── bokeh_server.py
│ ├── examples_report.jinja
│ ├── examples_report.py
│ ├── file_server.py
│ ├── implicit_mark.py
│ ├── integration_tests.py
│ ├── jupyter_notebook.py
│ ├── log_file.py
│ └── pandas.py
└── util
├── __init__.py
├── api.py
├── examples.py
├── filesystem.py
├── git.py
└── travis.py
Several of the plugins need to define new options with parser.addoption. I would like these calls to be made inside the respective plugin modules. However, if I do that, and include those plugins in a test file with, e.g.
# test_examples.py
pytest_plugins = (
"bokeh.testing.plugins.bokeh_server",
"bokeh.testing.plugins.examples_report",
)
# pytest.mark.examples test code here
Then pytest complains that any custom command line options are undefined:
(base) ❯ py.test -s -v -m examples --diff-ref FETCH_HEAD --report-path=examples.html
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: --diff-ref --report-path=examples.html
inifile: /Users/bryanv/work/bokeh/setup.cfg
rootdir: /Users/bryanv/work/bokeh
The only way I have found around this, is to collect ALL custom options in a single pytest_addoption in the top-level conftest.py:
# conftest.py
pytest_plugins = (
"bokeh.testing.plugins.implicit_mark",
"bokeh.testing.plugins.pandas",
)
# Unfortunately these seem to all need to be centrally defined at the top level
def pytest_addoption(parser):
parser.addoption(
"--upload", dest="upload", action="store_true", default=False, help="..."
)
# ... ALL other addoptions calls for all plugins here ...
As I said, this works, but is very inflexible in terms of code organization. It would be preferable to have a way for the options for the examples.py plugin to in the examples.py module, with the code it relates to.
Another possibility would presumably be to bring in all the plugins in the top-level conftest.py, but some of the plugins are very heavyweight, e.g. depend on Selenium, and I would not like to require that all that be installed just to run basic tests.
Is there another way to accomplish this, and if so, how?
As noted above, as of Pytest 3.6.2, options must be added "only in plugins or conftest.py files situated at the tests root directory due to how pytest discovers plugins during startup."
I am trying to document a package in Python. At the moment I have the following directory structure:
.
└── project
├── _build
│ ├── doctrees
│ └── html
│ ├── _sources
│ └── _static
├── conf.py
├── index.rst
├── __init__.py
├── make.bat
├── Makefile
├── mod1
│ ├── foo.py
│ └── __init__.py
├── mod2
│ ├── bar.py
│ └── __init__.py
├── _static
└── _templates
This tree is the result of the firing of sphinx-quickstart. In conf.py I uncommented sys.path.insert(0, os.path.abspath('.')) and I have extensions = ['sphinx.ext.autodoc'].
My index.rst is:
.. FooBar documentation master file, created by
sphinx-quickstart on Thu Aug 28 14:22:57 2014.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to FooBar's documentation!
==================================
Contents:
.. toctree::
:maxdepth: 2
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
In all the __init__.py's I have a docstring and same goes to the modules foo.py and bar.py. However, when running make html in the project I don't see any of the docstings.
Here is an outline:
Document your package using docstrings in the sources.
Use sphinx-quickstart to create a Sphinx project.
Run sphinx-apidoc to generate .rst sources set up for use with autodoc. More information here.
Using this command with the -F flag also creates a complete Sphinx project. If your API changes a lot, you may need to re-run this command several times.
Build the documentation using sphinx-build.
Notes:
Sphinx requires .rst files with directives like automodule or autoclass in order to generate API documentation. It does not automatically extract anything from the Python sources without these files. This is different from how tools like Epydoc or Doxygen work. The differences are elaborated a bit more here: What is the relationship between docutils and Sphinx?.
After you have run sphinx-apidoc, it may be necessary to adjust sys.path in conf.py for autodoc to find your modules.
In order to avoid strange errors like in these questions, How should I solve the conflict of OptionParser and sphinx-build in a large project?, Is OptionParser in conflict with sphinx?, make sure that the code is properly structured, using if __name__ == "__main__": guards when needed.