I am trying to create autosummary using sphinx-autosummary for my python code which looks like as follows:
main
├───modA
| ├───__init__.py
| ├───modA.py
├───modB
| ├───__init__.py
| ├───modB.py
├───docs
| ├───build
| └───source
| ├───refs
| | |───_autosummary
| | |───index.rst
| | |───modA.rst
| | |───modB.rst
| ├───index.rst
| ├───conf.py
As mentioned in Sphinx documentation, I inserted the abspath of my working directory, added sphinx.ext.autodoc to the list of extensions, and set autosummary_generate to True in conf.py.
import os
import sys
sys.path.insert(0, os.path.abspath('../..'))
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
"sphinx.ext.autosummary",
'sphinx.ext.coverage',
'sphinx.ext.napoleon'
]
autosummary_generate = True
Next, within docs/index.rst, I added a reference to the refs/ folder.
.. toctree::
:maxdepth: 2
refs/index
The refs/index.rst has reference to modA.rst and modB.rst.
.. toctree::
:maxdepth: 2
modA
modB
In modA.rst and modB.rst, I am trying to create autosummaries.
modA.rst
Attributes
~~~~~~~~~~
.. autosummary::
:toctree: _autosummary
modA.modA.create_job
modB.rst
Attributes
~~~~~~~~~~
.. autosummary::
:toctree: _autosummary
modB.modB.get_jobs
While the code is working for modA.rst, it fails for modB.rst. The error says,
failed to import 'modB.modB.get_jobs': no module named modB.modB.get_jobs
I tried putting .. currentmodule::modB and .. currentmodule::main before the autosummary, but with no success. I even tried putting .. module::modB and .. module::main before autosummary, but even that is not working. I searched a lot on the internet, but unable to understand why it's not working.
Edit-1: Added __init__.py in both the folders.
I had similar issue in using Sphinx.
After some experiments, I suspect that the error message like failed to import XXX: no module named XXX does not necessarily mean the module XXX cannot be found. Instead, in my experiment, there is some error in importing XXX.
In my case, the module XXX itself is importing some package YYY that has not been installed. Since Sphinx import XXX in order to extract docstring, it found an error in this import attempt. After I removed that import YYY statement, I can successfully build the documentation.
I had a similar issue.
My reason of failed import error was that the module modA.modA, specified under .. autosummary:: directive, had imports of not installed libraries.
Attributes
~~~~~~~~~~
.. autosummary::
:toctree: _autosummary
modA.modA.create_job
But in general your reason for failed to import error can be different, because the function that imports modules from .. autosummary:: directive catches all errors and autosummary prints to output the same error message.
Here is the related issue on github https://github.com/sphinx-doc/sphinx/issues/7989
I suggest to import your modules modA.modA, modB.modB and see if there are any errors during importing.
It seems that when Sphinx can't handle a python import it will show the generic module import warning (".. no module named ...") as you mentioned. It seems that this warning can even occur when all imported python package are correctly installed in the environment and it just means that something could not be imported.
A workaround to get rid of this warnings is to mock external python libraries by using the autodoc_mock_imports in the conf.py file of Sphinx.
For example:
import flask
import mongoengine
...
# your python code
Example conf.py with mocking flask:
autodoc_mock_imports = ["flask", "mongoengine"]
In this case Sphinx will ignore these imports and autosummary may be able to build the summary correctly without any errors.
To be more specific to your case: It could be that your python file 'modB.modB' contains an import statement that Sphinx can't handle and that you either need to install the python package correctly or to mock the specific libraries imported in the modB.modB file.
Related
I am running Sphinx on a rst file containing automodule but it does not seem to have any effect.
Here are the details: I have a Python project with a file agent.py containing a class Agent in it. I also have a subdirectory apidoc with a file agent.rst in it (generated by sphinx-apidoc):
agent module
============
.. automodule:: agent
:members:
:undoc-members:
:show-inheritance:
I run sphinx with sphinx-build -b html apidoc apidoc/_build with the project's directory as the current working directory.
To make sure the Python files are found, I've included the following in apidoc/conf.py:
import os
import sys
sys.path.insert(0, os.path.abspath('.'))
It runs without errors but when I open the resulting HTML file it only shows "agent module" and everything is blank. Why isn't it showing the class Agent and its members?
Update: the original problem was likely caused by the fact that I had not included sphinx.ext.autodoc in conf.py. Now that I did, though, I get warnings like:
WARNING: invalid signature for automodule ('My Project.agent')
WARNING: don't know which module to import for autodocumenting 'My Project.agent' (try placing a "module" or "currentmodule" directive in the document, or giving an explicit module name)
WARNING: autodoc: failed to import module 'agent'; the following exception was raised:
No module named 'agent'
I'll try answering by putting the "canonical" approach side-by-side with your case.
The usual "getting started approach" follows these steps:
create a doc directory in your project directory (it's from this directory the commands in the following steps are executed).
sphinx-quickstart (choosing separate source from build).
sphinx-apidoc -o ./source ..
make html
This would yield the following structure:
C:\Project
|
| agent.py
|
|---docs
| | make.bat
| | Makefile
| |
| |---build
| |
| |---source
| | conf.py
| | agent.rst
| | index.rst
| | modules.rst
In your conf.py you'd add (after step 2):
sys.path.insert(0, os.path.abspath(os.path.join('..', '..')))
and in index.rst you'd link modules.rst:
Welcome to Project's documentation!
================================
.. toctree::
:maxdepth: 2
:caption: Contents:
modules
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
Now compare the above with what you have - from what you shared in your question:
C:\Project
|
| agent.py
|
|---apidoc
| | agent.rst
| | conf.py
| |
| |-- _build
You ran:
sphinx-build -b html apidoc apidoc/_build
and in your conf.py:
sys.path.insert(0, os.path.abspath('.'))
Your error stacktrace says it couldn't find the module agent. That's probably because you didn't go 1 level down in your conf.py (it's pointing to the path with .rst, not the path with .py), this should work:
sys.path.insert(0, os.path.abspath('..')). Also, if you didn't manually edit/connect your modules.rst in your index.rst you are likely to only see that module.
You may care to notice the signatures of the sphinx commands at play:
sphinx-apidoc [OPTIONS] -o <OUTPUT_PATH> <MODULE_PATH>
sphinx-build [options] <sourcedir> <outputdir> [filenames …]
<sourcedir> refers to where .rst are, and <MODULE_PATH> to where .py are. <OUTPUT_PATH> to where .rst are placed, and <outputdir> to where .html are placed.
Please also notice, you mentioned: "the project's directory as the current working directory." I've seen "working directory" mentioned in sphinx threads on stackoverflow, interchangeably as both the Project base directory, or the docs directory. However, if you search the Sphinx documentation for "working directory" you'll find no mention of it.
Finally, there is an advantage to using the file/directory structure of the "getting started approach". It basically "puts you on the same page" with most threads on the Sphinx tag, and that way alleviates the mental work of mapping the cases to different directory/file structures.
I hope this helps.
I created a python package which I use for brain preprocessing - named bpt. I would also like to use the same package as a sub-module in another project named brain_seg.
As a results, I am having problems with import statements which either results in errors when running the package stand-alone or nested.
The structure of the bpt package is as follows:
bpt
|--__init__.py
|--module1.py
|--module2.py
|--run.py
Import statements in the file run.py look something like:
import os
import module1
import module2
.
.
.
The structure of the brain_seg package is as follows:
brain_seg
|--bpt
| |--__init__.py
| |--module1.py
| |--module2.py
| |--run.py
|
|--package2
| |--__init__.py
| |--module1a.py
|
|--__init__.py
When running the run.py script which is part of the stand alone bpt package, everything runs as expected.
If I try to run the same script as part of the brain_seg package, the following error is dispatched:
ModuleNotFoundError: No module named 'module1'
I tried to use relative imports so the nested project will work as expected, resulting in the following run.py file:
import os
from . import module1
from . import module2
.
.
.
Following the change, execution of the brain_seg project worked as expected but when I tried to call the run.py script from the stand alone bpt package, it resulted in the following error being dispatched:
ImportError: attempted relative import with no known parent package
How should I handle the imports such that both options will be supported?
Imports in bpt.run must be absolute or relative:
from bpt.module1 import ... # absolute
from .module1 import ... # relative
To run bpt.run from your repository root with your pythonpath getting correctly set, use -m:
python bpt/run.py # bad
python -m bpt.run # good
If brain_seg is not truly meant to be a package itself, but just a project consisting of multiple packages and modules, get rid of brain_seg/__init__.py.
If you do mean to use it as a package, then move it below the repo root, i.e.
- README.md (etc.)
- .gitignore (etc.)
- brain_seg
- __init__.py
- bps
- __init__.py
- main.py
and use e.g. python -m brain_seg.bps.run
from the repository root.
If you also intend to use bps as a standalone thing, I wouldn't recommend keeping it in the brain_seg hierarchy at all, but to make it pip installable (see https://packaging.python.org/tutorials/packaging-projects/) and then pip install -e ../somewhere/bps to have pip set up an editable link between your projects.
Forgive me for another "relative imports" post but I've read every stack overflow post about them and I'm still confused.
I have a file structure that looks something like this:
|---MyProject
| |---constants.py
| |---MyScripts
| | |---script.py
Inside of script.py I have from .. import constants.py
And of course when I run python3 script.py, I get the error ImportError: attempted relative import with no known parent package.
From what I understand, this doesn't work because when you run a python file directly you can't use relative imports. After hours of searching it seems that the most popular solution is to add "../" to sys.path and then do import constants.py but this feels a little hacky. I've got to believe that there is a better solution than this right? Importing constants has got to be a fairly common task.
I've also seen some people recommend adding __init__.py to make the project into a package but I don't really understand how this solves anything?
Are there other solutions out there that I've either missed or just don't understand?
Your library code should live in a package. For your folder to be a package, it needs to have __init__.py.
Your scripts can live in the top directory in development. Here is how you would structure this layout:
|---MyProject
| |---myproject
| | |---__init__.py
| | |---constants.py
| |---script.py
If you were to productionize this, you could place scripts into a special folder, but then they would rely on the package being installed on your system. The layout could look like this:
|---MyProject
| |---myproject
| | |---__init__.py
| | |---constants.py
| |---scripts
| | |---script.py
In both cases your imports would be normal absolute (non-relative) imports like
from myproject.constants import ...
OR
from myproject import constants
You are correct that attempting a relative import or a path modification in a standalone script is hacky. If you want to use the more flexible second layout, make a setup.py install script in the root folder, and run python setup.py develop.
|---MyProject
| |---__init__.py # File1
| |---constants.py
| |---MyScripts
| | |---__init__.py # File 2
| | |---script.py
And then the content of File1 __init__.py is:
from . import constants
from . import MyScripts
And then the content of File2 __init__.py is:
from . import script
By converting MyProject to a package, you would be able to import constants like:
from MyProject import constants
# Rest of your code
After that, you need to add the root of your MyProject to your python path. If you are using PyCharm, you do not need to do anything. By default the following line is added to your Python Console or Debugging sessions:
import sys
sys.path.extend(['path/to/MyProject'])
If you are not using PyCharm, one is is to add the above script before codes you run or for debugging, add that as a code to run before every session, or you define a setup.py for your MyProject package, install it in your environment and nothing else need to be changed.
The syntax from .. import x is only intended to be used inside packages (thus your error message). If you don't want to make a package, the other way to accomplish what you want is to manipulate the path.
In absolute terms, you can do:
import sys
sys.path.append(r'c:\here\is\MyProject')
import constants
where "MyProject" is the folder you described.
If you want to use relative paths, you need to do a little more work:
import inspect
import os
import sys
parent_dir = os.path.split(
os.path.dirname(inspect.getfile(inspect.currentframe())))[0]
sys.path.append(parent_dir)
import constants
To put it concisely, the error message is telling you exactly what's happening; t's just not explaining why. .. isn't a package. It resolves to MyProject but MyProject isn't a package. To make it a package, you must include an “__init__.py” file directly in the “MyProject” directory. It can even be an empty file if you're feeling lazy.
But given what you've shown us, it doesn't seem like your project is supposed to be a package. Try changing the import in script.py to import constants and running python3 MyScripts/script.py from your project directory.
I'm trying to document a project using Sphinx, and am running into an issue where only some modules are being imported from a folder. My project structure looks like this:
Project
|
|--Main
| |--Scripts
| __init__.py
| libsmop.py
| conv_table.py
| f_discrim.py
| recipes.py
| ...
When I try to run make html, libsmop and recipes are imported without any issue, however conv_table and f_discrim get the following error:
WARNING: autodoc: failed to import module u'conv_table' from module u'Scripts'; the following exception was raised:No module named conv_table
I don't think it's my config file because it's finding all of the files when I run sphinx-apidoc -o _rst Main/Scripts and I've confirmed that they appear in the resulting Scripts.rst file.
Why is autodoc finding some modules but not others?
Edit:
conv_table.py is of this form:
import re
import numpy as np
"""
conv_table dictionary at the bottom of this file maps from matlab functions
to their python equivalents.
"""
def get_args(line,separator=",", open_char='(', close_char=')'):
"""Returns the arguments of line
>>> get_args('ones(3,1,length(arr))')
...
< a bunch of function definitions>
...
conv_table = {... < a very big dictionary > ...}
Since your autodoc is picking up some of the modules, it may be because the dependencies of the failed modules are either 1) not imported correctly or 2) not installed under your python environment. You will want to check if all the import statements work within your failed modules.
You will want to check the module loading path, according to the Sphinx docs:
For Sphinx (actually, the Python interpreter that executes Sphinx) to find your module, it must be importable. That means that the module or the package must be in one of the directories on sys.path – adapt your sys.path in the configuration file accordingly.
Also it would be useful to know how your __init__.py in Scripts directory looks like and how the conv_table module looks like as well.
I had a similar issue like yours, the fix was to append the path that holds that module inside the ../source/conf.py file using
sys.path.insert(0, os.path.abspath('whatever relative path works for your folder structure'))
sys.path.append('/path/to/the/conv_table/')
installing this library in your environment should resolve the problem as of now:
pip install sphinxcontrib-bibtex
after running the make html command it may warn you about the configuration problems.
I used the sphinx-quickstart to set everything up. I used doc/ for the documentation root location. The folder containing my package is setup as:
myfolder/
doc/
mypackage/
__init__.py
moprob.py
...
After the quick start, I edited the path in conf.py to be:
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
Then I added one of the scripts in my package to index.rst to see how Sphinx works.
.. toctree::
:maxdepth: 2
:caption: Contents:
mypackage/moprob
The error code I get:
.../index.rst:9: WARNING: toctree contains reference to nonexisting document u'mypackage/moprob'
Solutions I have tried:
Adding sphinx.ext.napoleon to the extensions list since all of my doc strings are written using the NumPy format. The error did not go away. I also put the napoleon extension after autodoc because one of the documentation pages suggested that.
extensions = ['sphinx.ext.autodoc',
'sphinx.ext.napoleon']
Adding numpydoc_show_class_members = False to conf.py. I put this directly below the extensions but it also did not solve the error.
A couple of different configurations for folder locations. I've also tried setting the root location to be /myfolder and setting the source to be /mypackage and the build to be /doc. None has worked.
The toctree directive contains references to reStructuredText documents, not Python scripts or modules. Sphinx expects there to be a mypackage/moprob.rst file (in the doc folder), but there isn't one. Hence the error.
To quickly get some meaningful output, create the mypackage/moprob.rst file. Add a heading and an automodule directive in it:
moprob module
=============
.. automodule:: mypackage.moprob
:members:
Then run sphinx-build again.