I'm running into a bit of brick wall trying to work out the best way to implement automatic documentation generation for a Django project
my project structure is similar to the following:
myproj/ # project root
|- docs/ # docs root
|- myproj/ # package root
|- app1/
|- app2/
|- app3/
utils.py
...
I would like to automatically generate documentation for everything under the package root directory- and initially started using sphinx-apidoc, but then read something that implied I should be using autosummary instead and have not been able to find any information outlining what the difference between these two options is.
To anyone else struggling to work it out, this is the guide I finally found that helped me clear up the confusion:
https://romanvm.pythonanywhere.com/post/autodocumenting-your-python-code-sphinx-part-ii-6/
In short, use autosummary
Related
What configuration options should I use in setup.py:setup that would allow me to have the following package layout?
tests/
src/__init__.py
src/subpackage/__init__.py
I'm slightly embarrassed by not being able to figure this out.
I tried using
packages=find_namespace_packages("src")
package_dir={"": "src"}
and
packages=find_namespace_packages("src")
package_dir={"myrootpackage": "src"}
but neither seems to work.
I have a project including a Python package as well as some Matlab code, which should all go into the same documentation created by Sphinx.
The Python part of the documentation works flawlessly quite some time already - the Matlab part was added now and makes some trouble.
The data structure more or less is as follows:
|- python
|- modules
|- including subpackages
|- some sub-modules
|- matlab
|- scripts
|- functions
|- docs
|- source
|- conf.py
|- matlab.rst
|- python.rst
|- index.rst
|- ...
The most relevant lines in conf.py are prob. the following:
import os
import sys
# for the Python autodoc package
sys.path.insert(0, os.path.abspath('../..'))
extensions = ['sphinx.ext.napoleon', 'sphinxcontrib.matlab', 'sphinx.ext.autodoc']
# Path to the Matlab files
matlab_src_dir = '../../matlab'
#matlab_src_dir = '<<abssolute_path>>/matlab'
As you see, I tried both relative and absolute paths for matlab_src_dir. So far, without any difference.
The python.rst, which was created automatically by apidoc, contains:
python package
==============
.. automodule:: python
:members:
:undoc-members:
:show-inheritance:
Subpackages
-----------
.. toctree::
:maxdepth: 4
python.subpackages
The matlab.rst was created manually and contains:
matlab package
==============
.. mat:automodule:: matlab
:members:
:undoc-members:
:show-inheritance:
Subscripts
----------
.. toctree::
:maxdepth: 4
matlab.function_file
Thus, python.rst and matlab.rst are more or less the same but with mat:automodule for Matlab according to the documentation.
Finally, index.rst contains:
.. toctree::
:maxdepth: 2
python
.. toctree::
:maxdepth: 2
matlab
Now, when I run Sphinx' make html I receive the following error, which results in a nice Python documentation but an empty Matlab documentation:
WARNING: [sphinxcontrib-matlabdomain]: failed to import module 'matlab'; the following exception was raised:
Traceback (most recent call last):
File "C:\Users\<<username>>\AppData\Roaming\Python\Python39\site-packages\sphinxcontrib\mat_documenters.py", line 117, in import_object
obj = self.module = modules[self.modname]
KeyError: 'matlab'
What am I missing here. Why can't Sphinx find the matlab folder with the containing m-files?
And if that should be solved, maybe a secondary question: is there any similar function as apidoc for Matlab files, so that I do not need to create all *.rst files myself?
Thank you for any hints!
BTW: I am running on Sphinx v4.4.0 under Python 3.9.7.
Neither providing a hard-coded absolute path to matlab_src_dir nor providing os.path.abspath(relative_path) lead to success.
Thanks to Steve Piercy, who mentioned an example in the comments, I found that I should use os.path.dirname(os.path.abspath(relative_path)). As such, the errors are gone and Sphinx is working.
This is interesting, because I have already tried os.path.abspath(relative_path to the parent of the package) before, which I would expect is doing the same than os.path.dirname(). Anyway, I got a working solution now.
If anybody would have an idea about the second question ("is there any similar function as apidoc for Matlab files, so that I do not need to create all *.rst files myself?") I would be really happy.
I just started testing out PyCharm on my existing Django project, and it doesn't recognize any imports from apps within my project:
in my_app1/models.py:
from my_app2.models import thing
"Unresolved reference 'my_app2'"
Why is this? My project's directory structure matches the recommended layout, and it runs without errors, it's just PyCharm's magic doesn't want to work on it.
It seems related to this question:
Import app in django project
But I can't figure out what I am doing wrong. If I try:
from ..my_app2.models import thing
The PyCharm error goes away and it can auto predict, etc. But when I run the project Django throws:
ValueError: attempted relative import beyond top-level package
EDIT:
Project structure:
my_project/
src/
manage.py
db.sqlite3
my_app1/
templates/
__init.py__
admin.py
models.py
urls.py
views.py
...
my_app2/
templates/
__init.py__
admin.py
models.py
urls.py
views.py
...
my_project_app/
settings/
__init.py__
urls.py
...
I was having this issue using a "2 Scoops of Django" project layout, e.g.
/project_root
/project_dir
/config
/settings
/my_app
/tests
models.py
/requirements
readme.rst
The code was working, but in /tests, IntelliJ/PyCharm showed an unresolved reference:
from my_app.models import Something
I had all the __init__.py files in place. I ended up having to set the sources root to project_dir:
Right-click on project_dir, Mark Directory as > Sources Root
Now that I can take a look over you project structure I can tell you that the problem appears to be related to a missing __init__.py in your 'src' folder. Try adding an empty file named __init__.py in the root of 'src' folder.
Also, take a look to this question, I think is the same problem or a very similar one.
Hope this could be useful, cheers!
I was having this issue after I change my environment to virtualenv, so I changed my python interpreter to my current virtualenv.
Go to File > Settings > Project Interpreter.
In that window you would be able to see all packages includes on this interpreter, Django should be there.
This worked for me.
Link about: https://intellij-support.jetbrains.com/hc/en-us/community/posts/206598665-Unresolved-Reference-Errors-for-django
In our company we're using subversion. We have different python modules (own and third party) in different versions in use. The various applications we develop have various dependencies regarding the version of the shared modules.
One possibility is using virtualenv installing the modules from a local pypi server. So on every initial checkout we need to create a virtualenv, activate it and install dependent modules from requirements.txt.
Disadvantages:
Relatively complex operation for a simple task like checkout and run
Your able to miss the creation of the virtualenv and your working with the modules installed in site-packages
Need for a local pypi server (ok, your otherwise able to use urls pointing to your vcs)
So we came up with another solution and I ask for your opinion:
In the path of the application we use svn:externals (aka git submodules) to "link" to the specified module (from it's release path and with specified revision number to keep it read only), so the module will be placed locally in the path of the application. A "import mylib" will work as it was installed in python site-packages or in the virtualenv. This could be extended to even put a release of wx, numpy and other often used libraries into our repository and link them locally.
The advantages are:
After the initial checkout your ready to run (really important point for me)
version dependencies are fixed (like requirements.txt)
The actual question is:
Are there projects out there on github/sorceforge using this scheme? Why is everybody using virtualenv instead of this (seemingly) simpler scheme?
I never saw such a solution, so maybe we miss a point?
PS: I posted this already on pypa-dev mailinglist but it seems to be the wrong place for this kind of question. Please excuse this cross post.
In the path of the application we use svn:externals (aka git submodules) to "link" to the specified module (from it's release path and with specified revision number to keep it read only), so the module will be placed locally in the path of the application.
This is a more traditional method for managing package dependencies, and is the simpler of the two options for software which is only used internally. With regards to...
After the initial checkout you're ready to run
...that's not strictly true. If one of your dependencies is a Python library written in C, it will need to be compiled first.
We tried it with git's submodule functionality but it's not possible to get a subpath of a repository (like /source/lib)
This is fairly easy to work around if you check out the whole repository in a location outside your PYTHONPATH, then just symlink to the required files or directories inside your PYTHONPATH, although it does require you to be using a filesystem which support symlinks.
For example, with a layout like...
myproject
|- bin
| |- myprogram.py
|
|- lib
| |- mymodule.py
| |- mypackage
| | |- __init__.py
| |
| |- foopackage -> ../submodules/libfoo/lib/foopackage
| |- barmodule
| |- __init__.py -> ../../submodules/libbar/lib/barmodule.py
|
|- submodules
|- libfoo
| |- bin
| |- lib
| |- foopackage
| |- __init__.py
|
|- libbar
|- bin
|- lib
| barmodule.py
...you need only have my_project/lib in your PYTHONPATH, and everything should import correctly.
Are there projects out there on github/sourceforge using this scheme?
The submodule information is just stored in a file called .gitmodules, and a quick Google for "site:github.com .gitmodules" returns quite a few results.
Why is everybody using virtualenv instead of this (seemingly) simpler scheme?
For packages published on PyPI, and installed with pip, it's arguably easier from a dependency-management point-of-view.
If your software has a relatively simple dependency graph, like...
myproject
|- libfoo
|- libbar
...it's no big deal, but when it becomes more like...
myproject
|- libfoo
| |- libsubfoo
| |- libsubsubfoo
| |- libsubsubsubfoo
| |- libsubsubsubsubfoo
|- libbar
|- libsubbar1
|- libsubbar2
|- libsubbar3
|- libsubbar4
...you may not want to take on the responsibility of working out which versions of all those sub-packages are compatible, should you need to upgrade libbar for whatever reason. You can delegate that responsibility to the maintainer of the libbar package.
In your particular case, the decision as to whether your solution is the right one will depend on the answers to the questions:-
Are all of the external modules you need to use actually available from svn repositories?
Do those repositories use svn:externals correctly to include compatible versions of any dependencies they require, or if not, are you prepared to take on the responsibility of managing those dependencies yourself?
If the answer to both questions is "yes", then your solution is probably right for your case.
I have been trying to create a Google App Engine project that contains various modules, each with various versions; yet I cannot get it to work.
I went through everything on the google modules guide website but their description of the hierarchy of the application is very vague. I downloaded a simple test application to see how it works (which I cannot get to work).
This is the structure of the application right now:
http://www.iteratorium.eu/stackoverflow/structure.jpg
and the whole thing is zipped up here:
http://www.iteratorium.eu/stackoverflow/flask_app.zip
There are three modules: default, my-module and mobile-frontend, and both the my-module and mobile-frontend have two versions in directories v_one and v_two. Each version has its own .yaml file, which contains a single handler (keeping it simple for testing purposes)
Is the structure at least correct? When I load the application through the dev_appserver.py file, everything is fine as long as I do not load two versions of the same module. In that case, this happens:
me#MY_COMPUTER:~/flask_app$ python ~/google_appengine/dev_appserver.py dispatch.yaml app.yaml mobile-frontend/v_one/mobile-frontend.yaml my-module/v_one/my-module.yaml my-module/v_two/my-module.yaml
Results in a traceback and the following:
google.appengine.tools.devappserver2.errors.InvalidAppConfigError: Duplicate module: my-module
(Both versions get uploaded onto appengine via appcfg.py without any errors, it might be only the localhost server cannot handle many versions)
If I only load one version of each module, everything works, but the modules cannot import anything from the lib directory.
Accessing http://localhost:8082/mobiler through the browser results in this in the terminal:
from flask import Flask
ImportError: No module named flask
I defined the path to the lib folder in the appengine_config.py file but it does not seem to work for the modules. That is where I stopped and decided to come here, since I am not even certain the structure is correct and I might be way off with all of this.
So... How far off am I?
note: The dev server won't let you load different versions of the same module
You need the module definitions to be in the toplevel dir for appengine_config.py to get loaded.
Considering you really really want to keep the code for the two versions separated, a better app organization would be as follows (just showing mobile-frontend to keep it short):
root
|__ mobile-frontend
| |__ v_one
| |__ __init__.py
| |__ mobiler.py
| |__ v_two
| |__ __init__.py
| |__ mobiler.py
|__ appengine_config.py
|__ dispatch.yaml
|__ mobile_frontend_v_one.yaml
|__ mobile_frontend_v_two.yaml
And having mobile_frontend_v_one.yaml being having something like:
handlers:
- url: .*/mobiler
script: mobile-frontend.v_one.mobiler.app
Perhaps you're doing a little too much in the appengine_config.py, try reducing it to:
import sys
sys.path.insert(0, 'lib')