Since Python 3.3, the compulsion for having an __init__.py in every directory of a source code was removed. My aim is to read all the modules and submodules of a given source code using pyclbr. However, there are certain cases where a particular module is a submodule to other modules and not all the parent modules are packages. pyclbr does not understand the difference between directories and packages.
So, while attempting to solve this problem, I thought of adding the __init__.py file in every directory that is not a package. I am not sure as to what the downsides are. I have this inhibition of deviating from the standards set by the Python community and that might result in incorrectness sometime later.
As stated in the above hyperlink, I could go with specifying only the module and give its full-path to pyclbr but that does not work in cases like the following source code structure:
(gruml) ➜ gruml git:(run-sequence-diagram) ✗ tree a
a
└── a
└── a
├── __init__.py
└── a.py
2 directories, 2 files
This does not work because sending a to pyclbr would return the same data for a.a.a and for a.a.a.a. So, I have to send the full-module path and work for directories.
would return the same data for a.a.a and for a.a.a.a
Since I'm not familiar with pyclbr, it's unclear what data you're referring to, but if you want something specific to only a.a.a, then that would be need to be written inside of the init file
I'm trying to find out how the name/directory build/lib.system is chosen/created within a distutils(setuptools) python build process...
when i build dlib python module from source using python3 setup.py build i can see the following
build/
├── bdist.linux-aarch64
├── CMakeCache.txt
├── CMakeFiles
├── lib.linux-aarch64-3.8
└── temp.linux-aarch64-3.8```
reading at the official documentation in 4.1 says:
Depending on the system, the module file will end up in a subdirectory build/lib.system...
It's not clear to me where does lib.${system} comes from and if i can configure it using a class extension argument or it is a -DCMAKE_VARIABLE?
I'm using setuptools instead of distutils but for now it looks like the documentation in this subject is the same.
Please any help will be appreciated also, if anybody knows which is the path of the source code for class Extension will be very helpfull.
thanks!!
I'm writing a couple of packages that I'd like to release on PyPi for other people to use.
I've not released to PyPi before so I have been mocking up a submission template: https://github.com/chris-brown-nz/pypi-package-template
Here's a tree of the project template:
| MANIFEST.in
| README.rst
| setup.cfg
| setup.py
|
\---package
module_one.py
module_three.py
module_two.py
__init__.py
In terms of interacting with the package, this is what I would usually do - is it the best way?
To run a method:
from package import module_one
module_one.ClassOne().method_a()
To get a value from a method:
from package import module_two
print(module_two.ClassFive().method_e())
To set then use an attribute of an instance:
from package import module_three
cls = module_three.ClassSeven("Hello World")
print(cls.value)
'package' is a reserved name obviously and won't be used in the final project.
I'd be grateful for some feedback on how I've structured my project and whether it is considered standard, or if it should be modified in some way.
There are different approaches to this, whether one or the other is better is depending on a how you want to develop, usage of the package (e.g. if you ever install it using pip install -e packag_name), etc.
What is missing from your tree is the name of the directory where the setup.py resides, and that is usually the package name:
└── package
├── package
│ ├── __init__.py
│ ├── module_one.py
│ ├── module_three.py
│ └── module_two.py
├── MANIFEST.in
├── README.rst
├── setup.cfg
└── setup.py
as you can see you are doubling the 'package' name, and that means that your setup.py has to be adapted for each package, or dynamically determine the name of the directory where the module.py files resides. If you go for this route, I would suggest you put the module.py files in a generically named directory 'src' or 'lib'.
I don't like the above "standard" setup for multiple reasons:
it doesn't map well to how python programs "grow" before they are split up into packages. Before splitting up having such a 'src' directory would mean using:
from package.src.module_one import MyModuleOneClass
Instead you would have your module.py files directly under package
Having a setup.py to control installation, a README.rst for documentation and an __init__.py to satisfy Python's import is one thing, but all other stuff, apart from your module.py files containing the actual functionality, is garbage. Garbage that might be needed at some point during the package creation process, but is not necessary for the package functionality.
There are other considerations, such as being able to access the version number of the package from the setup.py as well as from the program, without the former having to import the package itself (which may lead to install complications), nor having another extra version.py file that needs importing.
In particular I always found the transition from using a directory structure under site-packages that looked like:
└── organisation
├── package1
└── package2
├── subpack1
└── subpack2
and that could intuitively be used for both importing and navigation to source files, to something like:
├── organisation_package1
│ └── src
├── organisation_package2_subpack1
│ └── src
└── organisation_package2_subpack2
└── src
unnatural. To rearrange and break a working structure to be able to package things seems wrong.
For my set of published packages I followed another way:
- I kept the natural tree structure that you can use "before packaging", 'src' or 'lib' directories.
- I have a generic setup.py which reads and parses (it does not import) the metadata (such as version number, package name, license information, whether to install a utility (and its name)), from a dictionary in the __init__.py file. A file you need anyway.
- The setup.py is smart enough to distinguish subdirectories containing other packages from subdirectories that are part of the parent package.
- setup.py generates files that are needed during package generation only (like setup.cfg), on the fly, and deletes them when no longer needed.
The above allows you to have nested namespaced packages (i.e. package2 can be a package you upload to PyPI, in addition to package2.subpack1 and package2.subpack2). The major thing it (currently) doesn't allow is using pip install -e to edit a single package (and not have the others editable). Given the way I develop, that is not a restriction.
The above embraces namespace packages, where many other approaches have problems with these (remember the last line of Zen of Python: Namespaces are one honking great idea – let’s do more of those)
Examples of the above can e.g be found in my packages ruamel.yaml (and e.g. ruamel.yaml.cmd), or generically by searching PyPI for ruamel.
As is probably obvious, the standard disclaimer: I am the author of those packages
As I use a utility to start packaging, which also runs the tests and does other sanity checks, the generic setup.py could be removed from the setup and inserted by that utility as well. But since subpackage detection is based upon setup.py availability or not, this requires some rework of the generic setup.py.
I have a Python package which contains submodules. Currently, my intention is to allow usage of functionality what the package exports, e.g.:
package_X
+-- __init__.py
+-- submodule_A.py
+-- submodule_B.py
Submodules are implementation details. Everything the user of the package needs to know is exported in the __init__.py file.
Now when building documentation with Sphinx, I get TOC and documentation as follows:
package_X
Submodules
submodule_A.py
submodule_B.py
Is there any setting or directive I can write to __init__.py or submodule_*.py to make public stuff appear as if it were directly in package_X? I am also ready to modify conf.py to make that work.
Since Python 2.6, it seems the documentation is in the new reStructuredText format, and it doesn't seem very easy to build a Texinfo Info file out of the box anymore.
I'm an Emacs addict and prefer my documentation installed in Info.
Does anyone have Python 2.6 or later docs in Texinfo format? How did you convert them? Or, is there a maintained build somewhere out there?
I know I can use w3m or haddoc to view the html docs - I really want them in Info.
I've played with Pandoc but after a few small experiments it doesn't seem to deal well with links between documents, and my larger experiment - running it across all docs cat'ed together to see what happens - is still chugging along two days since I started it!
Two good answers
Highlighting two answers below, because SO won't allow me to accept both answers:
#wilfred-hughes: Installing from MELPA is the quickest way to get pre-build info into Emacs
#alioth: Building it yourself looks like it's a lot easier than when I asked this question in 2009
Jon Waltman http://bitbucket.org/jonwaltman/sphinx-info has forked sphinx and written a texinfo builder, it can build the python documentation (I've yet done it). It seems that it will be merged soon into sphinx.
Here's the quick links for the downloads (temporary):
http://dl.dropbox.com/u/1276730/python.info
http://dl.dropbox.com/u/1276730/python.texi
Steps to generate python doc in texinfo format:
Download the python source code
Download and install the sphinx-info package (in a virtualenv)
Enter in the Python/Doc directory from the python sources
Edit the Makefile, to the build target replace $(PYTHON) tools/sphinx-build.py with sphinx-build, then add this target to the makefile, pay attention, the space before echo is a TAB:
texinfo: BUILDER = texinfo
texinfo: build
#echo
#echo "Build finished. The Texinfo files are in _build/texinfo."
#echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
Edit the Python/Doc/conf.py adding:
texinfo_documents = [
('contents', 'python', 'Python Documentation', 'Georg Brandl',
'Python', 'The Python Programming Language', 'Documentation tools',
1),
]
Then run make texinfo and it should produce the texifile in the build/texinfo directory.
To generate the info file run makeinfo python.texi
I've packaged up the Python docs as a texinfo file.
If you're using Emacs with MELPA, you can simply install this with M-x package-install python-info.
With no doubt it would be cool and challenging to generate the Python documentation on your particular Python version by yourself. Just follow EmacsWiki, or feel free to compile it locally (at Debian Jessy for Python3.4.2):
sudo apt-get install python3-sphinx
cd ~/Desktop
wget https://www.python.org/ftp/python/3.4.2/Python-3.4.2rc1.tar.xz
tar -xf Python-3.4.2rc1.tar.xz
cd Python-3.4.2rc1/Doc/
sphinx-build -b texinfo -d build/doctrees . build/texinfo
# extra time to build
cd build/texinfo/
makeinfo python.texi
# extra time for convertation
I got this tree:
.
├── logging_flow.png
├── Makefile
├── pathlib-inheritance.png
├── python.info
├── python.info-1
├── python.info-10
├── python.info-11
├── python.info-12
├── python.info-13
├── python.info-14
├── python.info-15
├── python.info-16
├── python.info-17
├── python.info-18
├── python.info-19
├── python.info-2
├── python.info-20
├── python.info-21
├── python.info-22
├── python.info-23
├── python.info-24
├── python.info-25
├── python.info-26
├── python.info-27
├── python.info-28
├── python.info-29
├── python.info-3
├── python.info-30
├── python.info-31
├── python.info-32
├── python.info-33
├── python.info-34
├── python.info-4
├── python.info-5
├── python.info-6
├── python.info-7
├── python.info-8
├── python.info-9
├── python.texi
├── python-video-icon.png
├── tulip_coro.png
└── turtle-star.png
And now it is possible to review python documentation natively in Emacs by
C-u C-h i python-info RET
python-info is a filename (fourth in the tree above), and even to bookmark some arbitrary nodes for habitual and regular reviewing convenience.
For those following this question in the hope of an answer, I found another rst2texinfo implementation which you might like to try:
http://bitbucket.org/jonwaltman/rst2texinfo/src
Another "workaround" is to execute pydoc as suggested by Nikokrock directly in Emacs:
(defun pydoc (&optional arg)
(interactive)
(when (not (stringp arg))
(setq arg (thing-at-point 'word)))
(setq cmd (concat "pydoc " arg))
(ad-activate-regexp "auto-compile-yes-or-no-p-always-yes")
(shell-command cmd)
(setq pydoc-buf (get-buffer "*Shell Command Output*"))
(switch-to-buffer-other-window pydoc-buf)
(python-mode)
(ad-deactivate-regexp "auto-compile-yes-or-no-p-always-yes")
)
Michael Ernst used to maintain Info formats of Python docs:
http://www.cs.washington.edu/homes/mernst/software/#python-info
You can try using his makefile and html2texi script to generate an updated version. Both are linked at the above URL. I'm not sure how well it works now (the last version was around 2001), but his script is well commented (grep for "python").
Python docs are now generated using Sphynx framework. This framework does not have texinfo output format. Currently it has:
HTML
latex
plain text
Maybe you can get what you want using the Latex output. With the text output you will lost the cross ref.
Personnaly I prefer using pydoc when I want textual output. With Vim I have a shorcut to call pydoc and open a window with the doc for the entity under my cursor...
For Python 3.8.0 and later, pre-built Info files are available at https://www.python.org/ftp/python/doc and/or https://docs.python.org/3/archives/.
The Ubuntu distribution provides packages pythonX.Y-doc (which include the documentation in Info format) at least since 18.04 (bionic); in 19.04 X.Y stands for 2.7, 3.7 and 3.8. The package does not have many dependencies, I assume it is possible to install it in other distributions too.
Believe it or not, the Python project actually provides us a way to do this through various Makefiles. The files utilize the Python Sphinx project to generate a texi file which makeinfo can then convert to info, the format Emacs uses for documentation.
In addition to Python3000, these instructions require GNU Make and Texinfo. These are packaged in most Linux distributions. Different distros may use different naming conventions. Refer to your distro's documentation for the corresponding package names. For Debian based distros:
# install make to utilize the Makefiles provided by the Python project
~/$ sudo apt-get install make
# install texinfo for the `makeinfo` command
~/$ sudo apt-get install texinfo
Package names are usually similar for non-Debian systems. For Windows users, I recommend WSL or creating a virtual machine.
1. Download the documentation
Navigate to https://www.python.org/ftp/python/ and download the tarball for your Python version. It will look like:
https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tar.xz
You can use wget to download the tarball and tar to unpack it. The options x and f are for "extract file":
# download the tarball
~/$ wget https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tar.xz
# extract the tarball
~/$ tar xf Python-3.7.9.tar.xz
2. Run make venv in Python-X.Y.Z/Doc
Sphinx requires more dependencies than are bundled with the basic pip install. Fortunately, the Python project provides a Makefile to create the necessary environment. See the Makefile for precise details.
# Navigate to the Doc/ directory
~/$ cd Python-3.7.9/Doc
# "create a venv with necessary tools"
~/Python-3.7.9/Doc$ make venv
# activate the venv created by make
~/Python-3.7.9/Doc$ source venv/bin/activate
3. Run sphinx-build
Now that the correct environment is set up, we can run Sphinx. This call creates a cache used during generation with the -d option. The documentation files found in the current directory are converted by the texinfo "builder" and output to build/texinfo:
# -b: Use the textinfo builder
# -d: Create "doctree pickles" cache in doctrees/
# Use the current directory as source
# Output to build/texinfo
(venv) ~/Python-3.7.9/Doc$ sphinx-build -b texinfo -d build/doctrees . build/texinfo
4. Use makeinfo to generate the info file
Again, the Python maintainers have given us what we need (even if they haven't documented it well). The previous command created a texi file along with another Makefile. The Makefile calls makeinfo.
# Navigate to the output directory
(venv) ~/Python-3.7.9/Doc$ cd build/texinfo
# Run the generated Makefile
(venv) ~/Python-3.7.9/Doc/build/texinfo$ make
# Hark, unto us an info file is born
(venv) ~/Python-3.7.9/Doc/build/texinfo$ ls
Makefile python-figures python.info python.texi
Like Indiana Jones, you behold the Holy Grail. Many have perished in this journey; you have prevailed. Take a moment to celebrate.
Note: The makeinfo conversion yields errors for me. No matter, I say. The desired info is obtained and I greedily drink from it.
5. Load python.info into Emacs...
Use C-u C-h i to directly open python.info.
To install the info file within the Emacs Help Directory node, first
check C-h v Info-default-directory-list for where info files are stored. Put python.info file there. There may be a file called dir in that directory. The dir file is generated by texinfo and contains the node listing. If no dir file exists, don't worry, that's what we're creating. Note that it's not recommended to edit dir files manually1.
Run update-info-dir in whichever directory you put python.info. This will update (or create) dir with python.info.
For complete details about the texinfo system, see https://www.gnu.org/software/texinfo/manual/texinfo/html_node/Installing-an-Info-File.html.
1Aside from human error, like mistyping a reference, issues may arise due to "malformed" dir files.