Python PEX loading - python

I've been trying to wrap my head around the python pex utility (https://pex.readthedocs.org/en/latest/) for bundling some applications into .pex files for deployment.
My app isn't large enough to require twitters pants build tool, as well as I have some build requirements which pants doesn't address. I did however, try the pants tools build system using python_binary which resulted in pex files with the sources loaded into the pex files root. The BUILD files in pants accept a sources property for python_binary which can be a glob for files within the directory the build is running in, however, pants is using the pex programmatic API and not the command-line utility.
The problem is when I use the pex command-line utility on it's own, it seems to want distributions (i.e. folders set up with a setup.py, etc...) and wants to install my code into the .deps folder in the pex file rather than just copying the python files into the root of the pex file like pants seems to do.
Is copying the files over (not installing a package) not possible through the command-line pex tool?

As of pex 1.0.0 there is no facility to glob up files and directories directly, you must have a setup.py as you suggest, or use pants (no longer Twitter's by the way - independent).
So you have 3 paths forward (#1 you already know, but spelling it out for others):
Create a setup.py and point the pex tool at its dir
$ tree -h
.
├── [4.0K] lib
│   ├── [ 0] __init__.py
│   ├── [ 38] lib.py
│   └── [ 68] main.py
└── [ 76] setup.py
1 directory, 4 files
$ cat lib/lib.py
def func():
return 'func in lib'
$ cat lib/main.py
from .lib import func
if __name__ == '__main__':
print(func())
$ cat setup.py
from setuptools import setup
setup(
name='lib',
packages=['lib']
)
$ pex . -e lib.main -o lib.pex
$ ./lib.pex
func in lib
NB: The . in the pex command line is the bit pointing pex at this dir's setup.py
File an issue against pex to support a set of files in place of a requirement / setup.py.
You can do that here.
File issues against pants to support the requirements you have that it does not address.
You can do that here
As a pants committer I can say that we're working towards making pants easier and easier to use such that no project is too small. You should be able to pip install pantsbuild.pants.backend.python && touch pants.ini and be up and running in a python-only repo but we're not there today.

Related

How to package my python project to be runnable from command line?

I've coded my python project and have succeeded in publishing it to test pypi. However, now I can't figure out how to correctly configure it as a console script. Upon running my_project on the command line, I get the following stack trace:
Traceback (most recent call last):
File "/home/thatcoolcoder/.local/bin/my_project", line 5, in <module>
from my_project.__main__ import main
ModuleNotFoundError: No module named 'my_project'
Clearly, it's created a script to run but the script is then failing to import my actual code.
Folder structure:
pyproject.toml
setup.cfg
my_project
├── __init__.py (empty)
├── __main__.py
Relevant sections of setup.cfg:
[metadata]
name = my-project
version = 1.0.5
...
[options]
package_dir =
= my_project
packages = find:
...
[options.packages.find]
where = my_project
[options.entry_points]
console_scripts =
my_project = my_project.__main__:main
pyproject.toml (probably not relevant)
[build-system]
requires = [
"setuptools>=42",
"wheel"
]
__main__.py:
from my_project import foo
def main():
foo.bar()
if __name__ == '__main__':
main()
To build and upload, I'm running the following: (python is python 3.10)
python -m build
python -m twine upload --repository testpypi dist/*
Then to install and run:
pip install -i https://test.pypi.org/pypi/ --extra-index-url https://pypi.org/simple my-project --upgrade
my_project
How can I make this console script work?
Also, this current method of setting console_scripts only allows it to be run as my_project; is it possible to also make it work by python -m my_project? Or perhaps this will work once my main issue is fixed.
It's funny, but I had the same frustration when trying to install scripts on multiple platforms. (As Python calls them; posix and nt.)
So I wrote setup-py-script in 2020. It's up on github now.
It installs scripts that use their own modules as a self-contained zip-file. (This method was inspired by youtube-dl.) That means no more leftover files when you delete a script but forget to remove the module et cetera.
It does not require root or administrator privileges; installation is done in user-accessible directories.
You might have to structure your project slightly differently; the script itself is not in the module directory. See the project README.
I finally got back to this problem today and it appears that I was using an incorrect source layout, which caused the pip module installation to not work. I switched to a directory structure like this one:
├── src
│ └── mypackage
│ ├── __init__.py
│ └── mod1.py
├── setup.py
└── setup.cfg
and modified the relevant parts of my setup.cfg:
[options]
package_dir=
=src
packages=find:
[options.packages.find]
where=src
Then I can run it like python -m mypackage. This also made the console scripts work. It works on Linux but I presume it also works on other systems.

PyPI - folder inside the package folder not getting uploaded [duplicate]

When using setuptools, I can not get the installer to pull in any package_data files. Everything I've read says that the following is the correct way to do it. Can someone please advise?
setup(
name='myapp',
packages=find_packages(),
package_data={
'myapp': ['data/*.txt'],
},
include_package_data=True,
zip_safe=False,
install_requires=['distribute'],
)
where myapp/data/ is the location of the data files.
I realize that this is an old question, but for people finding their way here via Google: package_data is a low-down, dirty lie. It is only used when building binary packages (python setup.py bdist ...) but not when building source packages (python setup.py sdist ...). This is, of course, ridiculous -- one would expect that building a source distribution would result in a collection of files that could be sent to someone else to built the binary distribution.
In any case, using MANIFEST.in will work both for binary and for source distributions.
I just had this same issue. The solution, was simply to remove include_package_data=True.
After reading here, I realized that include_package_data aims to include files from version control, as opposed to merely "include package data" as the name implies. From the docs:
The data files [of include_package_data] must be under CVS or Subversion control
...
If you want finer-grained control over what files are included (for example, if
you have documentation files in your package directories and want to exclude
them from installation), then you can also use the package_data keyword.
Taking that argument out fixed it, which is coincidentally why it also worked when you switched to distutils, since it doesn't take that argument.
Following #Joe 's recommendation to remove the include_package_data=True line also worked for me.
To elaborate a bit more, I have no MANIFEST.in file. I use Git and not CVS.
Repository takes this kind of shape:
/myrepo
- .git/
- setup.py
- myproject
- __init__.py
- some_mod
- __init__.py
- animals.py
- rocks.py
- config
- __init__.py
- settings.py
- other_settings.special
- cool.huh
- other_settings.xml
- words
- __init__.py
word_set.txt
setup.py:
from setuptools import setup, find_packages
import os.path
setup (
name='myproject',
version = "4.19",
packages = find_packages(),
# package_dir={'mypkg': 'src/mypkg'}, # didnt use this.
package_data = {
# If any package contains *.txt or *.rst files, include them:
'': ['*.txt', '*.xml', '*.special', '*.huh'],
},
#
# Oddly enough, include_package_data=True prevented package_data from working.
# include_package_data=True, # Commented out.
data_files=[
# ('bitmaps', ['bm/b1.gif', 'bm/b2.gif']),
('/opt/local/myproject/etc', ['myproject/config/settings.py', 'myproject/config/other_settings.special']),
('/opt/local/myproject/etc', [os.path.join('myproject/config', 'cool.huh')]),
#
('/opt/local/myproject/etc', [os.path.join('myproject/config', 'other_settings.xml')]),
('/opt/local/myproject/data', [os.path.join('myproject/words', 'word_set.txt')]),
],
install_requires=[ 'jsonschema',
'logging', ],
entry_points = {
'console_scripts': [
# Blah...
], },
)
I run python setup.py sdist for a source distrib (haven't tried binary).
And when inside of a brand new virtual environment, I have a myproject-4.19.tar.gz, file,
and I use
(venv) pip install ~/myproject-4.19.tar.gz
...
And other than everything getting installed to my virtual environment's site-packages, those special data files get installed to /opt/local/myproject/data and /opt/local/myproject/etc.
include_package_data=True worked for me.
If you use git, remember to include setuptools-git in install_requires. Far less boring than having a Manifest or including all path in package_data ( in my case it's a django app with all kind of statics )
( pasted the comment I made, as k3-rnc mentioned it's actually helpful as is )
Using setup.cfg (setuptools ≥ 30.3.0)
Starting with setuptools 30.3.0 (released 2016-12-08), you can keep your setup.py very small and move the configuration to a setup.cfg file. With this approach, you could put your package data in an [options.package_data] section:
[options.package_data]
* = *.txt, *.rst
hello = *.msg
In this case, your setup.py can be as short as:
from setuptools import setup
setup()
For more information, see configuring setup using setup.cfg files.
There is some talk of deprecating setup.cfg in favour of pyproject.toml as proposed in PEP 518, but this is still provisional as of 2020-02-21.
Update: This answer is old and the information is no longer valid. All setup.py configs should use import setuptools. I've added a more complete answer at https://stackoverflow.com/a/49501350/64313
I solved this by switching to distutils. Looks like distribute is deprecated and/or broken.
from distutils.core import setup
setup(
name='myapp',
packages=['myapp'],
package_data={
'myapp': ['data/*.txt'],
},
)
I had the same problem for a couple of days but even this thread wasn't able to help me as everything was confusing. So I did my research and found the following solution:
Basically in this case, you should do:
from setuptools import setup
setup(
name='myapp',
packages=['myapp'],
package_dir={'myapp':'myapp'}, # the one line where all the magic happens
package_data={
'myapp': ['data/*.txt'],
},
)
The full other stackoverflow answer here
I found this post while stuck on the same problem.
My experience contradicts the experiences in the other answers.
include_package_data=True does include the data in the
bdist! The explanation in the setuptools
documentation
lacks context and troubleshooting tips, but
include_package_data works as advertised.
My setup:
Windows / Cygwin
git version 2.21.0
Python 3.8.1 Windows distribution
setuptools v47.3.1
check-manifest v0.42
Here is my how-to guide.
How-to include package data
Here is the file structure for a project I published on PyPI.
(It installs the application in __main__.py).
├── LICENSE.md
├── MANIFEST.in
├── my_package
│ ├── __init__.py
│ ├── __main__.py
│ └── _my_data <---- folder with data
│ ├── consola.ttf <---- data file
│ └── icon.png <---- data file
├── README.md
└── setup.py
Starting point
Here is a generic starting point for the setuptools.setup() in
setup.py.
setuptools.setup(
...
packages=setuptools.find_packages(),
...
)
setuptools.find_packages() includes all of my packages in the
distribution. My only package is my_package.
The sub-folder with my data, _my_data, is not considered a
package by Python because it does not contain an __init__.py,
and so find_packages() does not find it.
A solution often-cited, but incorrect, is to put an empty
__init__.py file in the _my_data folder.
This does make it a package, so it does include the folder
_my_data in the distribution. But the data files inside
_my_data are not included.
So making _my_data into a package does not help.
The solution is:
the sdist already contains the data files
add include_package_data=True to include the data files in the bdist as well
Experiment (how to test the solution)
There are three steps to make this a repeatable experiment:
$ rm -fr build/ dist/ my_package.egg-info/
$ check-manifest
$ python setup.py sdist bdist_wheel
I will break these down step-by-step:
Clean out the old build:
$ rm -fr build/ dist/ my_package.egg-info/
Run check-manifest to be sure MANIFEST.in matches the
Git index of files under version control:
$ check-manifest
If MANIFEST.in does not exist yet, create it from the Git
index of files under version control:
$ check-manifest --create
Here is the MANIFEST.in that is created:
include *.md
recursive-include my_package *.png
recursive-include my_package *.ttf
There is no reason to manually edit this file.
As long as everything that should be under version control is
under version control (i.e., is part of the Git index),
check-manifest --create does the right thing.
Note: files are not part of the Git index if they are either:
ignored in a .gitignore
excluded in a .git/info/exclude
or simply new files that have not been added to the index yet
And if any files are under version control that should not be
under version control, check-manifest issues a warning and
specifies which files it recommends removing from the Git index.
Build:
$ python setup.py sdist bdist_wheel
Now inspect the sdist (source distribution) and bdist_wheel
(build distribution) to see if they include the data files.
Look at the contents of the sdist (only the relevant lines are
shown below):
$ tar --list -f dist/my_package-0.0.1a6.tar.gz
my_package-0.0.1a6/
...
my_package-0.0.1a6/my_package/__init__.py
my_package-0.0.1a6/my_package/__main__.py
my_package-0.0.1a6/my_package/_my_data/
my_package-0.0.1a6/my_package/_my_data/consola.ttf <-- yay!
my_package-0.0.1a6/my_package/_my_data/icon.png <-- yay!
...
So the sdist already includes the data files because they are
listed in MANIFEST.in. There is nothing extra to do to include
the data files in the sdist.
Look at the contents of the bdist (it is a .zip file, parsed
with zipfile.ZipFile):
$ python check-whl.py
my_package/__init__.py
my_package/__main__.py
my_package-0.0.1a6.dist-info/LICENSE.md
my_package-0.0.1a6.dist-info/METADATA
my_package-0.0.1a6.dist-info/WHEEL
my_package-0.0.1a6.dist-info/entry_points.txt
my_package-0.0.1a6.dist-info/top_level.txt
my_package-0.0.1a6.dist-info/RECORD
Note: you need to create your own check-whl.py script to produce the
above output. It is just three lines:
from zipfile import ZipFile
path = "dist/my_package-0.0.1a6-py3-none-any.whl" # <-- CHANGE
print('\n'.join(ZipFile(path).namelist()))
As expected, the bdist is missing the data files.
The _my_data folder is completely missing.
What if I create a _my_data/__init__.py? I repeat the
experiment and I find the data files are still not there! The
_my_data/ folder is included but it does not contain the data
files!
Solution
Contrary to the experience of others, this does work:
setuptools.setup(
...
packages=setuptools.find_packages(),
include_package_data=True, # <-- adds data files to bdist
...
)
With the fix in place, redo the experiment:
$ rm -fr build/ dist/ my_package.egg-info/
$ check-manifest
$ python.exe setup.py sdist bdist_wheel
Make sure the sdist still has the data files:
$ tar --list -f dist/my_package-0.0.1a6.tar.gz
my_package-0.0.1a6/
...
my_package-0.0.1a6/my_package/__init__.py
my_package-0.0.1a6/my_package/__main__.py
my_package-0.0.1a6/my_package/_my_data/
my_package-0.0.1a6/my_package/_my_data/consola.ttf <-- yay!
my_package-0.0.1a6/my_package/_my_data/icon.png <-- yay!
...
Look at the contents of the bdist:
$ python check-whl.py
my_package/__init__.py
my_package/__main__.py
my_package/_my_data/consola.ttf <--- yay!
my_package/_my_data/icon.png <--- yay!
my_package-0.0.1a6.dist-info/LICENSE.md
my_package-0.0.1a6.dist-info/METADATA
my_package-0.0.1a6.dist-info/WHEEL
my_package-0.0.1a6.dist-info/entry_points.txt
my_package-0.0.1a6.dist-info/top_level.txt
my_package-0.0.1a6.dist-info/RECORD
How not to test if data files are included
I recommend troubleshooting/testing using the approach outlined
above to inspect the sdist and bdist.
pip install in editable mode is not a valid test
Note: pip install -e . does not show if data files are
included in the bdist.
The symbolic link causes the installation to behave as if the
data files are included (because they already exist locally on
the developer's computer).
After pip install my_package, the data files are in the
virtual environment's lib/site-packages/my_package/ folder,
using the exact same file structure shown above in the list of
the whl contents.
Publishing to TestPyPI is a slow way to test
Publishing to TestPyPI and then installing and looking in
lib/site-packages/my_packages is a valid test, but it is too
time-consuming.
Ancient question and yet... package management of python really leaves a lot to be desired. So I had the use case of installing using pip locally to a specified directory and was surprised both package_data and data_files paths did not work out. I was not keen on adding yet another file to the repo so I ended up leveraging data_files and setup.py option --install-data; something like this
pip install . --install-option="--install-data=$PWD/package" -t package
Moving the folder containing the package data into to module folder solved the problem for me.
See this question: MANIFEST.in ignored on "python setup.py install" - no data files installed?
Just remove the line:
include_package_data=True,
from your setup script, and it will work fine. (Tested just now with latest setuptools.)
Like others in this thread, I'm more than a little surprised at the combination of longevity and still a lack of clarity, BUT the best answer for me was using check-manifest as recommended in the answer from #mike-gazes
So, using just a setup.cfg and no setup.py and additional text and python files required in the package, what worked for me was keeping this in setup.cfg:
[options]
packages = find:
include_package_data = true
and updating the MANIFEST.in based on the check-manifest output:
include *.in
include *.txt
include *.yml
include LICENSE
include tox.ini
recursive-include mypkg *.py
recursive-include mypkg *.txt
For a directory structure like:
foo/
├── foo
│   ├── __init__.py
│   ├── a.py
│   └── data.txt
└── setup.py
and setup.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from setuptools import setup
NAME = 'foo'
DESCRIPTION = 'Test library to check how setuptools works'
URL = 'https://none.com'
EMAIL = 'gzorp#bzorp.com'
AUTHOR = 'KT'
REQUIRES_PYTHON = '>=3.6.0'
setup(
name=NAME,
version='0.0.0',
description=DESCRIPTION,
author=AUTHOR,
author_email=EMAIL,
python_requires=REQUIRES_PYTHON,
url=URL,
license='MIT',
classifiers=[
'Programming Language :: Python',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6',
],
packages=['foo'],
package_data={'foo': ['data.txt']},
include_package_data=True,
install_requires=[],
extras_require={},
cmdclass={},
)
python setup.py bdist_wheel works.
Starting with Setuptools 62.3.0, you can now use recursive wildcards ("**") to include a (sub)directory recursively. This way you can include whole folders with all their folders and files in it.
For example, when using a pyproject.toml file, this is how you include two folders recursively:
[tool.setuptools.package-data]
"ema_workbench.examples.data" = ["**"]
"ema_workbench.examples.models" = ["**"]
But you can also only include certain file-types, in a folder and all subfolders. If you want to include all markdown (.md) files for example:
[tool.setuptools.package-data]
"ema_workbench.examples.data" = ["**/*.md"]
It should also work when using setup.py or setup.cfg.
See https://github.com/pypa/setuptools/pull/3309 for the details.

How to tell pbr to include non-code files in package

I only run into this problem when I build in a python:alpine image. Reproducing it is a bit of a pain, but these are the steps:
Docker container setup:
$ docker run -it python:3.7-rc-alpine /bin/ash
$ pip install pbr
Small package setup, including non-python files
test
├── .git
├── setup.cfg
├── setup.py
└── src
   └── test
   ├── __init__.py
├── test.yml
   └── sub_test
      ├── __init__.py
      └── test.yml
setup.py:
from setuptools import find_packages, setup
setup(
setup_requires=['pbr'],
pbr=True,
package_dir={'': 'src'},
packages=find_packages(where='src'),
)
setup.cfg:
[metadata]
name = test
All other files are empty. I copy them to the container with docker cp test <docker_container>:/test.
Back in the container I now try to build the package with cd test; pip wheel -w wheel ., the test.yml in test/src/test will be included in it, but the one in test/src/test/sub_test won't.
I have no clue why this happens, since the (pitifully sparse, and imo quite confusing) documentation of pbr on that matter states that
Just like AUTHORS and ChangeLog, why keep a list of files you wish to include when you can find many of these in git. MANIFEST.in generation ensures almost all files stored in git, with the exception of .gitignore, .gitreview and .pyc files, are automatically included in your distribution.
I could not find a pbr-parameter that lets me explicitly include some file or file type, which I expected to exist.
Creating a MANIFEST.in with import src/test/sub_test.test.yml actually solves this problem, but I'd rather understand and avoid this behavior all together instead.
pbr needs git in order to correctly compile its files-to-include list, so the problem can be solved by installing git into the build environment before building the package. With an alpine image, that would be apk add --no-cache git.
pbr uses the .git file to figure out which files should be part of the package and which ones shouldn't. The short version is that it grabs the intersection of the file list from the packages parameter in the setup-call in setup.py and everything that is commited or staged in the currently checked out git branch.
So if the project came with no .git file, you'd need to additionally execute git init; git add src as well.
The reason for the 'bug' is that pbr silently assumes that all .py files should be added regardles of them being commited or not, which makes the actual problem harder to identify. It will also only throw an error if it can't find a .git file, and not if it is there but it can't get any info from it because git isn't installed.

Excluding source files from built rpm distribution with setuptool

I have a typical project structure that looks as follows:
EngineEmulator
src
ship
engine
emulator
mapping
tests
emulator
mapping
utils
common
doc
....
tools
....
setup.py
MANIFEST.in
setup.cfg
README.rst
My setup.py looks as follows:
from setuptools import setup, find_packages
setup(
name='Engine',
version=1.0.0,
description='Engine Project',
package_dir={'': 'src'},
packages=find_packages(
'src',
exclude=["*.tests", "*.tests.*", "tests.*", "tests"]),
install_requires =['pycrypto',
'kombu >=1.1.3'],
author='Demo',
author_email='demo#eliza.net'
license='MIT',
classifiers=[
'Topic :: Demo Engine',
'Development Status:: 3 - Iteration',
'Programming Language :: Python -2.6'
]
)
My setup.cfg looks as follows:
[egg_info]
tag_build = .dev
tag_svn_revision = 1
[rotate]
#keep last 15 eggs, clean up order
match = .egg
keep = 15
And My MANIFEST.in looks as follows:
include README.rst
recursive-include src/ship/Engine
prune src/utils
prune src/ship/tests
prune tools/
When I run python setup.py bdist_egg and python setup.py bdist_rpm I get the egg file and two rpm files generated (noarch.rpm and src.rpm).
In my destination machine when I run easy_install <generated egg file> my eg.info file gets copied over but the source files don't get copied over to /usr/lib/python2.6/site-packages. I was expecting I would have a directory called Engine.
Can anybody point out what I am doing wrong? Thanks in advance.
Try to keep things as simple as possible.
Quick check with sdist
Try this:
$ python setup.py sdist
It shall create source distribution file for your package.
It is in zip format, so unpack it and check, if there are all expected files inside present.
If not, you have to find the reason, why expected files are missing in your distribution.
Checking things step by step (and simplifying)
Do you use .py extension?
May be stupid question, but in your file listing I do not see any py files inside of src tree.
In case you have there just files without .py extension, find_packages will not find anything.
Where do you have your __init__.py files located?
Let us know, where the files are:
$ cd src
$ find . -name "*.py"
If you miss __init__.py, find_packages will not find whole package.
Remove utils package
Why do you have it there?
Better have it installed out of your source code you develop or move it subdirectory in your project
root.
This will render prune src/utils unnecessary in your MANIFEST.in.
Put into MANIFEST.in only what must be there
If you read doc for MANIFEST.in, it states, what files are included automatically (all, what
mentioned in arguments of setup function, so in your case all python source files returned by
find_packages).
For this reason, you shall remove recursive-include src/shop/Engine as it shall be already
included by setup call.
Remove prune lines.
src/utils shall not be in your source tree - it is just messing things up.
tools is not to be included, so there is no need to prune it.
src/ship/tests can be there, it will not harm, if you keep these files in the destribution.
Assert, what packages were found
Make sure, your setup get proper names for packages.
For this purpuse, you can call find_package sooner and assert it containts, what you expect.
(temporarily) remove setup.cfg
Just to keep things simpler.
Proposed project reorganization
You shall have file structure in similar manner as follows:
src/ship/__init__.py
src/ship/engine/__init__.py
src/ship/engine/emulator/__init__.py
src/ship/engine/emulator/module.py
src/ship/engine/emulator/module2.py
src/ship/engine/mapping/other.py
src/ship/engine/mapping/another.py
src/ship/tests/__init__.py
src/ship/tests/emulator/__init__.py
src/ship/tests/emulator/test_module.py
src/ship/tests/emulator/test_module2.py
src/ship/tests/mapping/__init__.py
src/ship/tests/mapping/test_other.py
src/ship/tests/mapping/test_another.py
doc
doc/index.rst
tools
tools/knife.py
setup.py
MANIFEST.in
README.rst
setup.py
from setuptools import setup, find_packages
packages=find_packages("src")
assert "ship.engine" in packages
assert "ship.engine.emulator" in packages
assert "ship.engine.mapping" in packages
#etc
install_requires =['pycrypto', 'kombu>=1.1.3'] #watch the spaces around `>=`, shall not be there
setup(
name="Engine",
package_dir={'': 'src'},
packages=packages,
install_requires=install_requires
)
MANIFEST.in
include README.rst
Conclusions
It might happen, that running
$ python setup.py sdist
would fail on asserts. This is sign, some of expected files are missing. Check that.
After you make your project living in simple way, you might add more details around (and do it step
by step to be sure, you do not break something).

How do you get Python documentation in Texinfo Info format?

Since Python 2.6, it seems the documentation is in the new reStructuredText format, and it doesn't seem very easy to build a Texinfo Info file out of the box anymore.
I'm an Emacs addict and prefer my documentation installed in Info.
Does anyone have Python 2.6 or later docs in Texinfo format? How did you convert them? Or, is there a maintained build somewhere out there?
I know I can use w3m or haddoc to view the html docs - I really want them in Info.
I've played with Pandoc but after a few small experiments it doesn't seem to deal well with links between documents, and my larger experiment - running it across all docs cat'ed together to see what happens - is still chugging along two days since I started it!
Two good answers
Highlighting two answers below, because SO won't allow me to accept both answers:
#wilfred-hughes: Installing from MELPA is the quickest way to get pre-build info into Emacs
#alioth: Building it yourself looks like it's a lot easier than when I asked this question in 2009
Jon Waltman http://bitbucket.org/jonwaltman/sphinx-info has forked sphinx and written a texinfo builder, it can build the python documentation (I've yet done it). It seems that it will be merged soon into sphinx.
Here's the quick links for the downloads (temporary):
http://dl.dropbox.com/u/1276730/python.info
http://dl.dropbox.com/u/1276730/python.texi
Steps to generate python doc in texinfo format:
Download the python source code
Download and install the sphinx-info package (in a virtualenv)
Enter in the Python/Doc directory from the python sources
Edit the Makefile, to the build target replace $(PYTHON) tools/sphinx-build.py with sphinx-build, then add this target to the makefile, pay attention, the space before echo is a TAB:
texinfo: BUILDER = texinfo
texinfo: build
#echo
#echo "Build finished. The Texinfo files are in _build/texinfo."
#echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
Edit the Python/Doc/conf.py adding:
texinfo_documents = [
('contents', 'python', 'Python Documentation', 'Georg Brandl',
'Python', 'The Python Programming Language', 'Documentation tools',
1),
]
Then run make texinfo and it should produce the texifile in the build/texinfo directory.
To generate the info file run makeinfo python.texi
I've packaged up the Python docs as a texinfo file.
If you're using Emacs with MELPA, you can simply install this with M-x package-install python-info.
With no doubt it would be cool and challenging to generate the Python documentation on your particular Python version by yourself. Just follow EmacsWiki, or feel free to compile it locally (at Debian Jessy for Python3.4.2):
sudo apt-get install python3-sphinx
cd ~/Desktop
wget https://www.python.org/ftp/python/3.4.2/Python-3.4.2rc1.tar.xz
tar -xf Python-3.4.2rc1.tar.xz
cd Python-3.4.2rc1/Doc/
sphinx-build -b texinfo -d build/doctrees . build/texinfo
# extra time to build
cd build/texinfo/
makeinfo python.texi
# extra time for convertation
I got this tree:
.
├── logging_flow.png
├── Makefile
├── pathlib-inheritance.png
├── python.info
├── python.info-1
├── python.info-10
├── python.info-11
├── python.info-12
├── python.info-13
├── python.info-14
├── python.info-15
├── python.info-16
├── python.info-17
├── python.info-18
├── python.info-19
├── python.info-2
├── python.info-20
├── python.info-21
├── python.info-22
├── python.info-23
├── python.info-24
├── python.info-25
├── python.info-26
├── python.info-27
├── python.info-28
├── python.info-29
├── python.info-3
├── python.info-30
├── python.info-31
├── python.info-32
├── python.info-33
├── python.info-34
├── python.info-4
├── python.info-5
├── python.info-6
├── python.info-7
├── python.info-8
├── python.info-9
├── python.texi
├── python-video-icon.png
├── tulip_coro.png
└── turtle-star.png
And now it is possible to review python documentation natively in Emacs by
C-u C-h i python-info RET
python-info is a filename (fourth in the tree above), and even to bookmark some arbitrary nodes for habitual and regular reviewing convenience.
For those following this question in the hope of an answer, I found another rst2texinfo implementation which you might like to try:
http://bitbucket.org/jonwaltman/rst2texinfo/src
Another "workaround" is to execute pydoc as suggested by Nikokrock directly in Emacs:
(defun pydoc (&optional arg)
(interactive)
(when (not (stringp arg))
(setq arg (thing-at-point 'word)))
(setq cmd (concat "pydoc " arg))
(ad-activate-regexp "auto-compile-yes-or-no-p-always-yes")
(shell-command cmd)
(setq pydoc-buf (get-buffer "*Shell Command Output*"))
(switch-to-buffer-other-window pydoc-buf)
(python-mode)
(ad-deactivate-regexp "auto-compile-yes-or-no-p-always-yes")
)
Michael Ernst used to maintain Info formats of Python docs:
http://www.cs.washington.edu/homes/mernst/software/#python-info
You can try using his makefile and html2texi script to generate an updated version. Both are linked at the above URL. I'm not sure how well it works now (the last version was around 2001), but his script is well commented (grep for "python").
Python docs are now generated using Sphynx framework. This framework does not have texinfo output format. Currently it has:
HTML
latex
plain text
Maybe you can get what you want using the Latex output. With the text output you will lost the cross ref.
Personnaly I prefer using pydoc when I want textual output. With Vim I have a shorcut to call pydoc and open a window with the doc for the entity under my cursor...
For Python 3.8.0 and later, pre-built Info files are available at https://www.python.org/ftp/python/doc and/or https://docs.python.org/3/archives/.
The Ubuntu distribution provides packages pythonX.Y-doc (which include the documentation in Info format) at least since 18.04 (bionic); in 19.04 X.Y stands for 2.7, 3.7 and 3.8. The package does not have many dependencies, I assume it is possible to install it in other distributions too.
Believe it or not, the Python project actually provides us a way to do this through various Makefiles. The files utilize the Python Sphinx project to generate a texi file which makeinfo can then convert to info, the format Emacs uses for documentation.
In addition to Python3000, these instructions require GNU Make and Texinfo. These are packaged in most Linux distributions. Different distros may use different naming conventions. Refer to your distro's documentation for the corresponding package names. For Debian based distros:
# install make to utilize the Makefiles provided by the Python project
~/$ sudo apt-get install make
# install texinfo for the `makeinfo` command
~/$ sudo apt-get install texinfo
Package names are usually similar for non-Debian systems. For Windows users, I recommend WSL or creating a virtual machine.
1. Download the documentation
Navigate to https://www.python.org/ftp/python/ and download the tarball for your Python version. It will look like:
https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tar.xz
You can use wget to download the tarball and tar to unpack it. The options x and f are for "extract file":
# download the tarball
~/$ wget https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tar.xz
# extract the tarball
~/$ tar xf Python-3.7.9.tar.xz
2. Run make venv in Python-X.Y.Z/Doc
Sphinx requires more dependencies than are bundled with the basic pip install. Fortunately, the Python project provides a Makefile to create the necessary environment. See the Makefile for precise details.
# Navigate to the Doc/ directory
~/$ cd Python-3.7.9/Doc
# "create a venv with necessary tools"
~/Python-3.7.9/Doc$ make venv
# activate the venv created by make
~/Python-3.7.9/Doc$ source venv/bin/activate
3. Run sphinx-build
Now that the correct environment is set up, we can run Sphinx. This call creates a cache used during generation with the -d option. The documentation files found in the current directory are converted by the texinfo "builder" and output to build/texinfo:
# -b: Use the textinfo builder
# -d: Create "doctree pickles" cache in doctrees/
# Use the current directory as source
# Output to build/texinfo
(venv) ~/Python-3.7.9/Doc$ sphinx-build -b texinfo -d build/doctrees . build/texinfo
4. Use makeinfo to generate the info file
Again, the Python maintainers have given us what we need (even if they haven't documented it well). The previous command created a texi file along with another Makefile. The Makefile calls makeinfo.
# Navigate to the output directory
(venv) ~/Python-3.7.9/Doc$ cd build/texinfo
# Run the generated Makefile
(venv) ~/Python-3.7.9/Doc/build/texinfo$ make
# Hark, unto us an info file is born
(venv) ~/Python-3.7.9/Doc/build/texinfo$ ls
Makefile python-figures python.info python.texi
Like Indiana Jones, you behold the Holy Grail. Many have perished in this journey; you have prevailed. Take a moment to celebrate.
Note: The makeinfo conversion yields errors for me. No matter, I say. The desired info is obtained and I greedily drink from it.
5. Load python.info into Emacs...
Use C-u C-h i to directly open python.info.
To install the info file within the Emacs Help Directory node, first
check C-h v Info-default-directory-list for where info files are stored. Put python.info file there. There may be a file called dir in that directory. The dir file is generated by texinfo and contains the node listing. If no dir file exists, don't worry, that's what we're creating. Note that it's not recommended to edit dir files manually1.
Run update-info-dir in whichever directory you put python.info. This will update (or create) dir with python.info.
For complete details about the texinfo system, see https://www.gnu.org/software/texinfo/manual/texinfo/html_node/Installing-an-Info-File.html.
1Aside from human error, like mistyping a reference, issues may arise due to "malformed" dir files.

Categories