We Keep Coding

Is there a portable way to provide localization of a package distributed on PyPI?

Context:
This is kind of a followup to another question of mine.
I would like to provide localized versions of a package. Following the Python documentation, I have extracted a .pot file with pygettext, prepared a translation in a .po file, compiled it in a .mo file.
Everything is fine till there, and my package displays the translated messages.
But my final goal would be to make it available on PyPI. So I have done some research and found:
setuptools documentation: not even one single word about localization...
The Format of GNU MO Files
It explains that the format depends on the endianness of the platform where the file was generated. My understanding is that only the po files are portable...
What is the correct way to include localisation in python packages?
The answer is fully relevant and speaks of the setuptools/babel integration but:
that integration only allows to build mo file and does not speak of their distribution
author describes how they use it, no references for portability across systems
Babel: compile translation files when calling setup.py install
An interesting way, even if it requires the babel module on the target platform. Not so heavy but way heavier than my own package... In fact, the distributions contain only po files and they are compiled with babel at install time.
Question:
Is there a way to build platform specific wheels containing compiled mo files?
If not I will have to require babel on target and try to find my way through mo compilation at install time.

EDIT 7/12/2018:
After some work, I could build a specific package based on what is below in this answer. It can be used from other projects to automatically compile po files at build time through the magic of setuptools enty_points. It is now available on GitHUB (https://github.com/s-ball/mo_installer) and distributed on PyPI (https://pypi.org/project/mo_installer)
The researches that I did before asking the question gave me enough hints to reach a possible solution.
I can now say, thay is is possible to include a platform specific mo file in a wheel - unfortunately in my current solution the wheel gives no indication that it is platform specific. But the same solution allows to build a source distribution that build the mo file on the target platform.
Now for the details:
the tools needed to compile a mo file on the target:
Most solutions picked from Google or SO rely either on Babel or on the GNU gettext msgfmt program. But cPython tools include a pure Python module msgfmt.py that is enough here. Unfortunately, this tool is often not installed by default in many Linux/Unix-like. My solution just includes a copy of that module (a mere 7k file) for 3.7.1 version. It looks like a very stable code (few changes in recent years) and it should work for any Python >= 3.3
the setuptools integration
The magic of setuptools is that the same build subcommand is internally used to build a binary wheel, to install with pip from a source package or to directly install with python setup.py install from a copy (git clone) of the full source package. So I provide a build subclass in setup.py that generates the .mo files with their full path before calling the superclass method. I also use a MANIFEST.in file to list the files that should be copied in a source distribution and a package_data setup argument to list what should go in a binary package or installation folder
run time usage
Provided the mo hierarchy to be installed under a knows package, os.dirname(__file__) called from a module of that package gives its parent folder
Code (assuming the msgfmt.py file is copied under a tools_i18n folder and that po files are under a src folder):
in setup.py
...
sys.path.append(os.path.join(os.path.dirname(__file__), "tools_i18n"))
import msgfmt
from distutils.command.build import build as _build
class Builder(_build):
def run(self):
# po files in src folder are named domain_lang.po
po = re.compile(r"(.*)_(.*).po")
for file in os.listdir("src"):
m = po.match(file)
if m:
# create the LANG/LC_MESSAGES subdir of "locale"
path = os.path.join(self.build_lib, NAME, "locale",
m.group(2), "LC_MESSAGES")
os.makedirs(path, exist_ok=True)
# use msgfmt.py to compile the po file
msgfmt.make(os.path.join("src", file),
os.path.join(path, m.group(1) + ".mo"))
_build.run(self)
setup(
name=NAME,
...
package_data = { "": [..., "locale/*/*/*.mo"]}, # ensure .mo file are copied
cmdclass = {"build": Builder},
)
In MANIFEST.in:
...
include src/*
include tools_i18n/*
To use the translations at run time:
locpath = os.path.dirname(__file__)
lang = locale.getdefaultlocale()[0] # to get platform default language, or whatever...
tr = gettext.translation("argparse", os.path.join(locpath, "locale"),
[lang], fallback=True)
A full project using this method is available at https://github.com/s-ball/i18nparse
Last but not least, after a more in depth reading of the GNU gettext doc, I can say that gettext can process mo files whatever their endianness:
MO files of any endianness can be used on any platform. When a MO file has an endianness other than the platform’s one, the 32-bit numbers from the MO file are swapped at runtime. The performance impact is negligible.

How to get Read the Docs to generate py-modindex.html?

I'm trying to get Read the Docs to generate the py-modindex.html file. Research into this question lead me to the following setup:
setup.py in the project directory has the following contents, which were the minimum needed to get pytest to work and haven't been changed since I got that part of my project working:
import setuptools
setuptools.setup(
name='polygons',
packages=setuptools.find_packages(),
)
docs/requirements.txt contains a single line:
sphinx-autodoc-annotation
The Read the Docs repository URL points to my GitHub repository.
The RtD setting for "Install your project inside a virtualenv using setup.py install" is checked.
The RtD setting for "Requirements file" points to docs/requirements.txt.
The "Module Index" link gets included in index.html, but the py-modindex.html file is missing.
My understanding is that with the virtualenv setting above, RtD will use the setup.py file to install the project so that Sphinx can read the documentation found in the Python docstrings. I'm using function annotations and would like sphyinx-autodoc-annotation to make use of those when creating the built docs files. All of this works splendidly on my local machine when I run make html while in the docs folder. Now I'm trying to get it to work on Read the Docs.
Note: This is an exercise I'm going through to understand everything before I apply it to my real project, which the polygons project is a placeholder for.

reStructuredText: README.rst not working on PyPI

I have created a package on github django-joyride, after publishing it to pypi it is not showing the README.rst properly. I have checked my syntax here on an online viewer and you can see it works fine. What could be the issue?

I had the same problem when uploading my python module to pypi .
Later I checked the README.rst for errors using rst-lint which showed that my readme file was right. You can also use restructuredtext_link package for python to check the rst file for any errors or warnings .
I found that the problem was not in the README file but in setup.py itself.
Follow the below points while writing Readme and setup.py
DO NOT WRITE MULTI LINE python strings for description or summary or anything that goes into the setup( ) arguments .
Don't use relative links in the README file .(like ./path1/path2 ).
Make sure the rst syntax is all right using a checking tool like rst-lint.
If you have a markdown file , you can convert it to Restructured text using pandoc easily.
Also do not use any new docutils syntax since Pypi doesn't use the latest version as Github does.

It could be that pypi gets confused because you have both a README.md and a README.rst file. Try to delete the .md one; github can handle README.rst fine.

How to send a package to PyPi?

i wrote a little module and i would like to know what are the basic steps to package it in order to send it to pypi:
what is the file hierarchy?
how should i name files?
should i use distutils to create PKG-INFO?
where should i include my documentation (made with sphinx)?

I recommend reading The Hitchhiker's Guide to Packaging. Specifically, you should look at the Quick Start section, which describes how to:
Lay out your project
Describe your project
Create your first release
Register your package with the Python Package Index (PyPI)
Upload your release, then grab your towel and save the Universe!
You should also look at the Current State of Packaging in the Introduction to Packaging section, as this helps untangle some of the confusion surrounding setuptools, distutils, distutils2, and distribute.
Update Re: How to Name Files
The excerpt below is from PEP8, which answers your question about how to name files:
Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.
Since module names are mapped to file names, and some file systems are case insensitive and truncate long names, it is important that module names be chosen to be fairly short -- this won't be a problem on Unix, but it may be a problem when the code is transported to older Mac or Windows versions, or DOS.

an example is always the best way to see how to do:
http://packages.python.org/an_example_pypi_project/

Maybe this CheeseShopTutorial is of help for you. From there:
Submitting Packages to the Package
Index
If you have some Python modules or packages that you would like to
share with the Python community, we'd love to have them included in
the Python Package Index! First, if you haven't done so, you will want
to get your project organized. You might follow the guidelines at
ProjectFileAndDirectoryLayout. After that, you'll want to read the
Python documentation regarding creating distributions:
http://docs.python.org/distutils/index.html.
You can also check Writing a Package in Python by Tarek Ziadé from Tarek's book "Expert Python Programming" where questions about development and distribution are addressed in great detail

Matthew Rankin's answer tells you how to organize your project file heirarchy, but I find myself having to look up the commands to execute every time I want to update a project on PyPI. So here they are:
As described on the PyPi site:
Create a setup.py file (here's an example)
Register a username/password with PyPi
Do this:
python setup.py sdist
python setup.py bdist_wheel
python setup.py register
twine upload dist/*
Look for your package on the PyPi site:
https://pypi.python.org/pypi?%3Aaction=index
On another machine try running:
pip install [your package name]

Most important thing is prepare your setup.py properly. Then:
setup.py sdist bdist_wheel to generate distribution archives to dist/ folder
twine upload dist/* to upload the archives to PyPi (with your PyPi username/password)
Here is an example of setup.py:
from setuptools import setup, find_packages
with open('README.md') as readme_file:
README = readme_file.read()
with open('HISTORY.md') as history_file:
HISTORY = history_file.read()
setup_args = dict(
name='elastictools',
version='0.1.2',
description='Useful tools to work with Elastic stack in Python',
long_description_content_type="text/markdown",
long_description=README + '\n\n' + HISTORY,
license='MIT',
packages=find_packages(),
author='Thuc Nguyen',
author_email='gthuc.nguyen#gmail.com',
keywords=['Elastic', 'ElasticSearch', 'Elastic Stack', 'Python 3', 'Elastic 6'],
url='https://github.com/ncthuc/elastictools',
download_url='https://pypi.org/project/elastictools/'
)
install_requires = [
'elasticsearch>=6.0.0,<7.0.0',
'jinja2'
]
if __name__ == '__main__':
setup(**setup_args, install_requires=install_requires)
You can find more detail tutorial here: https://medium.com/#thucnc/how-to-publish-your-own-python-package-to-pypi-4318868210f9

Adding Version Control / Numbering (?) to Python Project

With my Java projects at present, I have full version control by declaring it as a Maven project. However I now have a Python project that I'm about to tag 0.2.0 which has no version control. Therefore should I come accross this code at a later date, I won't no what version it is.
How do I add version control to a Python project, in the same way Maven does it for Java?

First, maven is a build tool and has nothing to do with version control. You don't need a build tool with Python -- there's nothing to "build".
Some folks like to create .egg files for distribution. It's as close to a "build" as you get with Python. This is a simple setup.py file.
You can use SVN keyword replacement in your source like this. Remember to enable keyword replacement for the modules that will have this.
__version__ = "$Revision$"
That will assure that the version or revision strings are forced into your source by SVN.
You should also include version keywords in your setup.py file.

Create a distutils setup.py file. This is the Python equivalent to maven pom.xml, it looks something like this:
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
If you want dependency management like maven, take a look at setuptools.

Ants's answer is correct, but I would like to add that your modules can define a __version__ variable, according to PEP 8, which can be populated manually or via Subversion or CVS, e.g. if you have a module thingy, with a file thingy/__init__.py:
___version___ = '0.2.0'
You can then import this version in setup.py:
from distutils.core import setup
import thingy
setup(name='thingy',
version=thingy.__version__,
py_modules=['thingy'],
)