We Keep Coding

Is there a portable way to provide localization of a package distributed on PyPI?

Context:
This is kind of a followup to another question of mine.
I would like to provide localized versions of a package. Following the Python documentation, I have extracted a .pot file with pygettext, prepared a translation in a .po file, compiled it in a .mo file.
Everything is fine till there, and my package displays the translated messages.
But my final goal would be to make it available on PyPI. So I have done some research and found:
setuptools documentation: not even one single word about localization...
The Format of GNU MO Files
It explains that the format depends on the endianness of the platform where the file was generated. My understanding is that only the po files are portable...
What is the correct way to include localisation in python packages?
The answer is fully relevant and speaks of the setuptools/babel integration but:
that integration only allows to build mo file and does not speak of their distribution
author describes how they use it, no references for portability across systems
Babel: compile translation files when calling setup.py install
An interesting way, even if it requires the babel module on the target platform. Not so heavy but way heavier than my own package... In fact, the distributions contain only po files and they are compiled with babel at install time.
Question:
Is there a way to build platform specific wheels containing compiled mo files?
If not I will have to require babel on target and try to find my way through mo compilation at install time.

EDIT 7/12/2018:
After some work, I could build a specific package based on what is below in this answer. It can be used from other projects to automatically compile po files at build time through the magic of setuptools enty_points. It is now available on GitHUB (https://github.com/s-ball/mo_installer) and distributed on PyPI (https://pypi.org/project/mo_installer)
The researches that I did before asking the question gave me enough hints to reach a possible solution.
I can now say, thay is is possible to include a platform specific mo file in a wheel - unfortunately in my current solution the wheel gives no indication that it is platform specific. But the same solution allows to build a source distribution that build the mo file on the target platform.
Now for the details:
the tools needed to compile a mo file on the target:
Most solutions picked from Google or SO rely either on Babel or on the GNU gettext msgfmt program. But cPython tools include a pure Python module msgfmt.py that is enough here. Unfortunately, this tool is often not installed by default in many Linux/Unix-like. My solution just includes a copy of that module (a mere 7k file) for 3.7.1 version. It looks like a very stable code (few changes in recent years) and it should work for any Python >= 3.3
the setuptools integration
The magic of setuptools is that the same build subcommand is internally used to build a binary wheel, to install with pip from a source package or to directly install with python setup.py install from a copy (git clone) of the full source package. So I provide a build subclass in setup.py that generates the .mo files with their full path before calling the superclass method. I also use a MANIFEST.in file to list the files that should be copied in a source distribution and a package_data setup argument to list what should go in a binary package or installation folder
run time usage
Provided the mo hierarchy to be installed under a knows package, os.dirname(__file__) called from a module of that package gives its parent folder
Code (assuming the msgfmt.py file is copied under a tools_i18n folder and that po files are under a src folder):
in setup.py
...
sys.path.append(os.path.join(os.path.dirname(__file__), "tools_i18n"))
import msgfmt
from distutils.command.build import build as _build
class Builder(_build):
def run(self):
# po files in src folder are named domain_lang.po
po = re.compile(r"(.*)_(.*).po")
for file in os.listdir("src"):
m = po.match(file)
if m:
# create the LANG/LC_MESSAGES subdir of "locale"
path = os.path.join(self.build_lib, NAME, "locale",
m.group(2), "LC_MESSAGES")
os.makedirs(path, exist_ok=True)
# use msgfmt.py to compile the po file
msgfmt.make(os.path.join("src", file),
os.path.join(path, m.group(1) + ".mo"))
_build.run(self)
setup(
name=NAME,
...
package_data = { "": [..., "locale/*/*/*.mo"]}, # ensure .mo file are copied
cmdclass = {"build": Builder},
)
In MANIFEST.in:
...
include src/*
include tools_i18n/*
To use the translations at run time:
locpath = os.path.dirname(__file__)
lang = locale.getdefaultlocale()[0] # to get platform default language, or whatever...
tr = gettext.translation("argparse", os.path.join(locpath, "locale"),
[lang], fallback=True)
A full project using this method is available at https://github.com/s-ball/i18nparse
Last but not least, after a more in depth reading of the GNU gettext doc, I can say that gettext can process mo files whatever their endianness:
MO files of any endianness can be used on any platform. When a MO file has an endianness other than the platform’s one, the 32-bit numbers from the MO file are swapped at runtime. The performance impact is negligible.

How to create a basic setup.py script when one of the modules is named parser?

It seems from here and here, that setuptools does not work properly when a project has a module named parser in it. I am trying to write a basic setup script for this project: https://github.com/karlmoritz/bolinas . This is not my repository and I want to avoid making any changes to this code (or if it is not possible to make no changes, then only making minimal changes).
As you can see, there is a package called parser that conflicts with the default python installation. This is my setup.py:
from setuptools import setup, find_packages
setup(
name = "Bolinas",
version = "0.1",
packages = find_packages(),
scripts = ['bolinas.py', 'config.py'],
)
When I run it, I get the same errors as in the links I provided above.
Is there anything I can do to make this work without renaming the module?

These guys really have top level packages called "common", "config" and "parser"? No sane person would install this (or would later rue the day). Its not a problem with setup.py at all. Its a problem with a package that overrides standard system modules like "parser".
As stands, bolinas is not architected to be installable. To do so, its modules should all be moved under a package called 'bolinas' so that the base level namespace isn't littered with a bunch of vaguely named modules.

What is the cleanest way to add a directory of third-party packages to the beginning of the Python path?

My context is appengine_config.py, but this is really a general Python question.
Given that we've cloned a repo of an app that has an empty directory lib in it, and that we populate lib with packages by using the command pip install -r requirements.txt --target lib, then:
dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)
For importing purposes, we can add such a filesystem path to the beginning of the Python path in the following way (we use index 1 because the first position should remain '.', the current directory):
sys.path.insert(1, dirpath)
However, that won't work if any of the packages in that directory are namespace packages.
To support namespace packages we can instead use:
site.addsitedir(dirpath)
But that appends the new directory to the end of the path, which we don't want in case we need to override a platform-supplied package (such as WebOb) with a newer version.
The solution I have so far is this bit of code which I'd really like to simplify:
sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)
Is there a cleaner or more Pythonic way of accomplishing this?

For this answer I assume you know how to use setuptools and setup.py.
Assuming you would like to use the standard setuptools workflow for development, I recommend using this code snipped in your appengine_config.py:
import os
import sys
if os.environ.get('CURRENT_VERSION_ID') == 'testbed-version':
# If we are unittesting, fake the non-existence of appengine_config.
# The error message of the import error is handled by gae and must
# exactly match the proper string.
raise ImportError('No module named appengine_config')
# Imports are done relative because Google app engine prohibits
# absolute imports.
lib_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'libs')
# Add every library to sys.path.
if os.path.isdir(lib_dir):
for lib in os.listdir(lib_dir):
if lib.endswith('.egg'):
lib = os.path.join(lib_dir, lib)
# Insert to override default libraries such as webob 1.1.1.
sys.path.insert(0, lib)
And this piece of code in setup.cfg:
[develop]
install-dir = libs
always-copy = true
If you type python setup.py develop, the libraries are downloaded as eggs in the libs directory. appengine_config inserts them to your path.
We use this at work to include webob==1.3.1 and internal packages which are all namespaced using our company namespace.

You may want to have a look at the answers in the Stack Overflow thread, "How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)," but for your particular predicament with namespace packages, you're running up against a long-standing issue I filed against site.addsitedir's behavior of appending to sys.path instead of inserting after the first element. Please feel free to add to that discussion with a link to this use case.
I do want to address something else that you said that I think is misleading:
My context is appengine_config.py, but this is really a general Python
question.
The question actually arises from the limitations of Google App Engine and the inability to install third-party packages, and hence, seeking a workaround. Rather than manually adjusting sys.path and using site.addsitedir. In general Python development, if your code uses these, you're Doing It Wrong.
The Python Packaging Authority (PyPA) describes the best practices to put third party libraries on your path, which I outline below:
Create a virtualenv
Mark out your dependencies in your setup.py and/or requirements files (see PyPA's "Concepts and Analyses")
Install your dependencies into the virtualenv with pip
Install your project, itself, into the virtualenv with pip and the -e/--editable flag.
Unfortunately, Google App Engine is incompatible with virtualenv and with pip. GAE chose to block this toolset in an attempt sandbox the environment. Hence, one must use hacks to work around the limitations of GAE to use additional or newer third party libraries.
If you dislike this limitation and want to use standard Python tooling for managing third-party package dependencies, other Platform as a Service providers out there eagerly await your business.

setup.py's sdist, bdist and install behaving differently regarding data_files

I'm trying to distribute web assets along with a web app that I'm trying to package, but I'm failing miserably. I don't understand why I have a different list of files installed or packages when I run bdist, sdist, or install.
Project Layout
The project runs with python 3 on Arch. The results are the same with Py3 on Raspbian.
I've done a very trimmed down version to make things simpler, which I describe here.
The files layout is as follow :
data/index.html
MANIFEST.in
mylib.py
setup.py
The MANIFEST.in file is :
recursive-include data *
The setup.py is :
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(name='mylib',
version='0.1.2',
url='http://www.example.org',
author='Foo',
packages=find_packages(),
data_files = [ ('share/mylib', ['data/index.html']) ]
)
My goal is to install index.html in PREFIX/share/mylib/index.html.
Running setup.py
Now, bdist includes the file at the seemingly right location, while sdist and install just ignore it :
bdist
Using bdist, I have the following files in the resulting tar :
./usr/lib/python3.3/site-packages/mylib-0.1.2-py3.3.egg-info/SOURCES.txt
./usr/lib/python3.3/site-packages/mylib-0.1.2-py3.3.egg-info/top_level.txt
./usr/lib/python3.3/site-packages/mylib-0.1.2-py3.3.egg-info/dependency_links.txt
./usr/lib/python3.3/site-packages/mylib-0.1.2-py3.3.egg-info/PKG-INFO
./usr/share/mylib/index.html
This is exactly what I want to be installed, perfect. However, I really want sdist and install to work, since I want to distribute this thing on PyPI and be able to install from source checkouts.
sdist
When I untar the sdist file, everything seems ok and data is included :
...
mylib-0.1.2/data/
mylib-0.1.2/data/index.html
...
However, if I sudo python setup.py install --record=log.txt in the directory where it is untarred, the only file listed in the log is /usr/lib/python3.3/site-packages/mylib-0.1.2-py3.3.egg. No trace of data/index.html anywhere ('/usr/local/share', '/usr/share')
install
Same issue as sdist (I suppose this is expected). No trace of data/index.html anywhere ('/usr/local/share', '/usr/share').
I also tried to add a setup.cfg like this :
[install]
install-data=/usr/local/share/mylib/
install_data=/usr/local/share/mylib/
(I've added both install-data and install_data since docs state to use the later, while I saw other projects using the former). No luck.
Epilogue
Now, I quite new to python and it's environment, I'm probably missing something obvious or misunderstanding how setuptools works. I've been reading the doc back an forth, reading stackoverflow's Q&A in data_files at great length, but didn't make any progress.
If someone could point me to the right direction to solve this, this would be great. A link to a simple project distributing assets would be a good read too. I just couldn't find one that gave me that "Ah ah!" moment.
Thanks for reading.

I don't know whether this helps, as I always include my data files relative to the python packages they go with. Additionally to the MANIFFEST.in, you'd have a package_data key in setup.py:
setup(name='mylib',
version='0.1.2',
url='http://www.example.org',
author='Foo',
packages=find_packages(),
package_data={'package_name': 'package_dir/data/*'}
)
this would put the data to site-packages/mylib-0.1.2/data

Adding Version Control / Numbering (?) to Python Project

With my Java projects at present, I have full version control by declaring it as a Maven project. However I now have a Python project that I'm about to tag 0.2.0 which has no version control. Therefore should I come accross this code at a later date, I won't no what version it is.
How do I add version control to a Python project, in the same way Maven does it for Java?

First, maven is a build tool and has nothing to do with version control. You don't need a build tool with Python -- there's nothing to "build".
Some folks like to create .egg files for distribution. It's as close to a "build" as you get with Python. This is a simple setup.py file.
You can use SVN keyword replacement in your source like this. Remember to enable keyword replacement for the modules that will have this.
__version__ = "$Revision$"
That will assure that the version or revision strings are forced into your source by SVN.
You should also include version keywords in your setup.py file.

Create a distutils setup.py file. This is the Python equivalent to maven pom.xml, it looks something like this:
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
If you want dependency management like maven, take a look at setuptools.

Ants's answer is correct, but I would like to add that your modules can define a __version__ variable, according to PEP 8, which can be populated manually or via Subversion or CVS, e.g. if you have a module thingy, with a file thingy/__init__.py:
___version___ = '0.2.0'
You can then import this version in setup.py:
from distutils.core import setup
import thingy
setup(name='thingy',
version=thingy.__version__,
py_modules=['thingy'],
)