I have a project hosted on GitHub. For this I have written my README using the Markdown syntax in order to have it nicely formatted on GitHub.
As my project is in Python I also plan to upload it to PyPi. The syntax used for READMEs on PyPi is reStructuredText.
I would like to avoid having to handle two READMEs containing roughly the same content; so I searched for a markdown to RST (or the other way around) translator, but couldn't find any.
The other solution I see is to perform a markdown/HTML and then a HTML/RST translation. I found some ressources for this here and here so I guess it should be possible.
Would you have any idea that could fit better with what I want to do?
I would recommend Pandoc, the "swiss-army knife for converting files from one markup format into another" (check out the diagram of supported conversions at the bottom of the page, it is quite impressive). Pandoc allows markdown to reStructuredText translation directly. There is also an online editor here which lets you try it out, so you could simply use the online editor to convert your README files.
As #Chris suggested, you can use Pandoc to convert Markdown to RST. This can be simply automated using pypandoc module and some magic in setup.py:
from setuptools import setup
try:
from pypandoc import convert
read_md = lambda f: convert(f, 'rst')
except ImportError:
print("warning: pypandoc module not found, could not convert Markdown to RST")
read_md = lambda f: open(f, 'r').read()
setup(
# name, version, ...
long_description=read_md('README.md'),
install_requires=[]
)
This will automatically convert README.md to RST for the long description using on PyPi. When pypandoc is not available, then it just reads README.md without the conversion – to not force others to install pypandoc when they wanna just build the module, not upload to PyPi.
So you can write in Markdown as usual and don’t care about RST mess anymore. ;)
2019 Update
The PyPI Warehouse now supports rendering Markdown as well! You just need to update your package configuration and add the long_description_content_type='text/markdown' to it. e.g.:
setup(
name='an_example_package',
# other arguments omitted
long_description=long_description,
long_description_content_type='text/markdown'
)
Therefore, there is no need to keep the README in two formats any longer.
You can find more information about it in the documentation.
Old answer:
The Markup library used by GitHub supports reStructuredText. This means you can write a README.rst file.
They even support syntax specific color highlighting using the code and code-block directives (Example)
PyPI now supports Markdown for long descriptions!
In setup.py, set long_description to a Markdown string, add long_description_content_type="text/markdown" and make sure you're using recent tooling (setuptools 38.6.0+, twine 1.11+).
See Dustin Ingram's blog post for more details.
You might also be interested in the fact that it is possible to write in a common subset so that your document comes out the same way when rendered as markdown or rendered as reStructuredText: https://gist.github.com/dupuy/1855764 ☺
For my requirements I didn't want to install Pandoc in my computer. I used docverter. Docverter is a document conversion server with an HTTP interface using Pandoc for this.
import requests
r = requests.post(url='http://c.docverter.com/convert',
data={'to':'rst','from':'markdown'},
files={'input_files[]':open('README.md','rb')})
if r.ok:
print r.content
I ran into this problem and solved it with the two following bash scripts.
Note that I have LaTeX bundled into my Markdown.
#!/usr/bin/env bash
if [ $# -lt 1 ]; then
echo "$0 file.md"
exit;
fi
filename=$(basename "$1")
extension="${filename##*.}"
filename="${filename%.*}"
if [ "$extension" = "md" ]; then
rst=".rst"
pandoc $1 -o $filename$rst
fi
Its also useful to convert to html. md2html:
#!/usr/bin/env bash
if [ $# -lt 1 ]; then
echo "$0 file.md <style.css>"
exit;
fi
filename=$(basename "$1")
extension="${filename##*.}"
filename="${filename%.*}"
if [ "$extension" = "md" ]; then
html=".html"
if [ -z $2 ]; then
# if no css
pandoc -s -S --mathjax --highlight-style pygments $1 -o $filename$html
else
pandoc -s -S --mathjax --highlight-style pygments -c $2 $1 -o $filename$html
fi
fi
I hope that helps
Using the pandoc tool suggested by others I created a md2rst utility to create the rst files. Even though this solution means you have both an md and an rst it seemed to be the least invasive and would allow for whatever future markdown support is added. I prefer it over altering setup.py and maybe you would as well:
#!/usr/bin/env python
'''
Recursively and destructively creates a .rst file for all Markdown
files in the target directory and below.
Created to deal with PyPa without changing anything in setup based on
the idea that getting proper Markdown support later is worth waiting
for rather than forcing a pandoc dependency in sample packages and such.
Vote for
(https://bitbucket.org/pypa/pypi/issue/148/support-markdown-for-readmes)
'''
import sys, os, re
markdown_sufs = ('.md','.markdown','.mkd')
markdown_regx = '\.(md|markdown|mkd)$'
target = '.'
if len(sys.argv) >= 2: target = sys.argv[1]
md_files = []
for root, dirnames, filenames in os.walk(target):
for name in filenames:
if name.endswith(markdown_sufs):
md_files.append(os.path.join(root, name))
for md in md_files:
bare = re.sub(markdown_regx,'',md)
cmd='pandoc --from=markdown --to=rst "{}" -o "{}.rst"'
print(cmd.format(md,bare))
os.system(cmd.format(md,bare))
Related
I would like to offer my JupyterLab notebooks in several languages (de-de, en-us).
In order to do so, I marked some strings
import gettext
domain = 'my_application_name'
localedir = '.'
translate = gettext.translation(domain, localedir, fallback=True)
_ = translate.gettext # using _ as name for the translation function is kind of standard in python
# do not confuse with private markers or underscore library
# https://github.com/serkanyersen/underscore.py
print(_('Hello World'))
print(_('another translation key'))
Then I downloaded xgettext.exe for windows from
https://github.com/mlocati/gettext-iconv-windows/releases/download/v0.21-v1.16/gettext0.21-iconv1.16-static-64.zip
and tried to extract the strings with following console command:
xgettext.exe my_notebook.ipynb
I got the warning
xgettext.exe: warning: file 'my_notebook.ipynb' extension 'ipynb' is unknown; will try C
and no output file was generated.
=> What is the recommented way for extracting translatable strings from JupyterLab notebooks?
I would prefer a solution, where no extra binary (like xgettext.exe) would be required on windows.
Does JupyterLab itself provide some translation features/extension (not for the UI but for notebooks)?
As a possible workaround, the notebook could first be converted to a python file with nbconvert and then passed to xgettext.exe. However, that seems to be too complicated. There should be a more elegant solution.
(The extraction of translatable strings from python files does work on windows, e.g.
xgettext.exe my_python_file.py
)
The rough workflow for translations seems to be:
Mark all strings that should be translated
Generate a translation file template from the strings (="master list" or Portable Object Template (POT) file)
Translate the translation file template
Apply the translation file
Related:
https://github.com/jupyterlab/jupyterlab/issues/11753
https://docs.python.org/3/library/i18n.html
https://www.mattlayman.com/blog/2015/i18n/
Here is the already mentioned workaround based on nbconvert and xgettext:
jupyter nbconvert --to script my_notebook.ipynb --output z_temp_file_for_translation
xgettext.exe z_temp_file_for_translation.py -o translations.pot
del z_temp_file_for_translation.py
As an alternative to xgettext, the python package Babel can be used:
pip install Babel
Also see: http://babel.pocoo.org/en/latest/cmdline.html#extract
jupyter nbconvert --to script my_notebook.ipynb --output z_temp_file_for_translation
pybabel extract z_temp_file_for_translation.py -o translations.pot
del z_temp_file_for_translation.py
I am trying to create a python package (deb & rpm) from cmake, ideally using cpack. I did read
https://cmake.org/cmake/help/latest/cpack_gen/rpm.html and,
https://cmake.org/cmake/help/latest/cpack_gen/deb.html
The installation works just fine (using component install) for my shared library. However I cannot make sense of the documentation to install the python binding (glue) code. Using the standard cmake install mechanism, I tried:
install(
FILES __init__.py library.py
DESTINATION ${ACME_PYTHON_PACKAGE_DIR}/project_name
COMPONENT python)
And then using brute-force approach ended-up with:
# debian based package (relative path)
set(ACME_PYTHON_PACKAGE_DIR lib/python3/dist-packages)
and
# rpm based package (full path required)
set(ACME_PYTHON_PACKAGE_DIR /var/lang/lib/python3.8/site-packages)
The above is derived from:
debian % python -c 'import site; print(site.getsitepackages())'
['/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.9/dist-packages']
while:
rpm % python -c 'import site; print(site.getsitepackages())'
['/var/lang/lib/python3.8/site-packages']
It is pretty clear that the brute-force approach will not be portable, and is doomed to fail on the next release of python. The only possible solution that I can think of is generating a temporary setup.py python script (using setuptools), that will do the install. Typically cmake would call the following process:
% python setup.py install --root ${ACME_PYTHON_INSTALL_ROOT}
My questions are:
Did I understand the cmake/cpack documentation correctly for python package ? If so this means I need to generate an intermediate setup.py script.
I have been searching through the cmake/cpack codebase (git grep setuptools) but did not find helper functions to handle generation of setup.py and passing the result files back to cpack. Is there an existing cmake module which I could re-use ?
I did read, some alternative solution, such as:
How to build debian package with CPack to execute setup.py?
Which seems overly complex, and geared toward Debian-only based system. I need to handle RPM in my case.
As mentionned in my other solution, the ugly part is dealing with absolute path in cmake install() commands. I was able to refactor the code to avoid usage of absolute path in install(). I simply changed the installation into:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
# "." syntax is a reliable mechanism, see:
# https://gitlab.kitware.com/cmake/cmake/-/issues/22616
DESTINATION "."
COMPONENT python)
And then one simply needs to:
set(CMAKE_INSTALL_PREFIX "/")
set(CPACK_PACKAGING_INSTALL_PREFIX "/")
include(CPack)
At this point all install path now need to include explicitely /usr since we've cleared the value for CMAKE_INSTALL_PREFIX.
The above has been tested for deb and rpm packages. CPACK_BINARY_TGZ does properly run with the above solution:
https://gitlab.kitware.com/cmake/cmake/-/issues/22925
I am going to post the temporary solution I am using at the moment, until someone provide something more robust.
So I eventually manage to stumble upon:
https://alioth-lists.debian.net/pipermail/libkdtree-devel/2012-October/000366.html and,
Using CMake with setup.py
Re-using the above to do an install step instead of a build step can be done as follow:
find_package(Python COMPONENTS Interpreter)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(SETUP_DEPS "${CMAKE_CURRENT_SOURCE_DIR}/project_name/__init__.py")
set(SETUP_OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build-python")
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
COMMAND ${Python_EXECUTABLE} ARGS ${SETUP_PY} install --root ${SETUP_OUTPUT}
COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
DEPENDS ${SETUP_DEPS})
add_custom_target(target ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp)
And then the ugly part is:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
DESTINATION "/" # FIXME may cause issues with other cpack generators
COMPONENT python)
Turns out that the documentation for install() is pretty clear about absolute paths:
https://cmake.org/cmake/help/latest/command/install.html#introduction
DESTINATION
[...]
As absolute paths are not supported by cpack installer generators,
it is preferable to use relative paths throughout.
For reference, here is my setup.py.in:
from setuptools import setup
if __name__ == '__main__':
setup(name='project_name_python',
version='${PROJECT_VERSION}',
package_dir={'': '${CMAKE_CURRENT_SOURCE_DIR}'},
packages=['project_name'])
You can be fancy and remove the __pycache__ folder using the -B flag:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT}
You can be extra fancy and add debian option such as:
if(CPACK_BINARY_DEB)
set(EXTRA_ARG "--install-layout" "deb")
endif()
use as:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT} ${EXTRA_ARG}
I'm trying to create a HTML report using pyreport and it works up to the single point, that the LaTeX formulae are not generated.
Here is the input file I use for testing:
#$ This is \LaTeX : $c = 2\cdot(a+b)$
Than I run pyreport -l -t html --verbose file.py, but the report that I get is empty. When I add other comments to the input file, or some Python code, than it is displayed properly within the report. Here is the output from pyreport:
Running python script
/tmp/file.py:
Outputing report to
/tmp/file.html Ran script in 0.13s
I'm using Ubuntu and I have the texlive package installed. Why isn't the formula added to the report?
I think i have find the problem.
The problem is the RST tools to convert in html.
In pyreport, when you choose the math mode, the program will do the sentence in a bock .. raw:: LaTeX
But in the new version of rst2html, this command doesnt work, it's replace by:
.. math::
If you use the command:
pyreport -l -e -t rst --verbose file.py
and after
rst2html file.rst > test.html
You will see the problem.
You can change that in pyreport code, in main.py of pyreport. (use a locate to find it). And replace the
.. raw:: Latex
, by
.. math::
The last problem is for the command \LaTeX, that's not in a math mode of latex. So it's not work.
You can report to RST documentation http://docutils.sourceforge.net/docs/ref/rst/directives.html#raw
In my travis setup I want to apply a patch to a very specific python file that is broken on some configurations. The installation procedure looks like this:
install:
# test with various Django versions
- pip install Django==$DJANGO
# patch Django 1.4 libgeos
- 'if [ $DJANGO == "1.4.13" ]; then find . -name libgeos.py -exec patch {} travis/geos-dev.patch \; ; fi'
But apparantly, find is not able to retrieve the file in question.
How can I realiably find this file, also across different python versions?
Seems like a simple find under /home/travis/virtualenv/ does the job.
For example, I have a Python script using the Google App Engine SDK:
from google.appengine.ext import db
from google.appengine.ext import webapp
from google.appengine.ext.webapp import template
from google.appengine.ext.webapp.util import run_wsgi_app
The module db has a submodule Key, so I try to use autocomplete on it:
db.KTab
But at the bottom of the Vim window, I get the following:
-- Omni completion (^O^N^P) Pattern not found
How do I include the path to non-standard Python libraries so that Vim autocompletion can find them? And also display their docstrings?
You need to add your library files to your tags file. For instance, if you have installed the Google App Engine via pip in a virtual environment located in env/:
virtualenv --no-site-package env/
source env/bin/activate
pip install google_appengine
... then you should execute:
ctags -R --python-kinds=-i -o tags env/
If you did not install google_appengine through pip, then you should locate the path to your python libraries (hint: it should be indicated by $PYTHONPATH. And according to this reference page: "on Unix, this is usually .:/usr/local/lib/python.") and replace env/ by the path you found.
Finally, your .vimrc file should parse your tags file. For instance, in my .vimrc, I have:
set tags+=/path/to/my/tags
I grabbed this from natw's vimrc (I think...maybe sontek), but it should do the trick, so long as your packages are findable by your current install of Python. This lets you use gf, but also sets up searching these files for autocompletion. Note the py <<EOF part, which starts a section interpreted in Python. This means you'd have to have the python interpreter installed in vim to use it.
function! LoadPythonPath()
py <<EOF
# load PYTHONPATH into vim, this lets you hover over a module name
# and type 'gf' (for goto file) and open that file in vim. Useful
# and easier than rope for simple tasks
import os.path
import sys
import vim
for p in sys.path:
if os.path.isdir(p):
vim.command(r"set path+=%s" % (p.replace(" ", r"\ ")))
EOF
endfunction
Btw, I don't like to have this load automatically, so I set it to a function that intelligently loads/unloads when I call it/first enter a Python doc. And I add a let g:PythonPathLoaded=1 to the previous function.
function! GetPythonPath()
if !exists("g:PythonPathLoaded")
call LoadPythonPath()
return
elseif g:PythonPathLoaded
return
else
call LoadPythonPath()
endif
endfunction
And I have an unload function too...though I'm not sure whether this makes a huge difference.
function! UnloadPythonPath()
py <<EOF
# load PYTHONPATH into vim, this lets you hover over a module name
# and type 'gf' (for goto file) and open that file in vim. Useful
# and easier than rope for simple tasks
for p in sys.path:
if os.path.isdir(p):
vim.command(r"set path-=%s" % (p.replace(" ", r"\ ")))
EOF
let g:PythonPathLoaded = 0
endfunction
Hope this helps! Plus, an added bonus is that this will load your packages regardless of whether you are using virtualenv (since it, I believe, runs whatever is set as 'python' at the moment).