In my travis setup I want to apply a patch to a very specific python file that is broken on some configurations. The installation procedure looks like this:
install:
# test with various Django versions
- pip install Django==$DJANGO
# patch Django 1.4 libgeos
- 'if [ $DJANGO == "1.4.13" ]; then find . -name libgeos.py -exec patch {} travis/geos-dev.patch \; ; fi'
But apparantly, find is not able to retrieve the file in question.
How can I realiably find this file, also across different python versions?
Seems like a simple find under /home/travis/virtualenv/ does the job.
Related
I want my users to be able to reference a file in my python package (specifically a docker-compose.yml file) directly from the shell.
I couldnt find a way to get only the location from pip show (and grep-ing out "location" from its output feels ugly), so my current (somewhat verbose) solution is:
docker compose -f $(python3 -c "import locust_plugins; print(locust_plugins.__path__[0])")/timescale/docker-compose.yml up
Is there a better way?
Edit: I solved it by installing a wrapper command I call locust-compose as part of the package. Not perfect, but it gets the job done:
#!/bin/bash
module_location=$(python3 -c "import locust_plugins; print(locust_plugins.__path__[0])")
set -x
docker compose -f $module_location/timescale/docker-compose.yml "$#"
Most of the support you need for this is in the core setuptools suite.
First of all, you need to make sure the data file is included in your package. In a setup.cfg file you can write:
[options.package_data]
timescale = docker-compose.yml
Now if you pip install . or pip wheel, that will include the Compose file as part of the Python package.
Next, you can retrieve this in Python code using the ResourceManager API:
#!/usr/bin/env python3
# timescale/compose_path.py
import pkg_resources
if __name__ == '__main__':
print(pkg_resources.resource_filename('timescale', 'docker-compose.yml'))
And finally, you can take that script and make it a setuptools entry point script (as distinct from the similarly-named Docker concept), so that you can just run it as a single command.
[options.entry_points]
console_scripts=
timescale_compose_path = timescale:compose_path
Again, if you pip install . into a virtual environment, you should be able to run timescale_compose_path and get the path name out.
Having done all of those steps, you can finally run a simpler
docker-compose -f $(timescale_compose_path) up
I am trying to create a python package (deb & rpm) from cmake, ideally using cpack. I did read
https://cmake.org/cmake/help/latest/cpack_gen/rpm.html and,
https://cmake.org/cmake/help/latest/cpack_gen/deb.html
The installation works just fine (using component install) for my shared library. However I cannot make sense of the documentation to install the python binding (glue) code. Using the standard cmake install mechanism, I tried:
install(
FILES __init__.py library.py
DESTINATION ${ACME_PYTHON_PACKAGE_DIR}/project_name
COMPONENT python)
And then using brute-force approach ended-up with:
# debian based package (relative path)
set(ACME_PYTHON_PACKAGE_DIR lib/python3/dist-packages)
and
# rpm based package (full path required)
set(ACME_PYTHON_PACKAGE_DIR /var/lang/lib/python3.8/site-packages)
The above is derived from:
debian % python -c 'import site; print(site.getsitepackages())'
['/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.9/dist-packages']
while:
rpm % python -c 'import site; print(site.getsitepackages())'
['/var/lang/lib/python3.8/site-packages']
It is pretty clear that the brute-force approach will not be portable, and is doomed to fail on the next release of python. The only possible solution that I can think of is generating a temporary setup.py python script (using setuptools), that will do the install. Typically cmake would call the following process:
% python setup.py install --root ${ACME_PYTHON_INSTALL_ROOT}
My questions are:
Did I understand the cmake/cpack documentation correctly for python package ? If so this means I need to generate an intermediate setup.py script.
I have been searching through the cmake/cpack codebase (git grep setuptools) but did not find helper functions to handle generation of setup.py and passing the result files back to cpack. Is there an existing cmake module which I could re-use ?
I did read, some alternative solution, such as:
How to build debian package with CPack to execute setup.py?
Which seems overly complex, and geared toward Debian-only based system. I need to handle RPM in my case.
As mentionned in my other solution, the ugly part is dealing with absolute path in cmake install() commands. I was able to refactor the code to avoid usage of absolute path in install(). I simply changed the installation into:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
# "." syntax is a reliable mechanism, see:
# https://gitlab.kitware.com/cmake/cmake/-/issues/22616
DESTINATION "."
COMPONENT python)
And then one simply needs to:
set(CMAKE_INSTALL_PREFIX "/")
set(CPACK_PACKAGING_INSTALL_PREFIX "/")
include(CPack)
At this point all install path now need to include explicitely /usr since we've cleared the value for CMAKE_INSTALL_PREFIX.
The above has been tested for deb and rpm packages. CPACK_BINARY_TGZ does properly run with the above solution:
https://gitlab.kitware.com/cmake/cmake/-/issues/22925
I am going to post the temporary solution I am using at the moment, until someone provide something more robust.
So I eventually manage to stumble upon:
https://alioth-lists.debian.net/pipermail/libkdtree-devel/2012-October/000366.html and,
Using CMake with setup.py
Re-using the above to do an install step instead of a build step can be done as follow:
find_package(Python COMPONENTS Interpreter)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(SETUP_DEPS "${CMAKE_CURRENT_SOURCE_DIR}/project_name/__init__.py")
set(SETUP_OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build-python")
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
COMMAND ${Python_EXECUTABLE} ARGS ${SETUP_PY} install --root ${SETUP_OUTPUT}
COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
DEPENDS ${SETUP_DEPS})
add_custom_target(target ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp)
And then the ugly part is:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
DESTINATION "/" # FIXME may cause issues with other cpack generators
COMPONENT python)
Turns out that the documentation for install() is pretty clear about absolute paths:
https://cmake.org/cmake/help/latest/command/install.html#introduction
DESTINATION
[...]
As absolute paths are not supported by cpack installer generators,
it is preferable to use relative paths throughout.
For reference, here is my setup.py.in:
from setuptools import setup
if __name__ == '__main__':
setup(name='project_name_python',
version='${PROJECT_VERSION}',
package_dir={'': '${CMAKE_CURRENT_SOURCE_DIR}'},
packages=['project_name'])
You can be fancy and remove the __pycache__ folder using the -B flag:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT}
You can be extra fancy and add debian option such as:
if(CPACK_BINARY_DEB)
set(EXTRA_ARG "--install-layout" "deb")
endif()
use as:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT} ${EXTRA_ARG}
I am trying to run statsd/graphite which uses django 1.6.
While accessing graphite URL, I get django module error
File "/opt/graphite/webapp/graphite/urls.py", line 15, in
from django.conf.urls.defaults import *
ImportError: No module named defaults
However, I do not find defaults django package inside /Library/Python/2.7/site-packages/django/conf/urls/
Please help fixing this issue.
django.conf.urls.defaults has been removed in Django 1.6. If the problem was in your own code, you would fix it by changing the import to
from django.conf.urls import patterns, url, include
However, in your case the problem is in a third party app, graphite. The issue has been fixed in graphite's master branch and version 0.9.14+.
In Django 1.8+ you can remove patterns from the import, and use a list of url()s instead.
from django.conf.urls import url, include
If for some reason you don't want to downgrade to Django 1.5.x or upgrade Graphite then you can apply the fix to your older Graphite with:
find ./ -type f -exec sed -i -e 's/from\ django\.conf\.urls\.defaults\ import\ \*/from\ django\.conf\.urls\ import\ \*/g' {} \;
..in your <graphite_dir>/webapp/graphite dir.
This helped me with my Graphite 0.9.12 and Django 1.7(.5).
(I also had to do:
find ./ -type f -exec sed -i -e 's/mimetype\=/content_type\=/g' {} \;
find ./ -type f -exec sed -i -e 's/content_type\=mimetype/content_type\=content_type/g' {} \;
..later on as after I managed to start Graphite some of its features didn't work. Now they work for me but YMMV.)
go to the file location where you have installed python.
open cmd on that path and then install django with command >> pip install django
Then cross check on python shell with import django(which should do nothing)
or simply use the command >> python -m django --version
it will simply give you version
enter image description here
enter image description here
I have a project hosted on GitHub. For this I have written my README using the Markdown syntax in order to have it nicely formatted on GitHub.
As my project is in Python I also plan to upload it to PyPi. The syntax used for READMEs on PyPi is reStructuredText.
I would like to avoid having to handle two READMEs containing roughly the same content; so I searched for a markdown to RST (or the other way around) translator, but couldn't find any.
The other solution I see is to perform a markdown/HTML and then a HTML/RST translation. I found some ressources for this here and here so I guess it should be possible.
Would you have any idea that could fit better with what I want to do?
I would recommend Pandoc, the "swiss-army knife for converting files from one markup format into another" (check out the diagram of supported conversions at the bottom of the page, it is quite impressive). Pandoc allows markdown to reStructuredText translation directly. There is also an online editor here which lets you try it out, so you could simply use the online editor to convert your README files.
As #Chris suggested, you can use Pandoc to convert Markdown to RST. This can be simply automated using pypandoc module and some magic in setup.py:
from setuptools import setup
try:
from pypandoc import convert
read_md = lambda f: convert(f, 'rst')
except ImportError:
print("warning: pypandoc module not found, could not convert Markdown to RST")
read_md = lambda f: open(f, 'r').read()
setup(
# name, version, ...
long_description=read_md('README.md'),
install_requires=[]
)
This will automatically convert README.md to RST for the long description using on PyPi. When pypandoc is not available, then it just reads README.md without the conversion – to not force others to install pypandoc when they wanna just build the module, not upload to PyPi.
So you can write in Markdown as usual and don’t care about RST mess anymore. ;)
2019 Update
The PyPI Warehouse now supports rendering Markdown as well! You just need to update your package configuration and add the long_description_content_type='text/markdown' to it. e.g.:
setup(
name='an_example_package',
# other arguments omitted
long_description=long_description,
long_description_content_type='text/markdown'
)
Therefore, there is no need to keep the README in two formats any longer.
You can find more information about it in the documentation.
Old answer:
The Markup library used by GitHub supports reStructuredText. This means you can write a README.rst file.
They even support syntax specific color highlighting using the code and code-block directives (Example)
PyPI now supports Markdown for long descriptions!
In setup.py, set long_description to a Markdown string, add long_description_content_type="text/markdown" and make sure you're using recent tooling (setuptools 38.6.0+, twine 1.11+).
See Dustin Ingram's blog post for more details.
You might also be interested in the fact that it is possible to write in a common subset so that your document comes out the same way when rendered as markdown or rendered as reStructuredText: https://gist.github.com/dupuy/1855764 ☺
For my requirements I didn't want to install Pandoc in my computer. I used docverter. Docverter is a document conversion server with an HTTP interface using Pandoc for this.
import requests
r = requests.post(url='http://c.docverter.com/convert',
data={'to':'rst','from':'markdown'},
files={'input_files[]':open('README.md','rb')})
if r.ok:
print r.content
I ran into this problem and solved it with the two following bash scripts.
Note that I have LaTeX bundled into my Markdown.
#!/usr/bin/env bash
if [ $# -lt 1 ]; then
echo "$0 file.md"
exit;
fi
filename=$(basename "$1")
extension="${filename##*.}"
filename="${filename%.*}"
if [ "$extension" = "md" ]; then
rst=".rst"
pandoc $1 -o $filename$rst
fi
Its also useful to convert to html. md2html:
#!/usr/bin/env bash
if [ $# -lt 1 ]; then
echo "$0 file.md <style.css>"
exit;
fi
filename=$(basename "$1")
extension="${filename##*.}"
filename="${filename%.*}"
if [ "$extension" = "md" ]; then
html=".html"
if [ -z $2 ]; then
# if no css
pandoc -s -S --mathjax --highlight-style pygments $1 -o $filename$html
else
pandoc -s -S --mathjax --highlight-style pygments -c $2 $1 -o $filename$html
fi
fi
I hope that helps
Using the pandoc tool suggested by others I created a md2rst utility to create the rst files. Even though this solution means you have both an md and an rst it seemed to be the least invasive and would allow for whatever future markdown support is added. I prefer it over altering setup.py and maybe you would as well:
#!/usr/bin/env python
'''
Recursively and destructively creates a .rst file for all Markdown
files in the target directory and below.
Created to deal with PyPa without changing anything in setup based on
the idea that getting proper Markdown support later is worth waiting
for rather than forcing a pandoc dependency in sample packages and such.
Vote for
(https://bitbucket.org/pypa/pypi/issue/148/support-markdown-for-readmes)
'''
import sys, os, re
markdown_sufs = ('.md','.markdown','.mkd')
markdown_regx = '\.(md|markdown|mkd)$'
target = '.'
if len(sys.argv) >= 2: target = sys.argv[1]
md_files = []
for root, dirnames, filenames in os.walk(target):
for name in filenames:
if name.endswith(markdown_sufs):
md_files.append(os.path.join(root, name))
for md in md_files:
bare = re.sub(markdown_regx,'',md)
cmd='pandoc --from=markdown --to=rst "{}" -o "{}.rst"'
print(cmd.format(md,bare))
os.system(cmd.format(md,bare))
I downloaded and installed libjingle-0.5.2.zip, and according to the README also downloaded and installed swtoolkit.0.9.1.zip, scons-local-2.1.0.alpha.20101125.tar.gz, and expat-2.0.1.tar.gz, and got nrtp by cvs download. After overwriting my Makefile twice, attempting to follow the rather poorly-written README, I came up with the following Makefile that almost works:
# First, make sure the SCONS_DIR environment variable is set correctly.
SCONS_DIR ?= /usr/src/scons-local/scons-local-2.1.0.alpha.20101125/
#SCONS_DIR ?= /usr/src/scons-local/
export
default: build
# Second, run talk/third_party/expat-2.0.1/configure...
talk/third_party/expat-2.0.1/Makefile:
cd talk/third_party/expat-2.0.1 && ./configure
# ...and talk/third_party/srtp/configure.
talk/third_party/srtp/Makefile:
cd talk/third_party/srtp && ./configure
# Third, go to the talk/ directory and run $path_to_swtoolkit/hammer.sh. Run
# $path_to_swtoolkit/hammer.sh --help for information on how to build for
# different modes.
build: talk/third_party/expat-2.0.1/Makefile talk/third_party/srtp/Makefile
cd talk && ../../swtoolkit/hammer.sh
help:
../swtoolkit/hammer.sh --help
However, make gives me the following errors:
jcomeau#intrepid:/usr/src/libjingle-0.5.2$ make
cd talk && ../../swtoolkit/hammer.sh
*** Error loading site_init file './../../swtoolkit/site_scons/site_init.py':
AttributeError: 'Dir' object has no attribute 'endswith':
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 1338:
_exec_main(parser, values)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 1302:
_main(parser)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 929:
_load_site_scons_dir(d.path, options.site_dir)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 719:
exec fp in site_m
File "./../../swtoolkit/site_scons/site_init.py", line 455:
SiteInitMain()
File "./../../swtoolkit/site_scons/site_init.py", line 451:
SCons.Node.FS.get_default_fs().SConstruct_dir, None)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 677:
site_dir = os.path.join(topdir, site_dir_name)
File "/usr/lib/python2.6/posixpath.py", line 67:
elif path == '' or path.endswith('/'):
make: *** [build] Error 2
I'm guessing that something new (a 'Dir' object being where a POSIX path string is expected) in one of the packages is breaking the build process, but which one? There are just too many layers of cruft here for me to follow. For sure I could just keep trying older packages, particularly for swtoolkit and scons, but if anyone here has successfully compiled libjingle and could prod me in the right direction, I'd appreciate it.
I'm not familiar with the project, but think I have a fix to get you past that point. You need to cast those Dir instances using str() in swtoolkit/site_scons/site_init.py. That way they can safely be evaluated by path.endswith('/'). Odd that such an issue would exist for very long in the main part of the build infrastructure:
Line 330:
SCons.Script.Main._load_site_scons_dir(
str(SCons.Node.FS.get_default_fs().SConstruct_dir), site_dir)
Line 450:
SCons.Script.Main._load_site_scons_dir(
str(SCons.Node.FS.get_default_fs().SConstruct_dir), None)
I did following to build libjingle :
Building LibJingle for Linux
How to Build
Libjingle is built with swtoolkit ( http://code.google.com/p/swtoolkit/), which
is a set of extensions to the open-source SCons build tool ( http://www.scons.org).
First, install Python 2.4 or later from http://www.python.org/.
Please note that since swtoolkit only works with Python 2.x, you will
not be able to use Python 3.x.
Second, install the stand alone scons-local package 2.0.0 or later from
http://www.scons.org/download.php and set an environment variable,
SCONS_DIR, to point to the directory containing SCons, for example,
/src/libjingle/scons-local/scons-local-2.0.0.final.0/.
Third, install swtoolkit from http://code.google.com/p/swtoolkit/.
Finally, Libjingle depends on two open-source projects, expat and srtp.
Download expat from http://sourceforge.net/projects/expat/ to
talk/third_party/expat-2.0.1/. Follow the instructions at
http://sourceforge.net/projects/srtp/develop to download latest srtp to
talk/third_party/srtp. Note that srtp-1.4.4 does not work since it misses
the extensions used by Libjingle.
If you put expat or srtp in a different directory, you need to edit
talk/libjingle.scons correspondingly.
2.1 Build Libjingle under Linux or OS X
First, make sure the SCONS_DIR environment variable is set correctly.
Second, run talk/third_party/expat-2.0.1/configure and
talk/third_party/srtp/configure.
Third, go to the talk/ directory and run $path_to_swtoolkit/hammer.sh. Run
$path_to_swtoolkit/hammer.sh --help for information on how to build for
different modes.
Other than above given steps, See following as reference
Set SCONS_DIR Path
export SCONS_DIR=/home/esumit/libjingle/libjingle-0.5.2/talk/third_party/scons-local/scons-local-2.0.1
Install libasound2-dev Lib to compile libJingle, otherwise you will encounter errors.
sudo apt-get install libasound2-dev
Download SRTP using the following command. If it asks for a passowrd, just hit Enter.
cvs -z3 -d:pserver:anonymous#srtp.cvs.sourceforge.net:/cvsroot/srtp co -P srtp
Possible components in LibJingle Directory
libjingle-0.5.2/talk/third_party$ ls
expat-2.0.1 libudev scons-local srtp swtoolkit
Execute following command to build LibJingle
libjingle-0.5.2/talk$ ./third_party/swtoolkit/hammer.sh