We Keep Coding

How to get the location of installed Python package into the shell

I want my users to be able to reference a file in my python package (specifically a docker-compose.yml file) directly from the shell.
I couldnt find a way to get only the location from pip show (and grep-ing out "location" from its output feels ugly), so my current (somewhat verbose) solution is:
docker compose -f $(python3 -c "import locust_plugins; print(locust_plugins.__path__[0])")/timescale/docker-compose.yml up
Is there a better way?
Edit: I solved it by installing a wrapper command I call locust-compose as part of the package. Not perfect, but it gets the job done:
#!/bin/bash
module_location=$(python3 -c "import locust_plugins; print(locust_plugins.__path__[0])")
set -x
docker compose -f $module_location/timescale/docker-compose.yml "$#"

Most of the support you need for this is in the core setuptools suite.
First of all, you need to make sure the data file is included in your package. In a setup.cfg file you can write:
[options.package_data]
timescale = docker-compose.yml
Now if you pip install . or pip wheel, that will include the Compose file as part of the Python package.
Next, you can retrieve this in Python code using the ResourceManager API:
#!/usr/bin/env python3
# timescale/compose_path.py
import pkg_resources
if __name__ == '__main__':
print(pkg_resources.resource_filename('timescale', 'docker-compose.yml'))
And finally, you can take that script and make it a setuptools entry point script (as distinct from the similarly-named Docker concept), so that you can just run it as a single command.
[options.entry_points]
console_scripts=
timescale_compose_path = timescale:compose_path
Again, if you pip install . into a virtual environment, you should be able to run timescale_compose_path and get the path name out.
Having done all of those steps, you can finally run a simpler
docker-compose -f $(timescale_compose_path) up

Creating a python package (deb/rpm) from cmake

I am trying to create a python package (deb & rpm) from cmake, ideally using cpack. I did read
https://cmake.org/cmake/help/latest/cpack_gen/rpm.html and,
https://cmake.org/cmake/help/latest/cpack_gen/deb.html
The installation works just fine (using component install) for my shared library. However I cannot make sense of the documentation to install the python binding (glue) code. Using the standard cmake install mechanism, I tried:
install(
FILES __init__.py library.py
DESTINATION ${ACME_PYTHON_PACKAGE_DIR}/project_name
COMPONENT python)
And then using brute-force approach ended-up with:
# debian based package (relative path)
set(ACME_PYTHON_PACKAGE_DIR lib/python3/dist-packages)
and
# rpm based package (full path required)
set(ACME_PYTHON_PACKAGE_DIR /var/lang/lib/python3.8/site-packages)
The above is derived from:
debian % python -c 'import site; print(site.getsitepackages())'
['/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.9/dist-packages']
while:
rpm % python -c 'import site; print(site.getsitepackages())'
['/var/lang/lib/python3.8/site-packages']
It is pretty clear that the brute-force approach will not be portable, and is doomed to fail on the next release of python. The only possible solution that I can think of is generating a temporary setup.py python script (using setuptools), that will do the install. Typically cmake would call the following process:
% python setup.py install --root ${ACME_PYTHON_INSTALL_ROOT}
My questions are:
Did I understand the cmake/cpack documentation correctly for python package ? If so this means I need to generate an intermediate setup.py script.
I have been searching through the cmake/cpack codebase (git grep setuptools) but did not find helper functions to handle generation of setup.py and passing the result files back to cpack. Is there an existing cmake module which I could re-use ?
I did read, some alternative solution, such as:
How to build debian package with CPack to execute setup.py?
Which seems overly complex, and geared toward Debian-only based system. I need to handle RPM in my case.

As mentionned in my other solution, the ugly part is dealing with absolute path in cmake install() commands. I was able to refactor the code to avoid usage of absolute path in install(). I simply changed the installation into:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
# "." syntax is a reliable mechanism, see:
# https://gitlab.kitware.com/cmake/cmake/-/issues/22616
DESTINATION "."
COMPONENT python)
And then one simply needs to:
set(CMAKE_INSTALL_PREFIX "/")
set(CPACK_PACKAGING_INSTALL_PREFIX "/")
include(CPack)
At this point all install path now need to include explicitely /usr since we've cleared the value for CMAKE_INSTALL_PREFIX.
The above has been tested for deb and rpm packages. CPACK_BINARY_TGZ does properly run with the above solution:
https://gitlab.kitware.com/cmake/cmake/-/issues/22925

I am going to post the temporary solution I am using at the moment, until someone provide something more robust.
So I eventually manage to stumble upon:
https://alioth-lists.debian.net/pipermail/libkdtree-devel/2012-October/000366.html and,
Using CMake with setup.py
Re-using the above to do an install step instead of a build step can be done as follow:
find_package(Python COMPONENTS Interpreter)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(SETUP_DEPS "${CMAKE_CURRENT_SOURCE_DIR}/project_name/__init__.py")
set(SETUP_OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build-python")
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
COMMAND ${Python_EXECUTABLE} ARGS ${SETUP_PY} install --root ${SETUP_OUTPUT}
COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
DEPENDS ${SETUP_DEPS})
add_custom_target(target ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp)
And then the ugly part is:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
DESTINATION "/" # FIXME may cause issues with other cpack generators
COMPONENT python)
Turns out that the documentation for install() is pretty clear about absolute paths:
https://cmake.org/cmake/help/latest/command/install.html#introduction
DESTINATION
[...]
As absolute paths are not supported by cpack installer generators,
it is preferable to use relative paths throughout.
For reference, here is my setup.py.in:
from setuptools import setup
if __name__ == '__main__':
setup(name='project_name_python',
version='${PROJECT_VERSION}',
package_dir={'': '${CMAKE_CURRENT_SOURCE_DIR}'},
packages=['project_name'])
You can be fancy and remove the __pycache__ folder using the -B flag:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT}
You can be extra fancy and add debian option such as:
if(CPACK_BINARY_DEB)
set(EXTRA_ARG "--install-layout" "deb")
endif()
use as:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT} ${EXTRA_ARG}

Python BDD Behave allure reports failing with format=allure_behave.formatter:AllureFormatter is unknown

I have installed allure 2.8.17. It is on a pipenv environment. When I am running the below command in terminal:
behave -f allure_behave.formatter:AllureFormatter -o reports/
It is failing with the below error
usage: behave [options] [ [DIR|FILE|FILE:LINE] ]+
behave: error: format=allure_behave.formatter:AllureFormatter is unknown
(behave) sharathkrishnan#sharaths-mbp features %

I think you are missing allure-behave:
pip install allure-behave ?

you need to install using pip install allure-behave and then run the above command

This also commonly happens when you point the file path with a typo.
Mostly case sensitive problems like ./Reports instead of ./reports

make file target to check for installed dependencies

I have a makefile where I have targets that depend on having some external clients installed (python3, libxml2, etc).
Here is my makefile
.PHONY: test install-packages mac-setup checkenv target help
EXTERNALS = python3 pip3 xmllint pytest pipenv
P := $(foreach exec,$(EXTERNALS),$(if $(shell which $(exec)),missing,$(warning "===>>>WARNING: No required `$(exec)` in PATH, run `make mac-setup` + `make install-packages` <<<===")))
test: ## run all tests in test directory
pipenv run pytest -v --ignore=path payload_files .
install-packages: ##install python packages listed in Pipfile
pipenv install
mac-setup: ## setup mac for testing
brew install libxml2
brew install python3
brew install pipenv
# see https://github.mycompany.com/ea/ea_test_player_unified/blob/master/run-feature.sh
help:
#grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
.DEFAULT_GOAL := help
Notice the line
P := $(foreach exec,$(EXTERNALS),$(if $(shell which $(exec)),missing,$(warning "===>>>WARNING: No required `$(exec)` in PATH, run `make mac-setup` + `make install-packages` <<<===")))
This checks for the binaries required. This works.... however I would rather have a checkenv target that performs this and errors so I can attach it too specific targets like test instead of printing out a WARNING that might be overlooked.
Want:
checkenv: # error if which ${binary} fails or *even better* if if binary --version doesn't return the right version: python3 pip3 xmllint pytest pipenv
I tried various techniques that I found around the web including stackoverflow.... but most use the technique I am using above that don't use a make target or just check for one binary. I tried building a loop through an array of binaries but just couldn't get the syntax correct due to make being a PITA :)
Any suggestions?
Note I'm a python newbie, task is to rewrite some jmeter tests in python....so if you have any thoughts on the above approach feel free to share.
Thanks,
Phil

Don't see what the problem is. It looks very straightforward to me, as make allows using multiple targets on the same line:
EXTERNALS := python3 pip3 xmllint pytest pipenv
python3_version := Python 3.7.3
pip3_version := ...
...
.PHONY: checkenv $(EXTERNALS)
checkenv: $(EXTERNALS)
$(EXTERNALS):
if [ "`$# --version`" != "$($#_version)" ]; then echo "$# check failed"; false; fi

pyreport LaTeX formulae not working

I'm trying to create a HTML report using pyreport and it works up to the single point, that the LaTeX formulae are not generated.
Here is the input file I use for testing:
#$ This is \LaTeX : $c = 2\cdot(a+b)$
Than I run pyreport -l -t html --verbose file.py, but the report that I get is empty. When I add other comments to the input file, or some Python code, than it is displayed properly within the report. Here is the output from pyreport:
Running python script
/tmp/file.py:
Outputing report to
/tmp/file.html Ran script in 0.13s
I'm using Ubuntu and I have the texlive package installed. Why isn't the formula added to the report?

I think i have find the problem.
The problem is the RST tools to convert in html.
In pyreport, when you choose the math mode, the program will do the sentence in a bock .. raw:: LaTeX
But in the new version of rst2html, this command doesnt work, it's replace by:
.. math::
If you use the command:
pyreport -l -e -t rst --verbose file.py
and after
rst2html file.rst > test.html
You will see the problem.
You can change that in pyreport code, in main.py of pyreport. (use a locate to find it). And replace the
.. raw:: Latex
, by
.. math::
The last problem is for the command \LaTeX, that's not in a math mode of latex. So it's not work.
You can report to RST documentation http://docutils.sourceforge.net/docs/ref/rst/directives.html#raw