We Keep Coding

How to get the location of installed Python package into the shell

I want my users to be able to reference a file in my python package (specifically a docker-compose.yml file) directly from the shell.
I couldnt find a way to get only the location from pip show (and grep-ing out "location" from its output feels ugly), so my current (somewhat verbose) solution is:
docker compose -f $(python3 -c "import locust_plugins; print(locust_plugins.__path__[0])")/timescale/docker-compose.yml up
Is there a better way?
Edit: I solved it by installing a wrapper command I call locust-compose as part of the package. Not perfect, but it gets the job done:
#!/bin/bash
module_location=$(python3 -c "import locust_plugins; print(locust_plugins.__path__[0])")
set -x
docker compose -f $module_location/timescale/docker-compose.yml "$#"

Most of the support you need for this is in the core setuptools suite.
First of all, you need to make sure the data file is included in your package. In a setup.cfg file you can write:
[options.package_data]
timescale = docker-compose.yml
Now if you pip install . or pip wheel, that will include the Compose file as part of the Python package.
Next, you can retrieve this in Python code using the ResourceManager API:
#!/usr/bin/env python3
# timescale/compose_path.py
import pkg_resources
if __name__ == '__main__':
print(pkg_resources.resource_filename('timescale', 'docker-compose.yml'))
And finally, you can take that script and make it a setuptools entry point script (as distinct from the similarly-named Docker concept), so that you can just run it as a single command.
[options.entry_points]
console_scripts=
timescale_compose_path = timescale:compose_path
Again, if you pip install . into a virtual environment, you should be able to run timescale_compose_path and get the path name out.
Having done all of those steps, you can finally run a simpler
docker-compose -f $(timescale_compose_path) up

Creating a python package (deb/rpm) from cmake

I am trying to create a python package (deb & rpm) from cmake, ideally using cpack. I did read
https://cmake.org/cmake/help/latest/cpack_gen/rpm.html and,
https://cmake.org/cmake/help/latest/cpack_gen/deb.html
The installation works just fine (using component install) for my shared library. However I cannot make sense of the documentation to install the python binding (glue) code. Using the standard cmake install mechanism, I tried:
install(
FILES __init__.py library.py
DESTINATION ${ACME_PYTHON_PACKAGE_DIR}/project_name
COMPONENT python)
And then using brute-force approach ended-up with:
# debian based package (relative path)
set(ACME_PYTHON_PACKAGE_DIR lib/python3/dist-packages)
and
# rpm based package (full path required)
set(ACME_PYTHON_PACKAGE_DIR /var/lang/lib/python3.8/site-packages)
The above is derived from:
debian % python -c 'import site; print(site.getsitepackages())'
['/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.9/dist-packages']
while:
rpm % python -c 'import site; print(site.getsitepackages())'
['/var/lang/lib/python3.8/site-packages']
It is pretty clear that the brute-force approach will not be portable, and is doomed to fail on the next release of python. The only possible solution that I can think of is generating a temporary setup.py python script (using setuptools), that will do the install. Typically cmake would call the following process:
% python setup.py install --root ${ACME_PYTHON_INSTALL_ROOT}
My questions are:
Did I understand the cmake/cpack documentation correctly for python package ? If so this means I need to generate an intermediate setup.py script.
I have been searching through the cmake/cpack codebase (git grep setuptools) but did not find helper functions to handle generation of setup.py and passing the result files back to cpack. Is there an existing cmake module which I could re-use ?
I did read, some alternative solution, such as:
How to build debian package with CPack to execute setup.py?
Which seems overly complex, and geared toward Debian-only based system. I need to handle RPM in my case.

As mentionned in my other solution, the ugly part is dealing with absolute path in cmake install() commands. I was able to refactor the code to avoid usage of absolute path in install(). I simply changed the installation into:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
# "." syntax is a reliable mechanism, see:
# https://gitlab.kitware.com/cmake/cmake/-/issues/22616
DESTINATION "."
COMPONENT python)
And then one simply needs to:
set(CMAKE_INSTALL_PREFIX "/")
set(CPACK_PACKAGING_INSTALL_PREFIX "/")
include(CPack)
At this point all install path now need to include explicitely /usr since we've cleared the value for CMAKE_INSTALL_PREFIX.
The above has been tested for deb and rpm packages. CPACK_BINARY_TGZ does properly run with the above solution:
https://gitlab.kitware.com/cmake/cmake/-/issues/22925

I am going to post the temporary solution I am using at the moment, until someone provide something more robust.
So I eventually manage to stumble upon:
https://alioth-lists.debian.net/pipermail/libkdtree-devel/2012-October/000366.html and,
Using CMake with setup.py
Re-using the above to do an install step instead of a build step can be done as follow:
find_package(Python COMPONENTS Interpreter)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(SETUP_DEPS "${CMAKE_CURRENT_SOURCE_DIR}/project_name/__init__.py")
set(SETUP_OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build-python")
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
COMMAND ${Python_EXECUTABLE} ARGS ${SETUP_PY} install --root ${SETUP_OUTPUT}
COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
DEPENDS ${SETUP_DEPS})
add_custom_target(target ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp)
And then the ugly part is:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
DESTINATION "/" # FIXME may cause issues with other cpack generators
COMPONENT python)
Turns out that the documentation for install() is pretty clear about absolute paths:
https://cmake.org/cmake/help/latest/command/install.html#introduction
DESTINATION
[...]
As absolute paths are not supported by cpack installer generators,
it is preferable to use relative paths throughout.
For reference, here is my setup.py.in:
from setuptools import setup
if __name__ == '__main__':
setup(name='project_name_python',
version='${PROJECT_VERSION}',
package_dir={'': '${CMAKE_CURRENT_SOURCE_DIR}'},
packages=['project_name'])
You can be fancy and remove the __pycache__ folder using the -B flag:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT}
You can be extra fancy and add debian option such as:
if(CPACK_BINARY_DEB)
set(EXTRA_ARG "--install-layout" "deb")
endif()
use as:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT} ${EXTRA_ARG}

git executable not found in python

I was trying to clone a git repo with access key, but when I am trying to run it, It throws an exception saying git executable not found.
But i have installed git and the in_it.py shows correct path "C:\Program Files\Git\bin" Also I have installed gitpython to use the library in python
here's my code...
import git
git.Git("D:/madhav/myrep/").clone("#github.com:myrepo/scripts")
========= and it throws the following exception =================
Traceback (most recent call last): File
"C:\Users\1096506\Desktop\gitclone.py", line 1, in <module>
from git import Repo File "C:\Users\1096506\AppData\Local\Programs\Python\Python36-32\lib\site-packages\git\__init__.py",
line 84, in <module>
refresh() File "C:\Users\1096506\AppData\Local\Programs\Python\Python36-32\lib\site-packages\git\__init__.py",
line 73, in refresh
if not Git.refresh(path=path): File "C:\Users\1096506\AppData\Local\Programs\Python\Python36-32\lib\site-packages\git\cmd.py",
line 293, in refresh
raise ImportError(err) ImportError: Bad git executable. The git executable must be specified in one of the following ways:
- be included in your $PATH
- be set via $GIT_PYTHON_GIT_EXECUTABLE
- explicitly set via git.refresh()
All git commands will error until this is rectified.
This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
- quiet|q|silence|s|none|n|0: for no warning or exception
- warn|w|warning|1: for a printed warning
- error|e|raise|r|2: for a raised exception
Example:
export GIT_PYTHON_REFRESH=quiet

I had the same issue. What I did is:
I went to: System Properties -> Environment Variables
On System Variables Section I clicked Path, Edit and Move Up.
Environment Variables
Edit and Moved Up two places from the bottom

Error occurs because git is not in the path. So its not able to import git module.
Couple of ways to resolve it.
As suggested above adding the git binary path to environment variable path.
If git is not being used directly in the module and its only a dependent module import thats throwing this exception before importing git we could add
os.environ["GIT_PYTHON_REFRESH"] = "quiet"
and import git after this line, this would suppress the error caused due to git import

Had the same problem got it to work thanks to Muthukumaran. Just make Muthukumaran answer more clear.
Follow these steps:
import os
os.environ["GIT_PYTHON_REFRESH"] = "quiet"
import git

See if you have installed Git in the OS.
If not install git first this will solve your error.
Centos
sudo yum -y install git
Ubuntu/Debian
sudo apt-get install git
Mac Os
sudo brew install git
This solved my problem.

Make sure you're not in an inaccessible directory on *nix, such as when you've just been root and then done a su username
you may still be in root's home folder and that will trigger this error (assuming you have the correct environment variables set, and have sourced the .profile or .bashrc etc with source ~/.bashrc )
which git
/usr/bin/git
I was getting this error even after setting the environment:
in ~/.bashrc
# for bench
PATH=$PATH:/usr/bin/git
export PATH
GIT_PYTHON_GIT_EXECUTABLE=/usr/bin/git
export GIT_PYTHON_GIT_EXECUTABLE
cd
and it's working
$ bench --version
WARN: Command not being executed in bench directory
5.3.0

I came across similar problem recently and installing git followed by restarting Windows Powershell CommandLine solved the problem. May it helps.

For those who are using a Lambda layer with it. It worked adding as the comment above says just adding GIT_PYTHON_REFRESH=quiet as an environment variable.

Execute GIT_PYTHON_REFRESH=quiet in your terminal and then try to run the code.

Debianzing a Python program to get a .deb [duplicate]

This question already has answers here:
Is there a standard way to create Debian packages for distributing Python programs?
(5 answers)
Closed 5 years ago.
Aim
To create an installable .deb file (or package). Which when clicked would install the software on a Linux machine and an icon would be put on the GNOME panel. So as to launch this application from there.
What I have referred to
I referred to two debianizing guides.
Guide 1
Guide 2
The first one had a video which was impossible to understand, partly because of the accent and partly because it was hopelessly outdated.(it was uploaded in 2007)
And the second one was completely text. I got till the 4th Step, Builds the package. But when I did it I got output that did not match what was given in the guide.
What I require
I have a simple python program. It takes your age and then prints back out if the age is below, equal to, or above 18 years. There is just one file and no other dependency for this program. And I want to build this into a .deb.
Specs
-Python 2.7
-Linux Mint
Edit
I followed the exact directory structure as you instructed as you. And replaced all myscript with cowsandbulls. The build completed and I got the Debian. When I installed it and then ran the command cowsandbulls from the terminal I got the following error:
Traceback (most recent call last):
File "/usr/bin/cowsandbulls", line 9, in <module>
load_entry_point('cowsandbulls==1.0', 'gui_scripts', 'cowsandbulls')()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 337, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2311, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2017, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
ImportError: No module named cowsandbulls

I just tested stdeb (see https://pypi.python.org/pypi/stdeb), a Python package for turning any other Python package into a Debian package.
First I installed stdeb:
apt-get install python-stdeb
Then I made a simple script called myscript.py with the following contents:
def main():
print "Hello world, says myscript!"
# wait for input from the user
raw_input()
if __name__ == '__main__':
main()
Importantly, your directory structure should be:
somewhere/myscript/
setup.py
myscript/
__init__.py
myscript.py
In the setup.py file, you do something like:
import os
from setuptools import setup
from nvpy import nvpy
setup(
name = "myscript",
version = "1.0",
author = "Charl P. Botha",
author_email = "cpbotha#vxlabs.com",
description = "Demo of packaging a Python script as DEB",
license = "BSD",
url = "https://github.com/cpbotha/nvpy",
packages=['myscript'],
entry_points = {
'console_scripts' : ['myscript = myscript.myscript:main']
},
data_files = [
('share/applications/', ['vxlabs-myscript.desktop'])
],
classifiers=[
"License :: OSI Approved :: BSD License",
],
)
The console_scripts directive is important, and it'll create an executable script called my_script, which will be available system-wide after you install the resultant DEB. If your script uses something like tkinter or wxpython and has a graphical user interface, you should use gui_scripts instead of console_scripts.
The data_files directive will install a suitable desktop file into /usr/share/applications, so that you can also start myscript from your desktop environment. vxlabs-myscript.desktop looks like this:
[Desktop Entry]
Version=1.0
Type=Application
Name=myscript
Comment=Minimal stdeb example
# myscript should wait for user input at the end, else the terminal
# window will disappear immediately.
Exec=myscript
Icon=/usr/share/icons/gnome/48x48/apps/file-manager.png
Categories=Utility;
# desktop should run this in a terminal application
Terminal=true
StartupNotify=true
StartupWMClass=myscript
To build the DEB, you do the following in the top-level myscript:
python setup.py --command-packages=stdeb.command bdist_deb
Which will create a .deb in the deb_dist directory.
After having installed the DEB I created like this, I could run myscript from the command-line, and I could also invoke it from my desktop environment.
Here's a GitHub repository with the example code above: https://github.com/cpbotha/stdeb-minimal-example

The right way of building a deb package is using dpkg-buildpackage, but sometimes it is a little bit complicated. Instead you can use dpkg -b <folder> and it will create your Debian package.
These are the basics for creating a Debian package with dpkg -b <folder> with any binary or with any kind of script that runs automatically without needing manual compilation (Python, Bash, Pearl, and Ruby):
Create the files and folders in order to recreate the following structure:
ProgramName-Version/
ProgramName-Version/DEBIAN
ProgramName-Version/DEBIAN/control
ProgramName-Version/usr/
ProgramName-Version/usr/bin/
ProgramName-Version/usr/bin/your_script
The scripts placed at /usr/bin/ are directly called from the terminal. Note that I didn't add an extension to the script.
Also, you can notice that the structure of the deb package will be the structure of the program once it's installed. So if you follow this logic if your program has a single file, you can directly place it under ProgramName-Version/usr/bin/your_script, but if you have multiple files, you should place them under ProgramName-Version/usr/share/ProgramName/all your files and place only one file under /usr/bin/ that will call your scripts from /usr/share/ProgramName/.
Change all the folder permission to root:
chown root:root -R /path/to/ProgramName-Version
Change the script's permissions:
chmod 0755 /path/to/the/script
Finally, you can run: dpkg -b /path/to/the/ProgramName-Version and your deb package will be created! (You can also add the post/pre inst scripts and everything you want; it works like a normal Debian package.)
Here is an example of the control file. You only need to copy-paste it in to an empty file called "control" and put it in the DEBIAN folder.
Package: ProgramName
Version: VERSION
Architecture: all
Maintainer: YOUR NAME <EMAIL>
Depends: python2.7, etc , etc,
Installed-Size: in_kb
Homepage: http://foo.com
Description: Here you can put a one line description. This is the short Description.
Here you put the long description, indented by 1 space.
If you want to build using dpkg -b <folder> you can use this program that will do everything with one command. If you regularly build packages, it is a pain to do all the stuff that I mentioned!
*The guide was taken from Basics of Debian Packages.

python + distutils: add .py file to package at build/install time?

For various not-very-good-but-unfortunately-necessary reasons I'm using a setup.py file to manage some binary assets.
During py setup.py build or install I would like to create a .py file in the "normal" Python package being installed by setup.py which contains some details about these binary assets (their absolute path, version information, etc).
What's the best way to create that file?
For example, I'd like it to work something like this:
$ cd my-python-package
$ py setup.py install
...
Installing version 1.23 of my_binary_assets to /some/path...
...
$ python -c "from my_python_package import binary_asset_version_info as info; print info"
{"path": "/some/path", "version": "1.23"}
(note: I'm using the cmdclass argument to setup(…) to manage the building + installation of the binary assets… I'd just like to know how to create the binary_asset_version_info.py file used in the example)

At first sight, there is a catch-22 in your requirements: The most obvious place to create this .py file would be in the build or build_py command (to get usual distutils operations like byte-compilation), but you want that file to contain the paths to the installed assets, so you’d have to create it during the install step. I see two ways to solve that:
a) Create your info.py file during build_py, and use distutils machinery to get the installation paths of the assets files
b) Create info.py during install and call distutils.util.byte_compile to byte-compile it
I find both ideas distasteful, but well :) Now, do you know how to fill in the file (i.e. get the install paths from distutils)?