We Keep Coding

How to delete python on new M1 from back up

I have a new m1 laptop and I transferred all my data from my old machine onto. Including old python installations. I can't work out how to find and delete all of them so I can clean install with brew in the new version.
Brew on M1 also looks in a different directory for installs now so it can't find the old versions (inside local/bin) to uninstall them.

If brew list pythonfails, try “the brute force” method, i.e. search your entire drive for executables that are called python-×.×.
This command will not delete anything. It is up to you to manually delete the files you don’t want anymore.
find / -perm +111 -type f -or -type l 2> /dev/null | grep '/python[-0123456789\.]*$'
Explanation
find / \ # search starting at the topmost level of your drive
-perm +111 \ # executables files only
-type f -or -type l \ # only files or symlinks
2> /dev/null \ # discard errors
| grep '/python[-0123456789\.]*$' # regexp to filter names

How to get the location of installed Python package into the shell

I want my users to be able to reference a file in my python package (specifically a docker-compose.yml file) directly from the shell.
I couldnt find a way to get only the location from pip show (and grep-ing out "location" from its output feels ugly), so my current (somewhat verbose) solution is:
docker compose -f $(python3 -c "import locust_plugins; print(locust_plugins.__path__[0])")/timescale/docker-compose.yml up
Is there a better way?
Edit: I solved it by installing a wrapper command I call locust-compose as part of the package. Not perfect, but it gets the job done:
#!/bin/bash
module_location=$(python3 -c "import locust_plugins; print(locust_plugins.__path__[0])")
set -x
docker compose -f $module_location/timescale/docker-compose.yml "$#"

Most of the support you need for this is in the core setuptools suite.
First of all, you need to make sure the data file is included in your package. In a setup.cfg file you can write:
[options.package_data]
timescale = docker-compose.yml
Now if you pip install . or pip wheel, that will include the Compose file as part of the Python package.
Next, you can retrieve this in Python code using the ResourceManager API:
#!/usr/bin/env python3
# timescale/compose_path.py
import pkg_resources
if __name__ == '__main__':
print(pkg_resources.resource_filename('timescale', 'docker-compose.yml'))
And finally, you can take that script and make it a setuptools entry point script (as distinct from the similarly-named Docker concept), so that you can just run it as a single command.
[options.entry_points]
console_scripts=
timescale_compose_path = timescale:compose_path
Again, if you pip install . into a virtual environment, you should be able to run timescale_compose_path and get the path name out.
Having done all of those steps, you can finally run a simpler
docker-compose -f $(timescale_compose_path) up

Creating a python package (deb/rpm) from cmake

I am trying to create a python package (deb & rpm) from cmake, ideally using cpack. I did read
https://cmake.org/cmake/help/latest/cpack_gen/rpm.html and,
https://cmake.org/cmake/help/latest/cpack_gen/deb.html
The installation works just fine (using component install) for my shared library. However I cannot make sense of the documentation to install the python binding (glue) code. Using the standard cmake install mechanism, I tried:
install(
FILES __init__.py library.py
DESTINATION ${ACME_PYTHON_PACKAGE_DIR}/project_name
COMPONENT python)
And then using brute-force approach ended-up with:
# debian based package (relative path)
set(ACME_PYTHON_PACKAGE_DIR lib/python3/dist-packages)
and
# rpm based package (full path required)
set(ACME_PYTHON_PACKAGE_DIR /var/lang/lib/python3.8/site-packages)
The above is derived from:
debian % python -c 'import site; print(site.getsitepackages())'
['/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.9/dist-packages']
while:
rpm % python -c 'import site; print(site.getsitepackages())'
['/var/lang/lib/python3.8/site-packages']
It is pretty clear that the brute-force approach will not be portable, and is doomed to fail on the next release of python. The only possible solution that I can think of is generating a temporary setup.py python script (using setuptools), that will do the install. Typically cmake would call the following process:
% python setup.py install --root ${ACME_PYTHON_INSTALL_ROOT}
My questions are:
Did I understand the cmake/cpack documentation correctly for python package ? If so this means I need to generate an intermediate setup.py script.
I have been searching through the cmake/cpack codebase (git grep setuptools) but did not find helper functions to handle generation of setup.py and passing the result files back to cpack. Is there an existing cmake module which I could re-use ?
I did read, some alternative solution, such as:
How to build debian package with CPack to execute setup.py?
Which seems overly complex, and geared toward Debian-only based system. I need to handle RPM in my case.

As mentionned in my other solution, the ugly part is dealing with absolute path in cmake install() commands. I was able to refactor the code to avoid usage of absolute path in install(). I simply changed the installation into:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
# "." syntax is a reliable mechanism, see:
# https://gitlab.kitware.com/cmake/cmake/-/issues/22616
DESTINATION "."
COMPONENT python)
And then one simply needs to:
set(CMAKE_INSTALL_PREFIX "/")
set(CPACK_PACKAGING_INSTALL_PREFIX "/")
include(CPack)
At this point all install path now need to include explicitely /usr since we've cleared the value for CMAKE_INSTALL_PREFIX.
The above has been tested for deb and rpm packages. CPACK_BINARY_TGZ does properly run with the above solution:
https://gitlab.kitware.com/cmake/cmake/-/issues/22925

I am going to post the temporary solution I am using at the moment, until someone provide something more robust.
So I eventually manage to stumble upon:
https://alioth-lists.debian.net/pipermail/libkdtree-devel/2012-October/000366.html and,
Using CMake with setup.py
Re-using the above to do an install step instead of a build step can be done as follow:
find_package(Python COMPONENTS Interpreter)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(SETUP_DEPS "${CMAKE_CURRENT_SOURCE_DIR}/project_name/__init__.py")
set(SETUP_OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build-python")
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
COMMAND ${Python_EXECUTABLE} ARGS ${SETUP_PY} install --root ${SETUP_OUTPUT}
COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
DEPENDS ${SETUP_DEPS})
add_custom_target(target ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp)
And then the ugly part is:
install(
# trailing slash is important:
DIRECTORY ${SETUP_OUTPUT}/
DESTINATION "/" # FIXME may cause issues with other cpack generators
COMPONENT python)
Turns out that the documentation for install() is pretty clear about absolute paths:
https://cmake.org/cmake/help/latest/command/install.html#introduction
DESTINATION
[...]
As absolute paths are not supported by cpack installer generators,
it is preferable to use relative paths throughout.
For reference, here is my setup.py.in:
from setuptools import setup
if __name__ == '__main__':
setup(name='project_name_python',
version='${PROJECT_VERSION}',
package_dir={'': '${CMAKE_CURRENT_SOURCE_DIR}'},
packages=['project_name'])
You can be fancy and remove the __pycache__ folder using the -B flag:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT}
You can be extra fancy and add debian option such as:
if(CPACK_BINARY_DEB)
set(EXTRA_ARG "--install-layout" "deb")
endif()
use as:
COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT} ${EXTRA_ARG}

no module name pip : after changing permission of python folder

here is what i did before i messed up everything :
i tried to install a package using pip3 , after a long time the download finished and suddenly the error about permission came up because i forgot to use sudo at first and because I didn't want to download the packages again and didn't know where is the pip cache folder is , I did a very stupid thing i changed the permission of the entire python folder in the /usr/bin/ to install package without sudo , after this i tried this :
pip3 install tensorflow
File "/usr/bin/pip3", line 7, in <module>
from pip import main
ImportError: No module named 'pip'
i got these damn error ,
can anybody help me fix this ?
Edit :
here is my sequence of the command i used :
1 - pip3 install tensorflow
-- the error came up
2 - sudo find /usr/lib/python3.5/ -type d -exec chmod 766 {} \;
3 - sudo find /usr/lib/python3.5/ -type f -exec chmod 766 {} \;

First, and foremost, I consider your approach quite unwise. You have now changed the permissions of all files and directories for the owner, the group, and others.
In principle you just needed to ensure that pip3 (by extension, your user-account) would be able write files and directories in a directory owned by root (presumably /usr/lib/python3.5/site-packages). You could have accomplished this by:
sudo chmod o+w /usr/lib/python3.5/site-packages
Alternatively you could have changed the ownership of this folder. IMPORTANT: when doing this kind of thing, be sure to know what you are doing, and don't forget to change everything back as soon as possible. Things can be broken, and security issues can be created.
Now as to a solution to your problem. You have now given the directories the following permissions -rwxrw-rw- (6 = 4 (read) + 2 (write)). However for users, and programs executed on its behalf, to do anything to/from a directory they need the right to execute. For this you should have used 5 instead of 6 (5 = 4 (read) + 1 (execute)). To correct:
sudo find /usr/lib/python3.5/ -type d -exec chmod 755 {} \;
Then, I think that for Python to correctly load compiled libraries (shared-objects or .so files) they should also have these permissions. Judging from my own Python directory I would probably do:
sudo find /usr/lib/python3.5/ -type f -exec chmod 644 {} \;
sudo find /usr/lib/python3.5/ -type f -iname '*.so' -exec chmod 755 {} \;
to set everything back to it's original state.
P.S. I am no expert in pip, so I have no idea what the protocol is to avoid pip from re-downloading upon retrying a failed installation.

Patching broken file

In my travis setup I want to apply a patch to a very specific python file that is broken on some configurations. The installation procedure looks like this:
install:
# test with various Django versions
- pip install Django==$DJANGO
# patch Django 1.4 libgeos
- 'if [ $DJANGO == "1.4.13" ]; then find . -name libgeos.py -exec patch {} travis/geos-dev.patch \; ; fi'
But apparantly, find is not able to retrieve the file in question.
How can I realiably find this file, also across different python versions?

Seems like a simple find under /home/travis/virtualenv/ does the job.