We Keep Coding

Python Readline on macOS behaves differently than on Linux/Windows

A project I'm working on uses a custom CLI handler instead of Python's cmd.Cmd class. Without getting too much in detail, the handler features TAB-key completion to assist the operator with command usage. The feature works as expected on Windows (using pyreadline) and Linux (using GNU's readline).
Here is an example of the expected behavior (assume "cmd > " is the prompt and that [TAB] is a push of the TAB key):
cmd > [TAB]
cd exit load save # all the available commands
cmd > c[TAB] # autocompletes to 'cd'
cmd > cd [TAB]
cd ./folder1 cd ./folder2 cd ./folder3 # folders in the cwd
cmd > cd C:\[TAB]
cd C:\Users cd C:\Windows... # enumerates folders in C:\ (on windows)
cmd > cd /[TAB]
cd /bin cd /opt cd /usr... # enumerates folders from root (on linux)
The custom class defines the following tab completion method, which is set using readline.set_completer():
def tab_completer(self, text, state):
# rl delims set to "" so we get the whole line as a single string
words = re.split(r'[\s\t\n]+', text)
# find_subcompleter populates a list of possible matches or next words
# each command implements its own completer_stub depending on the function (ex: cd will complete directory names)
retval = self.find_subcompleter(words.pop())
try:
return retval[state]
except IndexError:
return None
The function works as expected on Windows (10, Python 3.6.6) and Linux (CentOS 7, Python 3.6.8), but something strange happens on macOS (10.15.7, Python 3.8.2 via xcode on zsh terminal):
cmd > [TAB]
cd exit load save # this is good
cmd > c[TAB] # still autocompletes to 'cd', good
cmd > cd [TAB]
cd exit load save # as if I've typed nothing!
For those of you wondering, this behavior happens with ANY command, not just with cd.
I'm aware that the underlying readline implementation on macOS uses libedit due to GNU licensing. I just haven't seen anyone else (to date) mention this difference on any other forums. A possible solution that comes to mind is to add a conditional for libedit implementations to use get_line_buffer() and redisplay() to mimic the correct behavior. Any pointers in the right direction are appreciated!
Thank you

make file target to check for installed dependencies

I have a makefile where I have targets that depend on having some external clients installed (python3, libxml2, etc).
Here is my makefile
.PHONY: test install-packages mac-setup checkenv target help
EXTERNALS = python3 pip3 xmllint pytest pipenv
P := $(foreach exec,$(EXTERNALS),$(if $(shell which $(exec)),missing,$(warning "===>>>WARNING: No required `$(exec)` in PATH, run `make mac-setup` + `make install-packages` <<<===")))
test: ## run all tests in test directory
pipenv run pytest -v --ignore=path payload_files .
install-packages: ##install python packages listed in Pipfile
pipenv install
mac-setup: ## setup mac for testing
brew install libxml2
brew install python3
brew install pipenv
# see https://github.mycompany.com/ea/ea_test_player_unified/blob/master/run-feature.sh
help:
#grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
.DEFAULT_GOAL := help
Notice the line
P := $(foreach exec,$(EXTERNALS),$(if $(shell which $(exec)),missing,$(warning "===>>>WARNING: No required `$(exec)` in PATH, run `make mac-setup` + `make install-packages` <<<===")))
This checks for the binaries required. This works.... however I would rather have a checkenv target that performs this and errors so I can attach it too specific targets like test instead of printing out a WARNING that might be overlooked.
Want:
checkenv: # error if which ${binary} fails or *even better* if if binary --version doesn't return the right version: python3 pip3 xmllint pytest pipenv
I tried various techniques that I found around the web including stackoverflow.... but most use the technique I am using above that don't use a make target or just check for one binary. I tried building a loop through an array of binaries but just couldn't get the syntax correct due to make being a PITA :)
Any suggestions?
Note I'm a python newbie, task is to rewrite some jmeter tests in python....so if you have any thoughts on the above approach feel free to share.
Thanks,
Phil

Don't see what the problem is. It looks very straightforward to me, as make allows using multiple targets on the same line:
EXTERNALS := python3 pip3 xmllint pytest pipenv
python3_version := Python 3.7.3
pip3_version := ...
...
.PHONY: checkenv $(EXTERNALS)
checkenv: $(EXTERNALS)
$(EXTERNALS):
if [ "`$# --version`" != "$($#_version)" ]; then echo "$# check failed"; false; fi

In NixOS, how can I install an environment with the Python packages SpaCy, pandas, and jenks-natural-breaks?

I'm very new to NixOS, so please forgive my ignorance. I'm just trying to set up a Python environment---any kind of environment---for developing with SpaCy, the SpaCy data, pandas, and jenks-natural-breaks. Here's what I've tried so far:
pypi2nix -V "3.6" -E gcc -E libffi -e spacy -e pandas -e numpy --default-overrides, followed by nix-build -r requirements.nix -A packages. I've managed to get the first command to work, but the second fails with Could not find a version that satisfies the requirement python-dateutil>=2.5.0 (from pandas==0.23.4)
Writing a default.nix that looks like this: with import <nixpkgs> {};
python36.withPackages (ps: with ps; [ spacy pandas scikitlearn ]). This fails with collision between /nix/store/9szpqlby9kvgif3mfm7fsw4y119an2kb-python3.6-msgpack-0.5.6/lib/python3.6/site-packages/msgpack/_packer.cpython-36m-x86_64-linux-gnu.so and /nix/store/d08bgskfbrp6dh70h3agv16s212zdn6w-python3.6-msgpack-python-0.5.6/lib/python3.6/site-packages/msgpack/_packer.cpython-36m-x86_64-linux-gnu.so
Making a new virtualenv, and then running pip install on all these packages. Scikit-learn fails to install, with fish: Unknown command 'ar rc build/temp.linux-x86_64-3.6/liblibsvm-skl.a build/temp.linux-x86_64-3.6/sklearn/svm/src/libsvm/libsvm_template.o'
I guess ideally I'd like to install this environment with nix, so that I could enter it with nix-shell, and so other environments could reuse the same python packages. How would I go about doing that? Especially since some of these packages exist in nixpkgs, and others are only on Pypi.

Caveat
I had trouble with jenks-natural-breaks to the tune of
nix-shell ❯ poetry run python -c 'import jenks_natural_breaks'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/matt/2022/12/28-2/.venv/lib/python3.10/site-packages/jenks_natural_breaks/__init__.py", line 5, in <module>
from ._jenks_matrices import ffi as _ffi
ModuleNotFoundError: No module named 'jenks_natural_breaks._jenks_matrices'
So I'm going to use jenkspy which appears to be a bit livelier. If that doesn't scratch your itch, I'd contact the maintainer of jenks-natural-breaks for guidance
Flakes
you said:
so other environments could reuse the same python packages
Which makes me think that a flake.nix is what you need. What's cool about flakes is that you can define an environment that has spacy, pandas, and jenkspy with one flake. And then you (or somebody else) might say:
I want an env like Jonathan's, except I also want sympy
and rather than copying your env and making tweaks, they can declare your env as a build input and write a flake.nix with their modifications--which can be further modified by others.
One could imagine a sort of family-tree of environments, so you just need to pick the one that suits your task. The python community has not yet converged on this vision.
Poetry
Poetry will treat you like you're trying to publish a library when all you asked for is an environment, but a library's dependencies are pretty much an environment so there's nothing wrong with having an empty package and just using poetry as an environment factory.
Bonus: if you decide to publish a library after all, you're ready.
The Setup
nix flakes thinks in terms of git repo's, so we'll start with one:
$ git init
Then create a file called flake.nix. Usually I end up with poetry handling 90% of the python stuff, but both pandas and spacy are in that 10% that has dependencies which link to system libraries. So we ask nix to install them so that when poetry tries to install them in the nix develop shell, it has what it needs.
{
description = "Jonathan's awesome env";
inputs = {
nixpkgs.url = "github:nixos/nixpkgs";
};
outputs = { self, nixpkgs, flake-utils }: (flake-utils.lib.eachSystem [
"x86_64-linux"
"x86_64-darwin"
"aarch64-linux"
"aarch64-darwin"
] (system:
let
pkgs = nixpkgs.legacyPackages.${system};
in
rec {
packages.jonathansenv = pkgs.poetry2nix.mkPoetryApplication {
projectDir = ./.;
};
defaultPackage = packages.jonathansenv;
devShell = pkgs.mkShell {
buildInputs = [
pkgs.poetry
pkgs.python310Packages.pandas
pkgs.python310Packages.spacy
];
};
}));
}
Now we let git know about the flake and enter the environment:
❯ git add flake.nix
❯ nix develop
$
Then we initialize the poetry project. I've found that poetry, installed by nix, is kind of odd about which python it uses by default, so we'll set it explicitly
$ poetry init # follow prompts
$ poetry env use $(which python)
$ poetry run python --version
Python 3.10.9 # declared in the flake.nix
At this point, we should have a pyproject.toml:
[tool.poetry]
name = "jonathansenv"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.10"
jenkspy = "^0.3.2"
spacy = "^3.4.4"
pandas = "^1.5.2"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Usage
Now we create the venv that poetry will use, and run a command that depends on these.
$ poetry install
$ poetry run python -c 'import jenkspy, spacy, pandas'
You can also have poetry put you in a shell:
$ poetry shell
(venv)$ python -c 'import jenkspy, spacy, pandas'
It's kind of awkward to do so though, because we're two subshells deep and any shell customizations that we have the grandparent shell are not available. So I recommend using direnv, to enter the dev shell whenever I navigate to that directory and then just use poetry run ... to run commands in the environment.
Publishing the env
In addition to running nix develop with the flake.nix in your current dir, you can also do nix develop /local/path/to/repo or develop nix develop github:/githubuser/githubproject to achieve the same result.
To demonstrate the github example, I have pushed the files referenced above here. So you ought to be able to run this from any linux shell with nix installed:
❯ nix develop github:/MatrixManAtYrService/nix-flake-pandas-spacy
$ poetry install
$ poetry run python -c 'import jenkspy, spacy, pandas'
I say "ought" because if I run that command on a mac it complains about linux-headers-5.19.16 being unsupported on x86_64-darwin.
Presumably there's a way to write the flake (or fix a package) so that it doesn't insist on building linux stuff on a mac, but until I figure it out I'm afraid that this is only a partial answer.

Installing Opencv3 in Python on Mac error

I am trying to install Opencv 3 for Python on Mac using this link (https://www.codingforentrepreneurs.com/blog/install-opencv-3-for-python-on-mac/)
I am using python 3.6.4 and I am currently at step 6 (regarding the link)
I am using Python directly from the Mac terminal and when I slot in the following command this error appears:
$ ln -S /usr/local/Cellar/opencv/3.4.0_1/lib/python3.6/site-packages/cv2.cpython-36m-darwin.so /usr/local/lib/python3.6/site-packages/cv2.so
File "<stdin>", line 1
ln -S /usr/local/Cellar/opencv/3.4.0_1/lib/python3.6/site-packages/cv2.cpython-36m-darwin.so /usr/local/lib/python3.6/site-packages/cv2.so
^
SyntaxError: invalid syntax

You essentially need to run a command like:
ln -s SRC TGT
where SRC is the library you have just built/installed and TGT is where you want it to be visible to Python.
Find the SRC file, independently of the OpenCV versions you have installed, like this:
SRC=$(find /usr/local/Cellar/opencv -name "cv2.cpython*so")
and check it looks correct with:
echo "$SRC"
There should be one line. If there is more than one line, it means you have multiple versions of OpenCV installed, so it may look like:
/usr/local/Cellar/opencv/3.4.0_1/lib/python3.6/site-packages/cv2.cpython-36m-darwin.so
/usr/local/Cellar/opencv/3.3.1_7/lib/python3.6/site-packages/cv2.cpython-36m-darwin.so
In that case, choose the one you want and manually set SRC to that one, e.g.
SRC=/usr/local/Cellar/opencv/3.4.0_1/lib/python3.6/site-packages/cv2.cpython-36m-darwin.so
Find the TGT directory, independently of the Python versions you have installed, like this:
TGT=$(find /usr/local/lib/ -type d -name "site-packages")
and check it looks correct with:
echo "$TGT"
There should be one line. If there is more than one line, it means you have multiple versions of Python installed, and it may look something like:
/usr/local/lib/python3.6/site-packages
/usr/local/lib/python3.5/site-packages
If that is the case, choose the one you want to use and set TGT manually, e.g.:
TGT=/usr/local/lib/python3.6/site-packages
Now make the link:
ln -s "$SRC" "$TGT/cv2.so"

In step 6, you first get the sys path, notice that this is done while running python.
You can tell because of the prompt ">>>" in
>>> print(sys.path)
If you then enter:
ln -s /usr/local/Cellar/opencv/3.4.1_4/....
While in Python, you will get the exact error you mentioned.
In order to finish step 6 you now must exit python by typing
>>> exit()
Now you will see the $ prompt.
I realize the way you stated your problem that you were already at the $ prompt, so this might be moot... but if you typed your question rather than cutting and pasting it, maybe you typed it incorrectly here and you were actually still in Python when you had the error.
$ ln -s /usr/local/Cellar/opencv/3.4.1_4/lib/python3.6/site-packages/cv2.cpython-36m-darwin.so /usr/local/lib/python3.6/site-packages/cv2.so

Installing node.js on CentOS 5

I'm pretty new to CentOS (5) and also node.js, but I already got an older version of node.js to work on my virtual server.
Now I'm trying to install a newer version, and I know that CentOS needs Python 2.4 while node needs 2.6 or newer, so I installed Python 2.7 using altinstall.
But even if I set an alias for Python that points to version 2.7 before running ./configure, I still get this error:
/root/node/wscript: error: Traceback (most recent call last):
File "/root/node/tools/wafadmin/Utils.py", line 274, in load_module
exec(compile(code, file_path, 'exec'), module.__dict__)
File "/root/node/wscript", line 222
"-pre" if node_is_release == "0" else ""
^
SyntaxError: invalid syntax
That's the content of ./configure:
#! /bin/sh
# v8 doesn't like ccache
if [ ! -z "`echo $CC | grep ccache`" ]; then
echo "Error: V8 doesn't like cache. Please set your CC env var to 'gcc'"
echo " (ba)sh: export CC=gcc"
exit 1
fi
CUR_DIR=$PWD
#possible relative path
WORKINGDIR=`dirname $0`
cd "$WORKINGDIR"
#abs path
WORKINGDIR=`pwd`
cd "$CUR_DIR"
"${WORKINGDIR}/tools/waf-light" --jobs=1 configure $*
exit $?"
And at the top of wscript there is the following line: "#!/usr/bin/env python". I also tried replacing that with something else, though I think it should work when using a Python alias
Any ideas what I need to do to get this to work?
Thanks!

I have python 2.7.3 'altinstalled' on Centos 5.x, with the binary named "/usr/local/bin/python2.7"
I compile and install nodejs v0.8.16 using:
PYTHON=/usr/local/bin/python2.7
export PYTHON
python2.7 configure && make && make install
running configure with python2.7 overrides the default python handling
creating a PYTHON env var allows make install to find the correct version of python
(I still had to identify and install missing development modules one by one before the install would succeed)

I changed the PATH in bash_profile to include the path to the desired version of python as follows:
vi ~/.bash_profile
replace PATH=$PATH:$HOME/bin
with PATH=/usr/local/python272/bin:$PATH:$HOME/bin
source ~/.bash_profile
./configure
make
It picks up the correct python version. No need to change wscript
I'm on CentOS 5.6, python 2.7.2 (installed in /usr/local/python272) and using node.js 0.4.12

I ran into this same exact problem. I wound up editing the wscript file and changed that line (222) from this:
"-pre" if node_is_release == "0" else ""
...to this:
""