We Keep Coding

make file target to check for installed dependencies

I have a makefile where I have targets that depend on having some external clients installed (python3, libxml2, etc).
Here is my makefile
.PHONY: test install-packages mac-setup checkenv target help
EXTERNALS = python3 pip3 xmllint pytest pipenv
P := $(foreach exec,$(EXTERNALS),$(if $(shell which $(exec)),missing,$(warning "===>>>WARNING: No required `$(exec)` in PATH, run `make mac-setup` + `make install-packages` <<<===")))
test: ## run all tests in test directory
pipenv run pytest -v --ignore=path payload_files .
install-packages: ##install python packages listed in Pipfile
pipenv install
mac-setup: ## setup mac for testing
brew install libxml2
brew install python3
brew install pipenv
# see https://github.mycompany.com/ea/ea_test_player_unified/blob/master/run-feature.sh
help:
#grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
.DEFAULT_GOAL := help
Notice the line
P := $(foreach exec,$(EXTERNALS),$(if $(shell which $(exec)),missing,$(warning "===>>>WARNING: No required `$(exec)` in PATH, run `make mac-setup` + `make install-packages` <<<===")))
This checks for the binaries required. This works.... however I would rather have a checkenv target that performs this and errors so I can attach it too specific targets like test instead of printing out a WARNING that might be overlooked.
Want:
checkenv: # error if which ${binary} fails or *even better* if if binary --version doesn't return the right version: python3 pip3 xmllint pytest pipenv
I tried various techniques that I found around the web including stackoverflow.... but most use the technique I am using above that don't use a make target or just check for one binary. I tried building a loop through an array of binaries but just couldn't get the syntax correct due to make being a PITA :)
Any suggestions?
Note I'm a python newbie, task is to rewrite some jmeter tests in python....so if you have any thoughts on the above approach feel free to share.
Thanks,
Phil

Don't see what the problem is. It looks very straightforward to me, as make allows using multiple targets on the same line:
EXTERNALS := python3 pip3 xmllint pytest pipenv
python3_version := Python 3.7.3
pip3_version := ...
...
.PHONY: checkenv $(EXTERNALS)
checkenv: $(EXTERNALS)
$(EXTERNALS):
if [ "`$# --version`" != "$($#_version)" ]; then echo "$# check failed"; false; fi

In NixOS, how can I install an environment with the Python packages SpaCy, pandas, and jenks-natural-breaks?

I'm very new to NixOS, so please forgive my ignorance. I'm just trying to set up a Python environment---any kind of environment---for developing with SpaCy, the SpaCy data, pandas, and jenks-natural-breaks. Here's what I've tried so far:
pypi2nix -V "3.6" -E gcc -E libffi -e spacy -e pandas -e numpy --default-overrides, followed by nix-build -r requirements.nix -A packages. I've managed to get the first command to work, but the second fails with Could not find a version that satisfies the requirement python-dateutil>=2.5.0 (from pandas==0.23.4)
Writing a default.nix that looks like this: with import <nixpkgs> {};
python36.withPackages (ps: with ps; [ spacy pandas scikitlearn ]). This fails with collision between /nix/store/9szpqlby9kvgif3mfm7fsw4y119an2kb-python3.6-msgpack-0.5.6/lib/python3.6/site-packages/msgpack/_packer.cpython-36m-x86_64-linux-gnu.so and /nix/store/d08bgskfbrp6dh70h3agv16s212zdn6w-python3.6-msgpack-python-0.5.6/lib/python3.6/site-packages/msgpack/_packer.cpython-36m-x86_64-linux-gnu.so
Making a new virtualenv, and then running pip install on all these packages. Scikit-learn fails to install, with fish: Unknown command 'ar rc build/temp.linux-x86_64-3.6/liblibsvm-skl.a build/temp.linux-x86_64-3.6/sklearn/svm/src/libsvm/libsvm_template.o'
I guess ideally I'd like to install this environment with nix, so that I could enter it with nix-shell, and so other environments could reuse the same python packages. How would I go about doing that? Especially since some of these packages exist in nixpkgs, and others are only on Pypi.

Caveat
I had trouble with jenks-natural-breaks to the tune of
nix-shell ❯ poetry run python -c 'import jenks_natural_breaks'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/matt/2022/12/28-2/.venv/lib/python3.10/site-packages/jenks_natural_breaks/__init__.py", line 5, in <module>
from ._jenks_matrices import ffi as _ffi
ModuleNotFoundError: No module named 'jenks_natural_breaks._jenks_matrices'
So I'm going to use jenkspy which appears to be a bit livelier. If that doesn't scratch your itch, I'd contact the maintainer of jenks-natural-breaks for guidance
Flakes
you said:
so other environments could reuse the same python packages
Which makes me think that a flake.nix is what you need. What's cool about flakes is that you can define an environment that has spacy, pandas, and jenkspy with one flake. And then you (or somebody else) might say:
I want an env like Jonathan's, except I also want sympy
and rather than copying your env and making tweaks, they can declare your env as a build input and write a flake.nix with their modifications--which can be further modified by others.
One could imagine a sort of family-tree of environments, so you just need to pick the one that suits your task. The python community has not yet converged on this vision.
Poetry
Poetry will treat you like you're trying to publish a library when all you asked for is an environment, but a library's dependencies are pretty much an environment so there's nothing wrong with having an empty package and just using poetry as an environment factory.
Bonus: if you decide to publish a library after all, you're ready.
The Setup
nix flakes thinks in terms of git repo's, so we'll start with one:
$ git init
Then create a file called flake.nix. Usually I end up with poetry handling 90% of the python stuff, but both pandas and spacy are in that 10% that has dependencies which link to system libraries. So we ask nix to install them so that when poetry tries to install them in the nix develop shell, it has what it needs.
{
description = "Jonathan's awesome env";
inputs = {
nixpkgs.url = "github:nixos/nixpkgs";
};
outputs = { self, nixpkgs, flake-utils }: (flake-utils.lib.eachSystem [
"x86_64-linux"
"x86_64-darwin"
"aarch64-linux"
"aarch64-darwin"
] (system:
let
pkgs = nixpkgs.legacyPackages.${system};
in
rec {
packages.jonathansenv = pkgs.poetry2nix.mkPoetryApplication {
projectDir = ./.;
};
defaultPackage = packages.jonathansenv;
devShell = pkgs.mkShell {
buildInputs = [
pkgs.poetry
pkgs.python310Packages.pandas
pkgs.python310Packages.spacy
];
};
}));
}
Now we let git know about the flake and enter the environment:
❯ git add flake.nix
❯ nix develop
$
Then we initialize the poetry project. I've found that poetry, installed by nix, is kind of odd about which python it uses by default, so we'll set it explicitly
$ poetry init # follow prompts
$ poetry env use $(which python)
$ poetry run python --version
Python 3.10.9 # declared in the flake.nix
At this point, we should have a pyproject.toml:
[tool.poetry]
name = "jonathansenv"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.10"
jenkspy = "^0.3.2"
spacy = "^3.4.4"
pandas = "^1.5.2"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Usage
Now we create the venv that poetry will use, and run a command that depends on these.
$ poetry install
$ poetry run python -c 'import jenkspy, spacy, pandas'
You can also have poetry put you in a shell:
$ poetry shell
(venv)$ python -c 'import jenkspy, spacy, pandas'
It's kind of awkward to do so though, because we're two subshells deep and any shell customizations that we have the grandparent shell are not available. So I recommend using direnv, to enter the dev shell whenever I navigate to that directory and then just use poetry run ... to run commands in the environment.
Publishing the env
In addition to running nix develop with the flake.nix in your current dir, you can also do nix develop /local/path/to/repo or develop nix develop github:/githubuser/githubproject to achieve the same result.
To demonstrate the github example, I have pushed the files referenced above here. So you ought to be able to run this from any linux shell with nix installed:
❯ nix develop github:/MatrixManAtYrService/nix-flake-pandas-spacy
$ poetry install
$ poetry run python -c 'import jenkspy, spacy, pandas'
I say "ought" because if I run that command on a mac it complains about linux-headers-5.19.16 being unsupported on x86_64-darwin.
Presumably there's a way to write the flake (or fix a package) so that it doesn't insist on building linux stuff on a mac, but until I figure it out I'm afraid that this is only a partial answer.

Can't install gcloud on Amazon Linux : invalid syntax

I'm trying to install gcloud on my EC2 server running Amazon Linux 4.14.47-56.37 64bits, in interactive mode running the following command :
curl https://sdk.cloud.google.com | bash
The files download correctly, but the install then fails with the following Traceback :
File "/home/ec2-user/google-cloud-sdk/bin/bootstrapping/install.py", line 12, in <module>
import bootstrapping
File "/home/ec2-user/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 32, in <module>
import setup # pylint:disable=g-import-not-at-top
File "/home/ec2-user/google-cloud-sdk/bin/bootstrapping/setup.py", line 55, in <module>
from googlecloudsdk.core import properties
File "/home/ec2-user/google-cloud-sdk/lib/googlecloudsdk/core/properties.py", line 291
self.__sections = {section.name: section for section in sections}
^
SyntaxError: invalid syntax
Any idea why this for is causing issues?
I am running python 2.7 (2.7.14) as recommended by Google.

On top of python 2.7 installed on the "python" command, I also had python 2.6 installed on the "python2" command.
Uninstalling python 2.6 solved the issue, Google Cloud install went through without issue on the next try.

You locally update the install.sh file to use python2.7 instead of python2. This worked for me

In case someone meets this problem again, and if you do not want to uninstall python 2.6 or to modify install.sh, here is a procedure that worked for me on an ec2 instance :
open a new terminal and type :
curl https://sdk.cloud.google.com | bash
The install should not work as CLOUDSDK_PYTHON=python2 by default, which leads to Python 2.6. To correct this, enter this command :
export CLOUDSDK_PYTHON=python
this links CLOUDSDK_PYTHON to Python 3.6
then
bash google-cloud-sdk/install.sh
and finally
./google-cloud-sdk/bin/gcloud init
Once this is done, you can use any gcloud or gsutil command by doing this :
./google-cloud-sdk/bin/gcloud <your gcloud command>
./google-cloud-sdk/bin/gsutil <your gsutil command>
For example, in order to copy a folder from a gs bucket to your ec2 instance :
./google-cloud-sdk/bin/gsutil -m cp -r gs://<bucket_name>/path/to/folder /path/on/your/ec2/instance
the -m option allows fast transfer (I could transfer with an average speed of 81 MiB/s with this option)
Hope this helps !

Setup yum repo:
# Update YUM with Cloud SDK repo information:
sudo tee -a /etc/yum.repos.d/google-cloud-sdk.repo << EOM
[google-cloud-sdk]
name=Google Cloud SDK
baseurl=https://packages.cloud.google.com/yum/repos/cloud-sdk-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOM
# The indentation for the 2nd line of gpgkey is important.
Install yum downloader and download the rpm, then install with "--nodeps":
yum install yum-utils
yumdownloader google-cloud-sdk-231.0.0-1.el7.noarch
mv 23873bd6e8459ba6e70e96eb8f03f6ac03cd707ce3c80baa8264c714e030c915-google-cloud-sdk-231.0.0-1.el7.noarch.rpm /usr/local/src/google-cloud-sdk-231.0.0-1.el7.noarch
rpm -ivh --nodeps /usr/local/src/google-cloud-sdk-231.0.0-1.el7.noarch

This solved issue:
export CLOUDSDK_PYTHON=python

git executable not found in python

I was trying to clone a git repo with access key, but when I am trying to run it, It throws an exception saying git executable not found.
But i have installed git and the in_it.py shows correct path "C:\Program Files\Git\bin" Also I have installed gitpython to use the library in python
here's my code...
import git
git.Git("D:/madhav/myrep/").clone("#github.com:myrepo/scripts")
========= and it throws the following exception =================
Traceback (most recent call last): File
"C:\Users\1096506\Desktop\gitclone.py", line 1, in <module>
from git import Repo File "C:\Users\1096506\AppData\Local\Programs\Python\Python36-32\lib\site-packages\git\__init__.py",
line 84, in <module>
refresh() File "C:\Users\1096506\AppData\Local\Programs\Python\Python36-32\lib\site-packages\git\__init__.py",
line 73, in refresh
if not Git.refresh(path=path): File "C:\Users\1096506\AppData\Local\Programs\Python\Python36-32\lib\site-packages\git\cmd.py",
line 293, in refresh
raise ImportError(err) ImportError: Bad git executable. The git executable must be specified in one of the following ways:
- be included in your $PATH
- be set via $GIT_PYTHON_GIT_EXECUTABLE
- explicitly set via git.refresh()
All git commands will error until this is rectified.
This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
- quiet|q|silence|s|none|n|0: for no warning or exception
- warn|w|warning|1: for a printed warning
- error|e|raise|r|2: for a raised exception
Example:
export GIT_PYTHON_REFRESH=quiet

I had the same issue. What I did is:
I went to: System Properties -> Environment Variables
On System Variables Section I clicked Path, Edit and Move Up.
Environment Variables
Edit and Moved Up two places from the bottom

Error occurs because git is not in the path. So its not able to import git module.
Couple of ways to resolve it.
As suggested above adding the git binary path to environment variable path.
If git is not being used directly in the module and its only a dependent module import thats throwing this exception before importing git we could add
os.environ["GIT_PYTHON_REFRESH"] = "quiet"
and import git after this line, this would suppress the error caused due to git import

Had the same problem got it to work thanks to Muthukumaran. Just make Muthukumaran answer more clear.
Follow these steps:
import os
os.environ["GIT_PYTHON_REFRESH"] = "quiet"
import git

See if you have installed Git in the OS.
If not install git first this will solve your error.
Centos
sudo yum -y install git
Ubuntu/Debian
sudo apt-get install git
Mac Os
sudo brew install git
This solved my problem.

Make sure you're not in an inaccessible directory on *nix, such as when you've just been root and then done a su username
you may still be in root's home folder and that will trigger this error (assuming you have the correct environment variables set, and have sourced the .profile or .bashrc etc with source ~/.bashrc )
which git
/usr/bin/git
I was getting this error even after setting the environment:
in ~/.bashrc
# for bench
PATH=$PATH:/usr/bin/git
export PATH
GIT_PYTHON_GIT_EXECUTABLE=/usr/bin/git
export GIT_PYTHON_GIT_EXECUTABLE
cd
and it's working
$ bench --version
WARN: Command not being executed in bench directory
5.3.0

I came across similar problem recently and installing git followed by restarting Windows Powershell CommandLine solved the problem. May it helps.

For those who are using a Lambda layer with it. It worked adding as the comment above says just adding GIT_PYTHON_REFRESH=quiet as an environment variable.

Execute GIT_PYTHON_REFRESH=quiet in your terminal and then try to run the code.

problem compiling libjingle

I downloaded and installed libjingle-0.5.2.zip, and according to the README also downloaded and installed swtoolkit.0.9.1.zip, scons-local-2.1.0.alpha.20101125.tar.gz, and expat-2.0.1.tar.gz, and got nrtp by cvs download. After overwriting my Makefile twice, attempting to follow the rather poorly-written README, I came up with the following Makefile that almost works:
# First, make sure the SCONS_DIR environment variable is set correctly.
SCONS_DIR ?= /usr/src/scons-local/scons-local-2.1.0.alpha.20101125/
#SCONS_DIR ?= /usr/src/scons-local/
export
default: build
# Second, run talk/third_party/expat-2.0.1/configure...
talk/third_party/expat-2.0.1/Makefile:
cd talk/third_party/expat-2.0.1 && ./configure
# ...and talk/third_party/srtp/configure.
talk/third_party/srtp/Makefile:
cd talk/third_party/srtp && ./configure
# Third, go to the talk/ directory and run $path_to_swtoolkit/hammer.sh. Run
# $path_to_swtoolkit/hammer.sh --help for information on how to build for
# different modes.
build: talk/third_party/expat-2.0.1/Makefile talk/third_party/srtp/Makefile
cd talk && ../../swtoolkit/hammer.sh
help:
../swtoolkit/hammer.sh --help
However, make gives me the following errors:
jcomeau#intrepid:/usr/src/libjingle-0.5.2$ make
cd talk && ../../swtoolkit/hammer.sh
*** Error loading site_init file './../../swtoolkit/site_scons/site_init.py':
AttributeError: 'Dir' object has no attribute 'endswith':
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 1338:
_exec_main(parser, values)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 1302:
_main(parser)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 929:
_load_site_scons_dir(d.path, options.site_dir)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 719:
exec fp in site_m
File "./../../swtoolkit/site_scons/site_init.py", line 455:
SiteInitMain()
File "./../../swtoolkit/site_scons/site_init.py", line 451:
SCons.Node.FS.get_default_fs().SConstruct_dir, None)
File "/usr/src/scons-local/scons-local-2.1.0.alpha.20101125/SCons/Script/Main.py", line 677:
site_dir = os.path.join(topdir, site_dir_name)
File "/usr/lib/python2.6/posixpath.py", line 67:
elif path == '' or path.endswith('/'):
make: *** [build] Error 2
I'm guessing that something new (a 'Dir' object being where a POSIX path string is expected) in one of the packages is breaking the build process, but which one? There are just too many layers of cruft here for me to follow. For sure I could just keep trying older packages, particularly for swtoolkit and scons, but if anyone here has successfully compiled libjingle and could prod me in the right direction, I'd appreciate it.

I'm not familiar with the project, but think I have a fix to get you past that point. You need to cast those Dir instances using str() in swtoolkit/site_scons/site_init.py. That way they can safely be evaluated by path.endswith('/'). Odd that such an issue would exist for very long in the main part of the build infrastructure:
Line 330:
SCons.Script.Main._load_site_scons_dir(
str(SCons.Node.FS.get_default_fs().SConstruct_dir), site_dir)
Line 450:
SCons.Script.Main._load_site_scons_dir(
str(SCons.Node.FS.get_default_fs().SConstruct_dir), None)

I did following to build libjingle :
Building LibJingle for Linux
How to Build
Libjingle is built with swtoolkit ( http://code.google.com/p/swtoolkit/), which
is a set of extensions to the open-source SCons build tool ( http://www.scons.org).
First, install Python 2.4 or later from http://www.python.org/.
Please note that since swtoolkit only works with Python 2.x, you will
not be able to use Python 3.x.
Second, install the stand alone scons-local package 2.0.0 or later from
http://www.scons.org/download.php and set an environment variable,
SCONS_DIR, to point to the directory containing SCons, for example,
/src/libjingle/scons-local/scons-local-2.0.0.final.0/.
Third, install swtoolkit from http://code.google.com/p/swtoolkit/.
Finally, Libjingle depends on two open-source projects, expat and srtp.
Download expat from http://sourceforge.net/projects/expat/ to
talk/third_party/expat-2.0.1/. Follow the instructions at
http://sourceforge.net/projects/srtp/develop to download latest srtp to
talk/third_party/srtp. Note that srtp-1.4.4 does not work since it misses
the extensions used by Libjingle.
If you put expat or srtp in a different directory, you need to edit
talk/libjingle.scons correspondingly.
2.1 Build Libjingle under Linux or OS X
First, make sure the SCONS_DIR environment variable is set correctly.
Second, run talk/third_party/expat-2.0.1/configure and
talk/third_party/srtp/configure.
Third, go to the talk/ directory and run $path_to_swtoolkit/hammer.sh. Run
$path_to_swtoolkit/hammer.sh --help for information on how to build for
different modes.
Other than above given steps, See following as reference
Set SCONS_DIR Path
export SCONS_DIR=/home/esumit/libjingle/libjingle-0.5.2/talk/third_party/scons-local/scons-local-2.0.1
Install libasound2-dev Lib to compile libJingle, otherwise you will encounter errors.
sudo apt-get install libasound2-dev
Download SRTP using the following command. If it asks for a passowrd, just hit Enter.
cvs -z3 -d:pserver:anonymous#srtp.cvs.sourceforge.net:/cvsroot/srtp co -P srtp
Possible components in LibJingle Directory
libjingle-0.5.2/talk/third_party$ ls
expat-2.0.1 libudev scons-local srtp swtoolkit
Execute following command to build LibJingle
libjingle-0.5.2/talk$ ./third_party/swtoolkit/hammer.sh