We Keep Coding

Install newer version of sqlite3 on AWS Lambda for use with Python

I have a Python script running in a Docker container on AWS Lambda. I'm using the recommended AWS image (public.ecr.aws/lambda/python:3.9), which comes with SQLite version 3.7.17 (from 2013!). When I test the container locally on my M1 Mac, I see this:
$ docker run --env-file .env --entrypoint bash -ti my-image
bash-4.2# uname -a
Linux e9ed14d35cbe 5.10.104-linuxkit #1 SMP PREEMPT Thu Mar 17 17:05:54 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
bash-4.2# sqlite3 --version
3.7.17 2013-05-20 00:56:22 118a3b35693b134d56ebd780123b7fd6f1497668
However, I use newer SQLite features, so I need to find a way to use a newer version of the library. The most straightforward solution would be to install a binary package as suggested in this answer. The docs say it should be as simple as installing using pip. Unfortunately, when I attempt to use this approach inside the Docker container, I get this:
bash-4.2# pip3 install pysqlite3-binary
ERROR: Could not find a version that satisfies the requirement pysqlite3-binary (from versions: none)
ERROR: No matching distribution found for pysqlite3-binary
And I get the same error when I attempt to install it outside the container using pipenv (which is what I'm actually using for package management):
🕙 01:08:24 ❯ pipenv install pysqlite3-binary
Installing pysqlite3-binary...
Error: An error occurred while installing pysqlite3-binary!
Error text:
ERROR: Could not find a version that satisfies the requirement pysqlite3-binary (from versions: none)
ERROR: No matching distribution found for pysqlite3-binary
✘ Installation Failed
Am I doing something wrong? And if not, how can I get a recent version of SQLite which Python can use in this container? Do I really need to use a separate build stage in the Dockerfile as suggested here and copy the rpm components into place as laid out here? That feels like a lot of work for something that many people presumably need to do all the time.
Update: I tried the rpm approach inside the container using version 3.26 from EPEL8 (IIUC) and it failed with a bunch of dependency errors like this:
bash-4.2# curl --output-dir /tmp -sO https://vault.centos.org/centos/8/BaseOS/aarch64/os/Packages/sqlite-3.26.0-15.el8.aarch64.rpm
bash-4.2# yum localinstall /tmp/sqlite-3.26.0-15.el8.aarch64.rpm
Loaded plugins: ovl
Examining /tmp/sqlite-3.26.0-15.el8.aarch64.rpm: sqlite-3.26.0-15.el8.aarch64
# etc.
--> Finished Dependency Resolution
Error: Package: sqlite-3.26.0-15.el8.aarch64 (/sqlite-3.26.0-15.el8.aarch64)
Requires: libc.so.6(GLIBC_2.28)(64bit)
# Plus 6 other package dependency errors
Error: Package: nss-softokn-3.67.0-3.amzn2.0.1.aarch64 (#amzn2-core)
Requires: libsqlite3.so.0()(64bit)
Removing: sqlite-3.7.17-8.amzn2.1.1.aarch64 (#amzn2-core)
libsqlite3.so.0()(64bit)
Updated By: sqlite-3.26.0-15.el8.aarch64 (/sqlite-3.26.0-15.el8.aarch64)
Not found
Obsoleted By: sqlite-3.26.0-15.el8.aarch64 (/sqlite-3.26.0-15.el8.aarch64)
Not found
Available: sqlite-3.7.17-8.amzn2.0.2.aarch64 (amzn2-core)
libsqlite3.so.0()(64bit)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
When I try --skip-broken, it just skips installing the 3.26 package altogether.
Update 2: I've tried downloading the Python 3.9 wheel from pysqlite3-binary manually. However, it looks like that project only produces wheels for x86_64, not the aarch64 platform which Lambda uses. (This is not correct, see answer.) So presumably that's why pip is not finding it.

The problem was that I was running Docker locally to do my testing, on an M1 Mac. Hence the aarch64 architecture. Lambda does allow you to use ARM, but thankfully it still defaults to x86_64. I confirmed that my Lambda function was running x86_64, which is what the binary wheel uses, so that's good:
So I needed to do three things:
Change my Pipfile to conditionally install the binary package only on x86_64:
pysqlite3-binary = { version = "*", platform_machine = "== 'x86_64'" }
Tweak the sqlite import, as described in the original answer:
try:
import pysqlite3 as sqlite3
except ModuleNotFoundError:
import sqlite3 # for local testing because pysqlite3-binary couldn't be installed on macos
print(f"{sqlite3.sqlite_version=}")
Set my Docker container to launch in x86 emulation mode locally.
$ DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build -t my-image .
$ DOCKER_DEFAULT_PLATFORM=linux/amd64 docker run -ti my-image
Et, voilà!
sqlite3.sqlite_version='3.39.2'

How to install GDAL Python Bindings for Postgres.app

I installed the latest Postgre.app version Version 2.5.6 (139) Universal version for my new MacBook Pro M1. After setting up the CLI tools, I am able to access all gdal binaries that come bundled like gdalinfo.
However I also need to install the python bindings to use cli utilities like gdal2tiles.py. What's the right way to go about installing these so they work with the rest of gdal bundled with latest Postgress.app?
Current version bundled with Postgres.app is
➜ gdalinfo --version
GDAL 3.3.3, released 2021/10/25
Postgres.app installs GDAL binaries at:
/Applications/Postgres.app/Contents/Versions/latest/bin and currently has the following bundled:
clusterdb ecpg gdal_translate gdalinfo gdaltindex invproj pg_amcheck pg_ctl pg_resetwal pg_verifybackup proj vacuumdb
createdb gdal-config gdal_viewshed gdallocationinfo gdaltransform nearblack pg_archivecleanup pg_dump pg_restore pg_waldump psql vacuumlo
createuser gdal_contour gdaladdo gdalmanage gdalwarp ogr2ogr pg_basebackup pg_dumpall pg_rewind pgbench raster2pgsql
cs2cs gdal_create gdalbuildvrt gdalmdiminfo geod ogrinfo pg_checksums pg_isready pg_test_fsync pgsql2shp reindexdb
dropdb gdal_grid gdaldem gdalmdimtranslate initdb ogrtindex pg_config pg_receivewal pg_test_timing postgres shp2pgsql
dropuser gdal_rasterize gdalenhance gdalsrsinfo invgeod oid2name pg_controldata pg_recvlogical pg_upgrade postmaster testepsg
I realize there are plenty of ways to install gdal with python bindings using brew and pip, but I am looking to do this the right way while utilizing the core gdal that comes with Postgres.app
Any help is appreciated, Thanks!

Use pygdal, it is made for this exact purpose.
export GDALHOME="/Applications/Postgres.app/Contents/Versions/latest"
pip install pygdal=="`gdal-config --version`.*"

nrfutil - "ImportError: No module named main" on Nixos

I'm using the tool nrfutil which is implemented in Python. To be able to use it under NixOS I was using a default.nix file, that installed nrfutil into a venv. This worked for some time very well. (The last build on the build server using Nix within an alpine container could build the software I'm working on 11 days ago successfully.) When I do exactly the same things (i.e. restarting the CI server build without changes), the build fails now complaining about pip being incorrect:
$ nix-shell
New python executable in /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
Not overwriting existing python script /home/matthias/source/tbconnect/bootloader/.venv/bin/python (you must use /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7)
Installing pip, wheel...
done.
Traceback (most recent call last):
File "/home/matthias/source/tbconnect/bootloader/.venv/bin/pip", line 6, in <module>
from pip._internal.main import main
ImportError: No module named main
To me it seems that the module main should exist:
$ ls -l .venv/lib/python2.7/site-packages/pip/_internal/main.py
-rw-r--r-- 1 matthias matthias 1359 10月 15 12:27 .venv/lib/python2.7/site-packages/pip/_internal/main.py
I'm not very much into the Python environment, so I don't know any further. Has somebody any pointer for me where to continue debugging? How is Python resolving modules? Why doesn't it find the module, that seems to be present to me?
This is my default.nix that I use to install pip:
with import <nixpkgs> {};
with pkgs.python27Packages;
stdenv.mkDerivation {
name = "impurePythonEnv";
buildInputs = [
automake
autoconf
gcc-arm-embedded-7
# these packages are required for virtualenv and pip to work:
#
python27Full
python27Packages.virtualenv
python27Packages.pip
# the following packages are related to the dependencies of your python
# project.
# In this particular example the python modules listed in the
# requirements.txt require the following packages to be installed locally
# in order to compile any binary extensions they may require.
#
taglib
openssl
git
stdenv
zlib ];
src = null;
shellHook = ''
# set SOURCE_DATE_EPOCH so that we can use python wheels
SOURCE_DATE_EPOCH=$(date +%s)
virtualenv --no-setuptools .venv
export PATH=$PWD/.venv/bin:$PATH
#pip install nrfutil
pip help
# the following is required to build micro_ecc_lib_nrf52.a in the SDK
export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"
unset CC
'';
}
I replaced pip install nrfutil with pip help to make sure the problem is not the package I try to install itself.
I'm still using python 2.7 as the nrfutil still is not fit for Python 3.
Anyway replacing python27 with python37 did not change the error I get when trying to start pip.)
NixOS version used locally is 19.09. Nix in the CI docker container is nixos/nix:latest which is the nix package manager on Alpine Linux.
Update:
Actually it works when I replace the call to pip install nrfutil with python2.7 -m pip install nrfutil. This actually confuses me even more. python2.7 is exactly the binary that is in the shebang of pip:
[nix-shell:~/source/tbconnect/bootloader]$ type python2.7
python2.7 is /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
[nix-shell:~/source/tbconnect/bootloader]$ type pip
pip is /home/matthias/source/tbconnect/bootloader/.venv/bin/pip
[nix-shell:~/source/tbconnect/bootloader]$ head --lines 2 .venv/bin/pip
#!/home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
# -*- coding: utf-8 -*-
Update 2:
I found out that another way to fix the problem is to edit .venv/bin/pip. This script tried the following import:
from pip._internal.main import main
Which I think is the new module path starting with pip 19.3. But I still have pip 19.2. When I change this line to:
from pip._internal import main
Running pip by typing pip is working.
The thing is I have no idea why the pip script is trying to load the new module path while NixOS still has the old version of pip.
I also opened an issue for NixOS on GitHub: https://github.com/NixOS/nixpkgs/issues/71178

I got your shell derivation to work by dropping the Python27Packages.pip,
(nix-shell) 2d [azul:/tmp/lixo12333] $
>>> pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------------- -------
behave 1.2.6
Click 7.0
crcmod 1.7
ecdsa 0.13.3
enum34 1.1.6
future 0.18.2
intelhex 2.2.1
ipaddress 1.0.23
libusb1 1.7.1
linecache2 1.0.0
nrfutil 5.2.0
parse 1.12.1
parse-type 0.5.2
pc-ble-driver-py 0.11.4
piccata 1.0.1
pip 19.3.1
protobuf 3.10.0
pyserial 3.4
pyspinel 1.0.0a3
PyYAML 4.2b4
setuptools 41.6.0
six 1.12.0
tqdm 4.37.0
traceback2 1.4.0
virtualenv 16.4.3
wheel 0.33.6
wrapt 1.11.2
(nix-shell) 2d [azul:/tmp/lixo12333] $
and my default.nix
with import <nixpkgs> {};
with pkgs.python27Packages;
stdenv.mkDerivation {
name = "impurePythonEnv";
buildInputs = [
automake
autoconf
gcc-arm-embedded-7
# these packages are required for virtualenv and pip to work:
#
python27Full
python27Packages.virtualenv
# the following packages are related to the dependencies of your python
# project.
# In this particular example the python modules listed in the
# requirements.txt require the following packages to be installed locally
# in order to compile any binary extensions they may require.
#
taglib
openssl
git
stdenv
zlib ];
src = null;
shellHook = ''
# set SOURCE_DATE_EPOCH so that we can use python wheels
SOURCE_DATE_EPOCH=$(date +%s)
virtualenv .venv
export PATH=$PWD/.venv/bin:$PATH
pip install nrfutil
#pip help
# the following is required to build micro_ecc_lib_nrf52.a in the SDK
export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"
unset CC
'';
}

Psycopg2 Python SSL Support is not compiled in

I am trying to connect to my postgres database using psycopg2 with sslmode='required' param; however, I get the following error
psycopg2.OperationalError: sslmode value "require" invalid when SSL support is not compiled in
Heres a couple details about my system
Mac OS X El Capitan
Python 2.7
Installed psycopg2 via pip
Installed python via homebrew
Here is what I tried to do to fix the problem
brew uninstall python
which python still shows python living in /usr/local/bin/python, tried to uninstall this but couldnt. And heard that this is the python that the OS uses and should not be uninstalled anyways
brew install python --with-brewed-openssl --build-from-source
pip uninstall psycopg2
pip install psycopg2
After doing all of this, the exception still happens. I am running this python script via #!/usr/bin/env python Not sure if it matters, but that is a different directory than the one that which python shows

Since you're installing via pip, you should be using the most recent version of psycopg2 (2.6.1).
After a little digging through the code, it seems that the exception is being thrown in connection_int.c, which directly calls the postgresql-c-libraries to set up the db-connection. The call happens like so:
self->pgconn = pgconn = PQconnectStart(self->dsn);
Dprintf("conn_connect: new postgresql connection at %p", pgconn);
if (pgconn == NULL)
{
Dprintf("conn_connect: PQconnectStart(%s) FAILED", self->dsn);
PyErr_SetString(OperationalError, "PQconnectStart() failed");
return -1;
}
else if (PQstatus(pgconn) == CONNECTION_BAD)
{
Dprintf("conn_connect: PQconnectdb(%s) returned BAD", self->dsn);
PyErr_SetString(OperationalError, PQerrorMessage(pgconn));
return -1;
}
The keywords which were specified in your connect statement to psycopg2.connect() are being handled to that function and errors are returned as OperationalError exception.
The error is actually being generated directly in the postgresql-lib - you may want to check which version you are using, how it was built and, if possible, upgrade it to a version with SSL support or rebuilt it from source with SSL enabled.
The postgresql-docs also state that missing SSL support will raise an error, if the sslmode is set to require, verify-ca or verify-full. See here under sslmode for reference.
The postgres-website lists several ways to install postgres from binary packages, so you might choose one which suits your needs. I'm not familiar with OSX, so I don't have a recommendation what's best.
This question may also be helpful.
You also need to reinstall the psycopg2-module, be sure to use the newly installed lib when rebuilding it. Refer to the linked question (in short, you will need to place the path to pg_config which is included in your new installation to $PATH when running pip install psycopg2).

I had this same error, which turned out to be because I was using the Anaconda version of psycopg2. To fix it, I had adapt VictorF's solution from here and run:
conda uninstall psycopg2
sudo ln -s /Users/YOURUSERNAME/anaconda/lib/libssl.1.0.0.dylib /usr/local/lib
sudo ln -s /Users/YOURUSERNAME/anaconda/lib/libcrypto.1.0.0.dylib /usr/local/lib
pip install psycopg2
Then when you run conda list you'll see psycopg2 installed with <pip> in the far right column. After that, I just restarted Python and everything worked.

The error you are receiving is caused by a problem with Postgres itself, and not psycopg2.
psycopg2.OperationalError: sslmode value "require" invalid when SSL support is not compiled in
The above indicates that the version of Postgres that psycopg2 is built against does not have SSL support compiled in. When you attempt to connect to the running Posgres server with ssl=require it throws this error.
You don't mention how you installed Postgres but since you are using Homebrew for other things, I recommend you also install Postgres the same way:
$ brew update
$ brew install postgresql
The formula for postgresql shows that it depends on openssl and compiles with the --with-openssl flag set. It will also install the neccessary libpq headers. You may need to reinstall psycopg2 after this step to ensure it picks up the correct libraries/version.
Interestingly, there is a bug listed against conda which lists the same error that you report occurring when the conda version of psycopg2 — linked on a system with Homebrew postgres — was installed on a system without, suggesting missing SSL libraries can also trigger this.
I would suggest uninstalling any existing Postgres versions (including any installed via Homebrew) before reinstalling to minimise the risk of the wrong one being used.

As others have said, the error message looks to be coming from Postgres. You can verify this by typing: psql sslmode=require if it gives you a pgsql terminal then ssl works with postgres, if it errors then it doesn't
Try and remove postgres and reinstall with openssl support:
brew uninstall postgres
brew update
brew install postgres --with-openssl
Alternatively, and this is the way I'd suggest, there is a one click installer at http://postgresapp.com that might also make it easier to get it installed. Follow the instructions on the site to get it installed correctly.
When I did it on Yosemite it installed at ~/Library/Application\ Support/Postgres93/var
You'll also want to create a certificate (this could also be where the error is coming from) either from a registrar if this is going to be public facing in the slightest or self signed if it's for a dev/test environment.
openssl req -new -text -out server.req
openssl rsa -in privkey.pem -out server.key
rm privkey.pem
openssl req -x509 -in server.req -text -key server.key -out server.crt
chmod og-rwx server.key
Navigate to your config directory, by default it is: ~/Library/Application\ Support/Postgres93/var
Enable ssl support:
vim postgresql.conf
# change this:
# ssl = on
# to this:
ssl = on
Restart the app and then check ssl with psql "sslmode=require"
If that works then try it through your Python code. If it works with the code above, but not Python then it's definitely a Python code problem that needs to be worked through.

As I can not comment:
Adding to Brideau's answer that this only worked for me after changing my version of Postgres.
brew uninstall postgres
brew update
brew install postgres --with-openssl
Then run the code provided by Brideau and it should work.

If you're using v2.6.1 or v2.6.2 of psycopg2 (like me), the answer was a simple upgrade to v2.7. Reading the release notes for psycopg2, there was a minor fix for SSL albeit it doesn't look particularly relevant.
My setup was as follows:
Mac OS X El Capitan 10.11.6
psycopg2 2.6.2 installed via pip
PostgreSQL installed via Enterprise DB Installer
Running pip uninstall psycopg2 followed by pip install psycopg2 resolved matters.

Try to install psycopg2 from MacPorts
sudo port install py27-psycopg2

Install m2crypto on a virtualenv without system packages

I have created a virtual environment without the system packages with python's virtualenv in Ubuntu and installed m2crypto, but when I execute a shell and I try to import M2Crypto i get the following error:
ImportError: /home/imediava/.virtualenvs/myenv/local/lib/python2.7/site-
packages/M2Crypto/__m2crypto.so: undefined symbol: SSLv2_method
From outside the environment I run into the same problem unless from ubuntu I install python-m2crypto with apt-get. I know that I could create the environment with the system packages but I would prefer not to do it.
Is there anyway that I could create a virtual environment without the system packages and then install m2crypto with pip without running into the SSLv2_method?

There seems to be a regression bug from an earlier version of M2Crypto.
After placing M2Crypto's source in your virtualenv, you can try to patch it with the diff code below.
You do this by downloading the source code, untar it via:
tar -xzf M2Crypto-0.21.1.tar.gz
This should create the directory M2Crypto-0.21.1 which will contain the SWIG directory
In SWIG you'll find _ssl.i, which is the file to be patched. In the same directory create a file called _ssl.i.patch for example using the nano editor and paste the complete diff code listed below into it.
Next issue the patch _ssl.i _ssl.i.patch command to merge the patch into the code. (Afterwards you may remove the patch file if you want.)
Finally issue the commands:
python setup.py build
followed by:
python setup.py install
to install manually.
diff code:
--- SWIG/_ssl.i 2011-01-15 20:10:06.000000000 +0100
+++ SWIG/_ssl.i 2012-06-17 17:39:05.292769292 +0200
## -48,8 +48,10 ##
%rename(ssl_get_alert_desc_v) SSL_alert_desc_string_long;
extern const char *SSL_alert_desc_string_long(int);
+#ifndef OPENSSL_NO_SSL2
%rename(sslv2_method) SSLv2_method;
extern SSL_METHOD *SSLv2_method(void);
+#endif
%rename(sslv3_method) SSLv3_method;
extern SSL_METHOD *SSLv3_method(void);
%rename(sslv23_method) SSLv23_method;

You can install this lib in your global environment and then just copy from your global site-packages to virtualenv.

M2Crypto 0.22.3 (the current version in pypi) fixes this problem, so the simplest solution is now simply:
pip install --upgrade M2Crypto
M2Crypto 0.22.3 has been released from martinpaljak's github repository, rather than from the original M2Crypto repository.

I had the same problem with the current release (M2Crypto-0.22.5). The latest RC build worked for me.
pip install M2Crypto==0.22.6rc4

There is a patch slated for 0.26.1.
In the meantime, you can clone the repo, apply the patch, and install from source.
git clone https://gitlab.com/m2crypto/m2crypto.git
(
cd m2crypto
git checkout 0.25.1
curl 'https://gitlab.com/m2crypto/m2crypto/merge_requests/117.diff' | git apply -v
)
pip install -U m2crypto