I get a weird error when I try to build a conda package.
$ conda-build pkg2
....
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
....
The following specifications were found to be incompatible with your system:
- feature:/linux-64::__glibc==2.17=0
- feature:|#/linux-64::__glibc==2.17=0
- pkg1 -> __glibc[version='>=2.17,<3.0.a0']
Your installed version is: 2.17
It looks to me as if glibc 2.17 satisfies all three requirements, however conda thinks there is a conflict.
To clarify, pkg2 depends on another package that I have built locally, pkg1. pkg1 is a C++ library with a python interface, that depends on libfftw. I could not find libfftw on conda, so had to install it via yum on the build host, which runs CentOS7 with glibc 2.17, hence the dependency.
In pkg1 meta.yaml I have:
requirements:
build:
- sysroot_linux-64 >=2.17 [linux]
pkg2 meta.yaml:
requirements:
host:
- pkg1
run:
- pkg1
Thanks #merv for the fftw tip.
I strongly suspect that the issue was caused by a mixture of packages from conda-forge and Anaconda default channels in the same environment. According to conda developers, this is considered a bad practice. Once I have changed it around to using conda-forge only, the problems went away.
Related
I have a package written on pure python which does not require any platform specific build procedures. However, one of run dependencies is available only for linux-64 platform. I surely understand that I am not able to run my package on win-64 platform and just want to create a package for linux-64 platform on my windows pc. I'd be rather happy to not publish the windows package and then run conda convert -p linux-64.
But if I'll try to build the following meta.yaml on windows platform
package:
name: device
version: 1.0.0
source:
path: ../
build:
entry_points:
- Starter = device:main
script: "{{ PYTHON }} -m pip install . -vv"
requirements:
build:
- python
- setuptools
run:
- python>=3.10
- pytango>=9.3.6
- slsdetlib==6.1.2 # linux specific
- slsdet==6.1.2 # linux specific
- bidict>=0.22.1
- numpy>=1.24.1
I get a reasonable error message:
conda_build.exceptions.DependencyNeedsBuildingError: Unsatisfiable dependencies for platform win-64: {'slsdetlib==6.1.2', 'slsdet==6.1.2'}
Can I somehow define a target platform which differs from the platform I working on and create package for linux-64 only? I know that I can just build this package in linux virtual machine and I already do so but would like to have a more elegant way.
I have a Python script running in a Docker container on AWS Lambda. I'm using the recommended AWS image (public.ecr.aws/lambda/python:3.9), which comes with SQLite version 3.7.17 (from 2013!). When I test the container locally on my M1 Mac, I see this:
$ docker run --env-file .env --entrypoint bash -ti my-image
bash-4.2# uname -a
Linux e9ed14d35cbe 5.10.104-linuxkit #1 SMP PREEMPT Thu Mar 17 17:05:54 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
bash-4.2# sqlite3 --version
3.7.17 2013-05-20 00:56:22 118a3b35693b134d56ebd780123b7fd6f1497668
However, I use newer SQLite features, so I need to find a way to use a newer version of the library. The most straightforward solution would be to install a binary package as suggested in this answer. The docs say it should be as simple as installing using pip. Unfortunately, when I attempt to use this approach inside the Docker container, I get this:
bash-4.2# pip3 install pysqlite3-binary
ERROR: Could not find a version that satisfies the requirement pysqlite3-binary (from versions: none)
ERROR: No matching distribution found for pysqlite3-binary
And I get the same error when I attempt to install it outside the container using pipenv (which is what I'm actually using for package management):
🕙 01:08:24 ❯ pipenv install pysqlite3-binary
Installing pysqlite3-binary...
Error: An error occurred while installing pysqlite3-binary!
Error text:
ERROR: Could not find a version that satisfies the requirement pysqlite3-binary (from versions: none)
ERROR: No matching distribution found for pysqlite3-binary
✘ Installation Failed
Am I doing something wrong? And if not, how can I get a recent version of SQLite which Python can use in this container? Do I really need to use a separate build stage in the Dockerfile as suggested here and copy the rpm components into place as laid out here? That feels like a lot of work for something that many people presumably need to do all the time.
Update: I tried the rpm approach inside the container using version 3.26 from EPEL8 (IIUC) and it failed with a bunch of dependency errors like this:
bash-4.2# curl --output-dir /tmp -sO https://vault.centos.org/centos/8/BaseOS/aarch64/os/Packages/sqlite-3.26.0-15.el8.aarch64.rpm
bash-4.2# yum localinstall /tmp/sqlite-3.26.0-15.el8.aarch64.rpm
Loaded plugins: ovl
Examining /tmp/sqlite-3.26.0-15.el8.aarch64.rpm: sqlite-3.26.0-15.el8.aarch64
# etc.
--> Finished Dependency Resolution
Error: Package: sqlite-3.26.0-15.el8.aarch64 (/sqlite-3.26.0-15.el8.aarch64)
Requires: libc.so.6(GLIBC_2.28)(64bit)
# Plus 6 other package dependency errors
Error: Package: nss-softokn-3.67.0-3.amzn2.0.1.aarch64 (#amzn2-core)
Requires: libsqlite3.so.0()(64bit)
Removing: sqlite-3.7.17-8.amzn2.1.1.aarch64 (#amzn2-core)
libsqlite3.so.0()(64bit)
Updated By: sqlite-3.26.0-15.el8.aarch64 (/sqlite-3.26.0-15.el8.aarch64)
Not found
Obsoleted By: sqlite-3.26.0-15.el8.aarch64 (/sqlite-3.26.0-15.el8.aarch64)
Not found
Available: sqlite-3.7.17-8.amzn2.0.2.aarch64 (amzn2-core)
libsqlite3.so.0()(64bit)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
When I try --skip-broken, it just skips installing the 3.26 package altogether.
Update 2: I've tried downloading the Python 3.9 wheel from pysqlite3-binary manually. However, it looks like that project only produces wheels for x86_64, not the aarch64 platform which Lambda uses. (This is not correct, see answer.) So presumably that's why pip is not finding it.
The problem was that I was running Docker locally to do my testing, on an M1 Mac. Hence the aarch64 architecture. Lambda does allow you to use ARM, but thankfully it still defaults to x86_64. I confirmed that my Lambda function was running x86_64, which is what the binary wheel uses, so that's good:
So I needed to do three things:
Change my Pipfile to conditionally install the binary package only on x86_64:
pysqlite3-binary = { version = "*", platform_machine = "== 'x86_64'" }
Tweak the sqlite import, as described in the original answer:
try:
import pysqlite3 as sqlite3
except ModuleNotFoundError:
import sqlite3 # for local testing because pysqlite3-binary couldn't be installed on macos
print(f"{sqlite3.sqlite_version=}")
Set my Docker container to launch in x86 emulation mode locally.
$ DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build -t my-image .
$ DOCKER_DEFAULT_PLATFORM=linux/amd64 docker run -ti my-image
Et, voilà!
sqlite3.sqlite_version='3.39.2'
Using a Mac with M1 chip, I'm trying to install the following Bioconda packages:
cutadapt
trim-galore
samtools
bedtools.
htseq.
bowtie2.
deeptools.
macs2
I've been able to install picard and fastqc with no issues, but all others turn out one of two error messages:
PackagesNotFoundError: The following packages are not available from current channels:
or
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError:
I do have the correct channels though:
conda config --show channels
channels:
- conda-forge
- bioconda
- defaults
Not sure how to fix this
I had the same problem and fixed the issue like this:
Install Rosetta2 by running softwareupdate --install-rosetta
Download the x86-64 bit version of miniconda instead of the M1 version
I'm learning how to create conda packages, and I am having trouble with a package that will work locally, but that when downloaded from anaconda into a different server will return an error:
/lib64/libm.so.6: version `GLIBC_2.29' not found
The meta.yaml looks like the following:
package:
name: app
version: 2.4
source:
git_url: https://gitlab.com/user/repo.git
git_tag: v2.4
requirements:
build:
host:
run:
about:
home: https://gitlab.com/user/repo
license: GPL-3
license_family: GPL
summary: blabla.
The app is build with a simple build.sh script:
#!/bin/bash
set -x
echo $(pwd)
make
BIN=$PREFIX/bin
mkdir -p $BIN
cp app $BIN
I assumed that build: glibc >= 2.29 under requirements would make the job, but that results in an error when running conda build ..
How can I include GLIBC in the package? is that something meant to be done manually? from the package version I can download from anaconda, I can see other packages are downloaded as well (e.g. libgcc-ng) that I did not really mention in the meta.yaml or anywhere.
How can I include GLIBC in the package?
You can't, for reasons explained here.
Your best bet is to build on a system (or in a docker container) which has the lowest version of GLIBC you need to support (i.e. the version installed on the server(s) you will be running your package on).
Anaconda keeps telling me that cx_oracle needs python 3.4.
My conda version is 4.3.17
This is after I used the cx_Oracle-5.3-11g.win-amd64-py3.5-2.exe installer. which is supposed to be used for Python 3.5.
conda install -c anaconda cx_oracle=5.2
Fetching package metadata ...............
Solving package specifications: .
UnsatisfiableError: The following specifications were found to be in conflict:
- cx_oracle 5.2* -> python 3.4*
- python 3.5*
Use "conda info <package>" to see the dependencies for each package.
I'm doing something stupid, but I'm not sure what. yet.
ideas?
hints?
The error's telling you that the cx_oracle package you're trying to install depends on Python 3.4 b/c that's the currently available build of cx_oracle on the anaconda channel.
If you search for cx_oracle on anaconda.org, you'll see a list of available builds on several other channels, and can look for one that meets your reqs (namely v5.2, Python 3.5, win64). Doesn't look like there's one that meets that exact set of reqs, but there are a few v5.21 & v5.3 builds available. If that's close enough, then installing from say, the amodig channel may get you going: conda install -c amodig cx_oracle=5.2.1