Apologies for newbie question. I've tried all of the answers from other questions here but the don't pay off either.
Dockerizing my python/django app with Postgres is proving... daunting. I'm getting the error "Error: pg_config executable not found." consistently when it starts working through my requirements.txt
Here's the Dockerfile:
FROM python:3.8.3-slim-buster
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
ADD ./ /code/
...and my requirements.txt...
asgiref==3.3.1
Django==3.1.4
psycopg2-binary==2.7.4
pytz==2020.5
sqlparse==0.4.1
django-compressor>=2.2
django-libsass>=0.7
and docker-compose.yml
v ersion: "3.9"
services:
db:
image: postgres
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
volumes:
- pgdata:/var/lib/postgresql/data
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
volumes:
pgdata:
When I run docker-compose up --build I'm getting this error over and over:
Step 6/7 : RUN pip install -r requirements.txt
---> Running in e0fd67d2d935
Collecting asgiref==3.3.1
Downloading asgiref-3.3.1-py3-none-any.whl (19 kB)
Collecting Django==3.1.4
Downloading Django-3.1.4-py3-none-any.whl (7.8 MB)
Collecting psycopg2-binary==2.7.4
Downloading psycopg2-binary-2.7.4.tar.gz (426 kB)
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-fha1c65p/psycopg2-binary/setup.py'"'"'; __file__='"'"'/tmp/pip-install-fha1c65p/psycopg2-binary/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-ewxlxmh6
cwd: /tmp/pip-install-fha1c65p/psycopg2-binary/
Complete output (23 lines):
running egg_info
creating /tmp/pip-pip-egg-info-ewxlxmh6/psycopg2_binary.egg-info
writing /tmp/pip-pip-egg-info-ewxlxmh6/psycopg2_binary.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-ewxlxmh6/psycopg2_binary.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-pip-egg-info-ewxlxmh6/psycopg2_binary.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-ewxlxmh6/psycopg2_binary.egg-info/SOURCES.txt'
Error: pg_config executable not found.
Ultimately, the answer was seemingly found in a couple of things in my Dockerfile, but... ultimately... downgrading from Python 3.8 to 3.7 unlocked everything.
FROM python:3.7
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN apt-get update
RUN apt-get install -y postgresql
RUN apt-get install libpq-dev gcc
RUN export PATH=/usr/lib/postgresql/X.Y/bin/:$PATH
RUN apt-get install -y python3-dev
RUN apt-get install -y python3-psycopg2
RUN pip3 install -r requirements.txt
ADD ./ /code/
Currently building the image:
FROM python:3.7-slim-stretch
WORKDIR /root/forstack-host
COPY requirements.txt /root/requirements.txt
RUN apt-get update && apt-get install -y libgomp1 gcc
RUN apt-get install -y libpq-dev
RUN apt-get install net-tools
RUN apt-get install -y libhdf5-serial-dev hdf5-tools
RUN python3 -m pip install --no-cache-dir -U pip && \
python3 -m pip install --no-cache-dir -r /root/requirements.txt
When installing tables=3.4.4 I get the error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-91revz2s/tables/setup.py'"'"'; __file__='"'"'/tmp/pip-install-91revz2s/tables/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-91revz2s/tables/pip-egg-info
cwd: /tmp/pip-install-91revz2s/tables/
Complete output (12 lines):
/tmp/H5close9pyniq85.c: In function ‘main’:
/tmp/H5close9pyniq85.c:2:5: warning: implicit declaration of function ‘H5close’ [-Wimplicit-function-declaration]
H5close();
^~~~~~~
/usr/bin/ld: cannot find -lhdf5
collect2: error: ld returned 1 exit status
* Using Python 3.7.5 (default, Oct 19 2019, 00:03:48)
* USE_PKGCONFIG: True
.. ERROR:: Could not find a local HDF5 installation.
You may need to explicitly state where your local HDF5 headers and
library can be found by setting the ``HDF5_DIR`` environment
variable or by using the ``--hdf5`` command-line option.
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Using RUN apt-get install -y libhdf5-serial-dev hdf5-tools doesn't seem to fix this error and I can't seem to set the HDF5_DIR env. Is there a working 3.7 image with a fix for this hdf5 issue?
you may try add this ARG to your Dockerfile before installing:
ARG HDF5_DIR=PATH_TO_YOUR_HDF5 # normally should be in /usr/local or /opt/local
you may also need to install build-dep
I am trying to create a docker container for my python application (this is the first time I am trying to use docker). I looked at the online tutorials and created a DockerFile as follows:
FROM python:3.6-alpine
COPY . /app
WORKDIR /app
RUN apk --update add --no-cache \
lapack-dev \
gcc \
freetype-dev
# Install dependencies
RUN apk add --no-cache --virtual .build-deps \
gfortran \
musl-dev \
g++
RUN pip3 install -r requirements.txt
RUN python3 setup.py install
RUN apk del .build-deps
ENTRYPOINT python3 testapp.py
My project requirements are:
numpy==1.13.3
Cython==0.28.2
nibabel==2.2.1
scipy==1.0.0
I build the docker file as: docker build -t myimg .
So, the docker file progresses but scipy fails to build with the following error:
Collecting numpy==1.13.3 (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/bf/2d/005e45738ab07a26e621c9c12dc97381f372e06678adf7dc3356a69b5960/numpy-1.13.3.zip (5.0MB)
Collecting Cython==0.28.2 (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/79/9d/dea8c5181cdb77d32e20a44dd5346b0e4bac23c4858f2f66ad64bbcf4de8/Cython-0.28.2.tar.gz (1.9MB)
Collecting nibabel==2.2.1 (from -r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/d7/de/1d96fd0b118c9047bf35f02090db8ef8fd3927dfce635f09a6f7d5b572e6/nibabel-2.2.1.zip (4.2MB)
Collecting scipy==1.0.0 (from -r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/d0/73/76fc6ea21818eed0de8dd38e1e9586725578864169a2b31acdeffb9131c8/scipy-1.0.0.tar.gz (15.2MB)
Collecting six>=1.3 (from nibabel==2.2.1->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Building wheels for collected packages: numpy, Cython, nibabel, scipy
Running setup.py bdist_wheel for numpy: started
Running setup.py bdist_wheel for numpy: still running...
Running setup.py bdist_wheel for numpy: still running...
Running setup.py bdist_wheel for numpy: still running...
Running setup.py bdist_wheel for numpy: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/b6/10/65/189b772e73b4505109d5a1e6671b07e65797023718777295e0
Running setup.py bdist_wheel for Cython: started
Running setup.py bdist_wheel for Cython: still running...
Running setup.py bdist_wheel for Cython: still running...
Running setup.py bdist_wheel for Cython: still running...
Running setup.py bdist_wheel for Cython: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/6f/24/5d/def09ad0aed8ba26186f2a38070906f70ab4b2287bf64d4414
Running setup.py bdist_wheel for nibabel: started
Running setup.py bdist_wheel for nibabel: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/46/50/8d/bcb0b8f7c030da5bac1752fbe9cc375cbf5725fa93ba79ad84
Running setup.py bdist_wheel for scipy: started
Running setup.py bdist_wheel for scipy: finished with status 'error'
Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-boosbyfg/scipy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-cczhwdqj --python-tag cp36:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-boosbyfg/scipy/setup.py", line 418, in <module>
setup_package()
File "/tmp/pip-install-boosbyfg/scipy/setup.py", line 398, in setup_package
from numpy.distutils.core import setup
ModuleNotFoundError: No module named 'numpy'
Not sure why it is having issue finding numpy as it was installed as part of the requirements?
Because to build scipy wheel you need to have numpy installed. However, only numpy wheel build was complete by the time pip attempts to build scipy wheel.
You will have to install the dependencies first. There are multiple ways to do this.
1) Use a shell script like the one below, copy it and RUN it instead of RUN pip install -r requirements.txt:
#!/bin/sh
while read module; do
pip install $module
done < requirements.txt
2) Install scipy in a seperate RUN command.
3) apk add py-numpy#community as discussed in this answer.
I am building a Django implementation with docker. I'm using the python container as base, but in my requirements file I have the below:-
boto==2.43.0
boto3==1.4.4
botocore==1.5.55
ciscoconfparse==1.2.47
Django==1.11.4
django-appconf==1.0.2
django-auth-ldap==1.2.10
django-dbtemplates==2.0
django-debug-toolbar==1.7
easy-thumbnails==2.3
easysnmp==0.2.4
ipaddress==1.0.18
Jinja2==2.9.5
mysqlclient-1.3.10
netmiko==1.2.8
O365==0.9.5
orionsdk==0.0.6
paramiko==2.1.2
python-dateutil==2.6.0
python-ldap==2.4.32
pytz==2016.10
pyOpenSSL==17.2.0
sqlparse==0.2.3
urllib3==1.21.1
joblib==0.11
some of these have dependencies that fail when using the python container on its own, for example...
#include <net-snmp/net-snmp-config.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for easysnmp
...
...
#include "lber.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for pyldap
...
...
In file included from Modules/LDAPObject.c:9:0:
Modules/errors.h:8:18: fatal error: lber.h: No such file or directory
#include "lber.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-aufmftap/pyldap/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-fjdz1te2-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-aufmftap/pyldap/
how do I install OS dependencies for python modules? within a container? what os does the python container run on for me to get the right dependencies?
Thanks
Assuming you are using the python:3.6 image which is currently the latest, you need to install some additional libraries.
Dockerfile example (the pip libraries are in the requirements.txt file):
FROM python:3.6
RUN apt-get update -y \
&& apt-get install libsasl2-dev python-dev libldap2-dev libssl-dev libsnmp-dev
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
Related question: I can't install python-ldap
I had to update the run command to get it to work
RUN apt-get update -y
RUN apt-get install libsasl2-dev python-dev libldap2-dev libssl-dev libsnmp-dev -y
I'm trying to install spacy by running pip install spacy for python version 3.6.1 but continuously i'm getting errors like below,how to get rid of this issue? previously i was having cl.exe not found error, after that i added visual studio path in environment variables where cl.exe exists.
Failed building wheel for spacy
Running setup.py clean for spacy
Running setup.py bdist_wheel for murmurhash ... error
Complete output from command c:\users\sh00428701\appdata\local\programs\python\python36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\SH0042~1\\AppData\\Local\\Temp\\pip-build-joi6voav\\murmurhash\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\SH0042~1\AppData\Local\Temp\tmpa6tzdkovpip-wheel- --python-tag cp36:
running bdist_wheel
running build
running build_py
----------------------------------------
Failed building wheel for murmurhash
Running setup.py clean for murmurhash
Running setup.py bdist_wheel for cymem ... error
Complete output from command c:\users\sh00428701\appdata\local\programs\python\python36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\SH0042~1\\AppData\\Local\\Temp\\pip-build-joi6voav\\cymem\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\SH0042~1\AppData\Local\Temp\tmpz7p6hkiwpip-wheel- --python-tag cp36:
----------------------------------------
Failed building wheel for cymem
Running setup.py clean for cymem
Running setup.py bdist_wheel for preshed ... error
Complete output from command c:\users\sh00428701\appdata\local\programs\python\python36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\SH0042~1\\AppData\\Local\\Temp\\pip-build-joi6voav\\preshed\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\SH0042~1\AppData\Local\Temp\tmpwppgmyp9pip-wheel- --python-tag cp36:
----------------------------------------
Failed building wheel for preshed
Running setup.py clean for preshed
Running setup.py bdist_wheel for thinc ... error
----------------------------------------
Failed building wheel for thinc
Running setup.py clean for thinc
Running setup.py bdist_wheel for ujson ... error
----------------------------------------
Failed building wheel for ujson
Running setup.py clean for ujson
Running setup.py bdist_wheel for cytoolz ... error
----------------------------------------
Failed building wheel for cytoolz
Running setup.py clean for cytoolz
Failed to build spacy murmurhash cymem preshed thinc ujson cytoolz
Installing collected packages: murmurhash, cymem, preshed, wrapt, tqdm, toolz, cytoolz, plac, pyreadline, dill, termcolor, pathlib, thinc, ujson, regex, spacy
Running setup.py install for murmurhash ... error
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\users\sh00428701\appdata\local\programs\python\python36\include -IC:\Users\SH0042~1\AppData\Local\Temp\pip-build-joi6voav\murmurhash\murmurhash\include -Ic:\users\sh00428701\appdata\local\programs\python\python36\include -Ic:\users\sh00428701\appdata\local\programs\python\python36\include /EHsc /Tpmurmurhash/mrmr.cpp /Fobuild\temp.win-amd64-3.6\Release\murmurhash/mrmr.obj /Ox /EHsc
mrmr.cpp
c1xx: fatal error C1083: Cannot open source file: 'murmurhash/mrmr.cpp': No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\cl.exe' failed with exit status 2
----------------------------------------
Command "c:\users\sh00428701\appdata\local\programs\python\python36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\SH0042~1\\AppData\\Local\\Temp\\pip-build-joi6voav\\murmurhash\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\SH0042~1\AppData\Local\Temp\pip-_j1cxej1-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\SH0042~1\AppData\Local\Temp\pip-build-joi6voav\murmurhash\
for me, pip install --no-cache-dir spacy worked
A couple thoughts:
Grab the various wheel files you need from http://www.lfd.uci.edu/~gohlke/pythonlibs/#spacy and install with pip install x.whl y.whl etc.
Upgrade your version of cpp build tools to 2017 and try pip again
I installed these packages, then it works:
sudo apt-get install python-dev
sudo apt-get install python3-dev
sudo apt-get install libevent-dev
This worked for me:
# Note, use sudo. I'm in a docker image, so i dont need it.
# install dev libs
apt-get install python-dev -y && \
apt-get install python3-dev -y && \
apt-get install libevent-dev -y && \
# install new gcc
apt-get update && \
apt-get install build-essential software-properties-common -y && \
add-apt-repository ppa:ubuntu-toolchain-r/test -y && \
apt-get update && \
apt-get install gcc-snapshot -y && \
apt-get update && \
apt-get install gcc-6 g++-6 -y && \
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-6 && \
apt-get install gcc-4.8 g++-4.8 -y && \
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.8;
see https://gist.github.com/application2000/73fd6f4bf1be6600a2cf9f56315a2d91
conda install -c conda-forge spacy worked for me. I have windows 10 and Python 3.5.3 :: Anaconda custom (64-bit)
If you are on Ubuntu, just do this
sudo apt-get install build-essential python-dev git
Now install spacy by
pip install -U spacy
Download and install from the Gihub source. Here is the link: https://github.com/explosion/spaCy
Use these commands:
python -m pip install -U pip venv # update pip & virtualenv
git clone https://github.com/explosion/spaCy # clone spaCy
cd spaCy # navigate into directory
venv .env # create environment in .env
source .env/bin/activate # activate virtual environment
export PYTHONPATH=`pwd` # set Python path to spaCy directory
pip install -r requirements.txt # install all requirements
python setup.py build_ext --inplace # compile spaCy
Feel free to use only last two commands if not using virtual environment. Follow the official Documentation here
If you are using it with manually installed python3.6 and trying to install in a 3.6 environment then you maybe missing python3.6-dev
sudo apt-get install python3.6-dev
also maybe
sudo apt-get install gcc
This is an answer to particular case.
yum -y groupinstall development
WORKED FOR ME
Spacy requires 64-bit python.
Uninstall 32-bit.
Install 64-bit python.
Try spacy installation again.
For anyone looking for an answer for spacy 2.X with Python 3.8 in Ubuntu, I had the same problem. I'm using venv. As the other answers were not enough, I write the solution which solved my problem.
install the required libraries as mentioned in other answers:
sudo apt-get install python3-dev libevent-dev build-essential gcc g++
pip install setuptools wheel
Update pip and install cython
pip install -U pip
pip install cython
I suspect that updating pip to the newest version solved my problem.
At first, install those packages and try to install spacy
sudo apt update
sudo apt install python3-dev
sudo apt install libpython3-dev
sudo apt install python3-devel
sudo apt-get install libevent-dev
sudo apt update