We Keep Coding

Unable to setup "sphinx" on gitlab-ci

Unable to setup sphinx on gitlab-ci
I'm trying to setup and run sphnix on gitlab-ci. It's installed on my local machine and server works fine. But not at gitlab-ci
Here's a part of my gitlab-ci:
before_script:
- apt-get update -qq && apt-get install -yqq [......] python-pip
- pip install sphinx
# ..........
- bundle exec rake ts:index
- bundle exec rake ts:start
The output of installing sphinx:
$ pip install sphinx
Downloading/unpacking sphinx
[................]
Installing collected packages: sphinx, babel, [.............]
Compiling /tmp/pip-build-V2Tzes/Jinja2/jinja2/asyncfilters.py ...
File "/tmp/pip-build-V2Tzes/Jinja2/jinja2/asyncfilters.py", line 7
async def auto_to_seq(value):
^
SyntaxError: invalid syntax
Compiling /tmp/pip-build-V2Tzes/Jinja2/jinja2/asyncsupport.py ...
File "/tmp/pip-build-V2Tzes/Jinja2/jinja2/asyncsupport.py", line 22
async def concat_async(async_gen):
^
SyntaxError: invalid syntax
Running setup.py install for MarkupSafe
building 'markupsafe._speedups' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o
markupsafe/_speedups.c:12:20: fatal error: Python.h: No such file or directory
#include <Python.h>
^
compilation terminated.
==========================================================================
WARNING: The C extension could not be compiled, speedups are not enabled.
Failure information, if any, is above.
Retrying the build without the C extension now.
==========================================================================
WARNING: The C extension could not be compiled, speedups are not enabled.
Plain-Python installation succeeded.
==========================================================================
Successfully installed sphinx babel sphinxcontrib-websupport Jinja2 alabaster imagesize Pygments snowballstemmer docutils packaging typing pytz MarkupSafe pyparsing
Cleaning up...
As you can see, it's successeded.
However, when it gets to this point, it fails:
$ bundle exec rake ts:index
Generating configuration to /builds/my_user/my_ap123/config/test.sphinx.conf
sh: 1: indexer: not found
The Sphinx indexing command failed:
Command: indexer --config "/builds/my_user/my_ap123/config/test.sphinx.conf" --all
Status: 127
Output: See above
Why is that? How to fix it?

pip install sphinx did not fully succeed. during installation, it is complaining about missing Python.h
fatal error: Python.h: No such file or directory
You need to install a development package of python as well.
For python3x use
sudo apt-get install python3-dev
For python2x use
sudo apt-get install python-dev
You can add it to before_script
before_script:
- apt-get update -qq && apt-get install -yqq [......] python-pip python3-dev
- pip install sphinx
Update:
To install sphinx search engine using gitlab ci update your before_script as follow
before_script:
- add-apt-repository ppa:builds/sphinxsearch-rel22 sphinxsearch
- apt-get update -qq && apt-get install -yqq [......] python-pip python3-dev mysql-client unixodbc libpq5
- pip install sphinxsearch

portaudio.h: No such file or directory

I got the following error while trying to install pyaudio using pip3 in ubuntu 16.04:
Collecting pyaudio
Downloading PyAudio-0.2.11.tar.gz
Installing collected packages: pyaudio
Running setup.py install for pyaudio ... error
Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mxgvewdb/pyaudio/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-v55chjee-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.5
copying src/pyaudio.py -> build/lib.linux-x86_64-3.5
running build_ext
building '_portaudio' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c src/_portaudiomodule.c -o build/temp.linux-x86_64-3.5/src/_portaudiomodule.o
src/_portaudiomodule.c:29:23: fatal error: portaudio.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mxgvewdb/pyaudio/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-v55chjee-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-mxgvewdb/pyaudio/

This should help
$ sudo apt-get install portaudio19-dev python-pyaudio python3-pyaudio
Tested on Ubuntu 18.04, Python 3.7.2, pyaudio 0.2.11
Added 8th of Jan 2021:
On Ubuntu 20.04 you have to use:
$ sudo apt install portaudio19-dev python3-pyaudio

For Windows, here's what I've been going through...
There're several ways to install PortAudio
vcpkg, https://vcpkg.info/port/portaudio
winget, https://winget.run/pkg/intxcc/pyaudio
Or, manually build from source and install manually
download source, http://files.portaudio.com/download.html
build with cmake, with the matching compiler(msvc/mingw) and arch(x64/x86) to your Python
put "portaudio.h" into python-include directory, e.g. c:\Python38-32\include\
put "portaudio.lib into python-lib directory, e.g. c:\Python38-32\libs
Arrh!
the "min()" macro in pyaudio's soucefile _portaudiomodule.c is troublesome, comment it out manually
pip install pyaudio --no-clean
find _portaudiomodule.c in the temp directory and manually comment-out the "min()" macro e.g. c:\Temp\pip-install-yysb8bme\pyaudio_c92585\src\_portaudiomodule.c
pip install from that directorye.g. pip install c:\Temp\pip-install-yysb8bme\pyaudio_c92585\
Or,3rd party wheels
this answer said:
pip install pipwinpipwin install pyaudio

For MacOS (Intel/Apple Silicon):
Step 1:
brew install portaudio
Step 2:
brew --prefix portaudio
The output provided from the above command, will be required in Step 3, which will look something similar to /opt/homebrew/opt/portaudio
Step 3:
Create .pydistutils.cfg in your home directory
sudo nano ~/.pydistutils.cfg
then paste the following in the file, save and exit
[build_ext]
include_dirs=<PATH FROM STEP 2>/include/
library_dirs=<PATH FROM STEP 2>/lib/
Step 4:
pip3 install pyaudio

For Centos7/RHEL:
yum install portaudio-devel alsa-lib-devel portaudio
pip install pyaudio --user

Failed building wheel for spacy

I'm trying to install spacy by running pip install spacy for python version 3.6.1 but continuously i'm getting errors like below,how to get rid of this issue? previously i was having cl.exe not found error, after that i added visual studio path in environment variables where cl.exe exists.
Failed building wheel for spacy
Running setup.py clean for spacy
Running setup.py bdist_wheel for murmurhash ... error
Complete output from command c:\users\sh00428701\appdata\local\programs\python\python36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\SH0042~1\\AppData\\Local\\Temp\\pip-build-joi6voav\\murmurhash\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\SH0042~1\AppData\Local\Temp\tmpa6tzdkovpip-wheel- --python-tag cp36:
running bdist_wheel
running build
running build_py
----------------------------------------
Failed building wheel for murmurhash
Running setup.py clean for murmurhash
Running setup.py bdist_wheel for cymem ... error
Complete output from command c:\users\sh00428701\appdata\local\programs\python\python36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\SH0042~1\\AppData\\Local\\Temp\\pip-build-joi6voav\\cymem\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\SH0042~1\AppData\Local\Temp\tmpz7p6hkiwpip-wheel- --python-tag cp36:
----------------------------------------
Failed building wheel for cymem
Running setup.py clean for cymem
Running setup.py bdist_wheel for preshed ... error
Complete output from command c:\users\sh00428701\appdata\local\programs\python\python36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\SH0042~1\\AppData\\Local\\Temp\\pip-build-joi6voav\\preshed\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\SH0042~1\AppData\Local\Temp\tmpwppgmyp9pip-wheel- --python-tag cp36:
----------------------------------------
Failed building wheel for preshed
Running setup.py clean for preshed
Running setup.py bdist_wheel for thinc ... error
----------------------------------------
Failed building wheel for thinc
Running setup.py clean for thinc
Running setup.py bdist_wheel for ujson ... error
----------------------------------------
Failed building wheel for ujson
Running setup.py clean for ujson
Running setup.py bdist_wheel for cytoolz ... error
----------------------------------------
Failed building wheel for cytoolz
Running setup.py clean for cytoolz
Failed to build spacy murmurhash cymem preshed thinc ujson cytoolz
Installing collected packages: murmurhash, cymem, preshed, wrapt, tqdm, toolz, cytoolz, plac, pyreadline, dill, termcolor, pathlib, thinc, ujson, regex, spacy
Running setup.py install for murmurhash ... error
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\users\sh00428701\appdata\local\programs\python\python36\include -IC:\Users\SH0042~1\AppData\Local\Temp\pip-build-joi6voav\murmurhash\murmurhash\include -Ic:\users\sh00428701\appdata\local\programs\python\python36\include -Ic:\users\sh00428701\appdata\local\programs\python\python36\include /EHsc /Tpmurmurhash/mrmr.cpp /Fobuild\temp.win-amd64-3.6\Release\murmurhash/mrmr.obj /Ox /EHsc
mrmr.cpp
c1xx: fatal error C1083: Cannot open source file: 'murmurhash/mrmr.cpp': No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\cl.exe' failed with exit status 2
----------------------------------------
Command "c:\users\sh00428701\appdata\local\programs\python\python36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\SH0042~1\\AppData\\Local\\Temp\\pip-build-joi6voav\\murmurhash\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\SH0042~1\AppData\Local\Temp\pip-_j1cxej1-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\SH0042~1\AppData\Local\Temp\pip-build-joi6voav\murmurhash\

for me, pip install --no-cache-dir spacy worked

A couple thoughts:
Grab the various wheel files you need from http://www.lfd.uci.edu/~gohlke/pythonlibs/#spacy and install with pip install x.whl y.whl etc.
Upgrade your version of cpp build tools to 2017 and try pip again

I installed these packages, then it works:
sudo apt-get install python-dev
sudo apt-get install python3-dev
sudo apt-get install libevent-dev

This worked for me:
# Note, use sudo. I'm in a docker image, so i dont need it.
# install dev libs
apt-get install python-dev -y && \
apt-get install python3-dev -y && \
apt-get install libevent-dev -y && \
# install new gcc
apt-get update && \
apt-get install build-essential software-properties-common -y && \
add-apt-repository ppa:ubuntu-toolchain-r/test -y && \
apt-get update && \
apt-get install gcc-snapshot -y && \
apt-get update && \
apt-get install gcc-6 g++-6 -y && \
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-6 && \
apt-get install gcc-4.8 g++-4.8 -y && \
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.8;
see https://gist.github.com/application2000/73fd6f4bf1be6600a2cf9f56315a2d91

conda install -c conda-forge spacy worked for me. I have windows 10 and Python 3.5.3 :: Anaconda custom (64-bit)

If you are on Ubuntu, just do this
sudo apt-get install build-essential python-dev git
Now install spacy by
pip install -U spacy

Download and install from the Gihub source. Here is the link: https://github.com/explosion/spaCy
Use these commands:
python -m pip install -U pip venv # update pip & virtualenv
git clone https://github.com/explosion/spaCy # clone spaCy
cd spaCy # navigate into directory
venv .env # create environment in .env
source .env/bin/activate # activate virtual environment
export PYTHONPATH=`pwd` # set Python path to spaCy directory
pip install -r requirements.txt # install all requirements
python setup.py build_ext --inplace # compile spaCy
Feel free to use only last two commands if not using virtual environment. Follow the official Documentation here

If you are using it with manually installed python3.6 and trying to install in a 3.6 environment then you maybe missing python3.6-dev
sudo apt-get install python3.6-dev
also maybe
sudo apt-get install gcc
This is an answer to particular case.

yum -y groupinstall development
WORKED FOR ME

Spacy requires 64-bit python.
Uninstall 32-bit.
Install 64-bit python.
Try spacy installation again.

For anyone looking for an answer for spacy 2.X with Python 3.8 in Ubuntu, I had the same problem. I'm using venv. As the other answers were not enough, I write the solution which solved my problem.
install the required libraries as mentioned in other answers:
sudo apt-get install python3-dev libevent-dev build-essential gcc g++
pip install setuptools wheel
Update pip and install cython
pip install -U pip
pip install cython
I suspect that updating pip to the newest version solved my problem.

At first, install those packages and try to install spacy
sudo apt update
sudo apt install python3-dev
sudo apt install libpython3-dev
sudo apt install python3-devel
sudo apt-get install libevent-dev
sudo apt update

Error while upgrading PyAudio to latest version

I have PyAudio (0.2.7) installed in my system.
When I tried to upgrade using
sudo pip install pyaudio --upgrade
I'm getting this.
Downloading/unpacking pyaudio from https://pypi.python.org/packages/44/39/f3b9b16d32cc8588fa06bb2e813cea35363f5502117cd6cc7f619b556d9f/PyAudio-0.2.10.tar.gz#md5=b65354d997fa00a9166f8dbcb83075e6
Downloading PyAudio-0.2.10.tar.gz (287kB): 287kB downloaded
Running setup.py (path:/tmp/pip_build_root/pyaudio/setup.py) egg_info for package pyaudio
Installing collected packages: pyaudio
Found existing installation: PyAudio 0.2.7
Not uninstalling PyAudio at /usr/lib/python3/dist-packages, owned by OS
.......
.......
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.4m -c src/_portaudiomodule.c -o build/temp.linux-x86_64-3.4/src/_portaudiomodule.o
src/_portaudiomodule.c:29:23: fatal error: portaudio.h: No such file or directory
#include "portaudio.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Can't roll back PyAudio; was not uninstalled
Cleaning up...
Command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/pyaudio/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ih3i9qsm-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_root/pyaudio
Storing debug log for failure in /home/jobin/.pip/pip.log
I'm using elementary os 14.04
What is the problem here?

Thanks #Cristian Ciupitu for the advice to install portaudio-dev.
But portaudio-dev did not installed with sudo apt-get install portaudio-dev in my case.
I got
E: Unable to locate package portaudio-dev
How this was solved
Downloaded the .tgz of portaudio file from here
Then Extract the downloaded file.
cd to the extracted folder.
Then ./configure && make
Now do sudo make install
Then upgrade pyaudio by sudo pip install pyaudio --upgrade
Now pyaudio was successfully installed.

run in terminal:
$ sudo apt install python3-pyaudio

This worked for me
sudo apt-get install portaudio19-dev python-pyaudio
then do this
pip install PyAudio

You need to install the portaudio-dev package, for example by running:
sudo apt-get install portaudio-dev

pip install pyaudio --upgrade
pip install pyaudio --user

gevent/libevent.h:9:19: fatal error: event.h: No such file or directory

I was trying to work on Pyladies website on my local folder. I cloned the repo, (https://github.com/pyladies/pyladies) ! and created the virtual environment. However when I do the pip install -r requirements, I am getting this error
Installing collected packages: gevent, greenlet
Running setup.py install for gevent
building 'gevent.core' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -I/opt/local/include -fPIC -I/usr/include/python2.7 -c gevent/core.c -o build/temp.linux-i686-2.7/gevent/core.o
In file included from gevent/core.c:253:0:
gevent/libevent.h:9:19: fatal error: event.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
Complete output from command /home/akoppad/virt/pyladies/bin/python -c "import setuptools;__file__='/home/akoppad/virt/pyladies/build/gevent/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-4MSIGy-record/install-record.txt --install-headers /home/akoppad/virt/pyladies/include/site/python2.7:
running install
running build
running build_py
running build_ext
building 'gevent.core' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -I/opt/local/include -fPIC -I/usr/include/python2.7 -c gevent/core.c -o build/temp.linux-i686-2.7/gevent/core.o
In file included from gevent/core.c:253:0:
gevent/libevent.h:9:19: fatal error: event.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command /home/akoppad/virt/pyladies/bin/python -c "import setuptools;__file__='/home/akoppad/virt/pyladies/build/gevent/setup.py'; exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-4MSIGy-record/install-record.txt --install-headers /home/akoppad/virt/pyladies/include/site/python2.7 failed with error code 1 in /home/akoppad/virt/pyladies/build/gevent
Storing complete log in /home/akoppad/.pip/pip.log.
I tried doing this,
sudo port install libevent
CFLAGS="-I /opt/local/include -L /opt/local/lib" pip install gevent
It says port command not found.
I am not sure how to proceed with this. Thanks!

I had the same problem and just as the other answer suggested I had to install "libevent". It's apparently not called "libevent-devel" anymore (apt-get couldn't find it) but doing:
$ apt-cache search libevent
listed a bunch of available packages.
$ apt-get install libevent-dev
worked for me.

I think you just forget to install the "libevent" in the environment. If you are on a OSX machine, please try to install brew here http://mxcl.github.io/homebrew/ and use brew install libevent to install the dependency. If you are on an ubuntu machine, you can try apt-get to install the corresponding library.

I had this issue while trying to
pip install Flask-Sockets
and
sudo apt-get install libevent-dev
worked for me

yum install libevent-*
This command works for me because I am on centos.

I ran into this error on MacOS 10.10.1 (Yosemite) running pip install. The solution for me was to add the path to libevent which pip was missing. These are C files, and CFLAGS and DFLAGS are bash environment variables that let pip know where to find these files. On my system, libevent is found at /usr/local/include/event2.
$ CFLAGS=-I/usr/local/include/event2 DFLAGS=-L/usr/local/lib pip install -r requires.txt