Repeatedly failing to install scrapy and lxml - python

I'd previously used Anaconda to handle python, but I'm and start working with virtual environments.
I set up virtualenv and virtualenvwrapper, and have been trying to add modules, specifically scrapy and lxml, for a project I want to try.
Each time I pip install, I hit an error.
For scrapy:
File "/home/philip/Envs/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1003, in run_setup
raise DistutilsError("Setup script exited with %s" % (v.args[0],))
distutils.errors.DistutilsError: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
---------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /home/philip/Envs/venv/build/cryptography
Storing debug log for failure in /home/philip/.pip/pip.log
For lxml:
In file included from src/lxml/lxml.etree.c:346:0:
/home/philip/Envs/venv/build/lxml/src/lxml/includes/etree_defs.h:9:31: fatal error: libxml/xmlversion.h: No such file or directory
include "libxml/xmlversion.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Cleaning up... Command /home/philip/Envs/venv/bin/python -c "import setuptools, tokenize;__file__='/home/philip/Envs/venv/build/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-zIsPdl-record/install-record.txt
--single-version-externally-managed --compile --install-headers /home/philip/Envs/venv/include/site/python2.7 failed with error code 1 in /home/philip/Envs/venv/build/lxml Storing debug log for failure in /home/philip/.pip/pip.log
I tried to install it following scrapy's documentation, but scrapy was still not listed when I called for python's installed modules.
Any ideas? Thanks--really appreciate it!
I'm on Ubuntu 13.10 if it matters. Other modules I've tried have installed fine (though I've only gone for a handful).

I had the same problem in Ubuntu 14.04. I've solved it with the instructions of the page linked by #jdigital and the openssl-dev library pointed by #user3115915. Just to help others:
sudo apt-get install libxslt1-dev libxslt1.1 libxml2-dev libxml2 libssl-dev
sudo pip install scrapy

In my case, I solve the problem installing all the libraries that Manuel mention plus the extra library: libffi-dev
https://askubuntu.com/questions/499714/error-installing-scrapy-in-virtualenv-using-pip

Updated from #Mario C. and #Manuel,
Here is commands:
sudo apt-get install libxslt1-dev libxslt1.1 libxml2-dev libxml2 libssl-dev libffi-dev
sudo pip install scrapy

Related

Installing requirements.txt throws an error on Django

The project was working perfectly until recently when I created a model for my database and tried to pip install requirement.txt. I have done everything I know including modifying my settings.py to add the missing library, reinstalling django and psycopg2, all to no avail. Here's my requirement.txt:
tweepy==3.8.0
django
gunicorn
django-heroku
The error i'm getting:
ld: library not found for -lssl
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/Users/apple/Downloads/font_text/python-getting-started/getting-started/bin/python3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/tn/843ff_ns4j5ddf0fyc998v6c0000gn/T/pip-install-phi9wroh/psycopg2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/tn/843ff_ns4j5ddf0fyc998v6c0000gn/T/pip-record-sc3y24vo/install-record.txt --single-version-externally-managed --compile --install-headers /Users/apple/###/###/PROJECTFOLDER/getting-started/include/site/python3.7/psycopg2" failed with error code 1 in /private/var/folders/tn/843ff_ns4j5ddf0fyc998v6c0000gn/T/pip-install-phi9wroh/psycopg2/
Based on the error message, you need to install the openssl-devel package on your system.
After long hours of searching and experimenting, I finally was able to solve the problem. But I'm still investigating on what really could have caused the error. Anyways, here's what fixed it
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
pip install psycopg2
I found it here: error installing psycopg2, library not found for -lssl
You need to reinstall gcc , gcc-c++ and dependencies.
For python 2.7
$ sudo yum -y install gcc gcc-c++ kernel-devel
$ sudo yum -y install python-devel libxslt-devel libffi-devel openssl-devel
$ pip install "your python packet"
For python 3.4
$ sudo apt-get install python3-dev
$ pip install "your python packet"
Hope this will help.

Docker - Python dependencies for installing modules via pip

I am building a Django implementation with docker. I'm using the python container as base, but in my requirements file I have the below:-
boto==2.43.0
boto3==1.4.4
botocore==1.5.55
ciscoconfparse==1.2.47
Django==1.11.4
django-appconf==1.0.2
django-auth-ldap==1.2.10
django-dbtemplates==2.0
django-debug-toolbar==1.7
easy-thumbnails==2.3
easysnmp==0.2.4
ipaddress==1.0.18
Jinja2==2.9.5
mysqlclient-1.3.10
netmiko==1.2.8
O365==0.9.5
orionsdk==0.0.6
paramiko==2.1.2
python-dateutil==2.6.0
python-ldap==2.4.32
pytz==2016.10
pyOpenSSL==17.2.0
sqlparse==0.2.3
urllib3==1.21.1
joblib==0.11
some of these have dependencies that fail when using the python container on its own, for example...
#include <net-snmp/net-snmp-config.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for easysnmp
...
...
#include "lber.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for pyldap
...
...
In file included from Modules/LDAPObject.c:9:0:
Modules/errors.h:8:18: fatal error: lber.h: No such file or directory
#include "lber.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-aufmftap/pyldap/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-fjdz1te2-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-aufmftap/pyldap/
how do I install OS dependencies for python modules? within a container? what os does the python container run on for me to get the right dependencies?
Thanks
Assuming you are using the python:3.6 image which is currently the latest, you need to install some additional libraries.
Dockerfile example (the pip libraries are in the requirements.txt file):
FROM python:3.6
RUN apt-get update -y \
&& apt-get install libsasl2-dev python-dev libldap2-dev libssl-dev libsnmp-dev
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
Related question: I can't install python-ldap
I had to update the run command to get it to work
RUN apt-get update -y
RUN apt-get install libsasl2-dev python-dev libldap2-dev libssl-dev libsnmp-dev -y

Error installing module with pip on Ubuntu 15.10

I have trouble installing modules using pip, for example
sudo -H pip3 install kivy
returns this error
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-b3rwh_e6/kivy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-bev6v2mx-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-b3rwh_e6/kivy/
I am fairly new to python and to programming in general so I'm probably doing something wrong, it's probably worth mentioning that I have python2.7 and python3.5 installed, and installing the packages python3.5-dev and build-essential didn't fix it.
Edit: build log:
https://drive.google.com/file/d/0B5e3GD1lfCbUZ19LU3RENzY4MU0/view?usp=sharing
Packages which include C extensions require compilation and so might require both gcc and python-dev (and in some cases gcc-c++).
run:
sudo apt-get install gcc python-dev
prior to pip installing your package.
Have you tried to install
sudo apt-get install python-dev
for python2.7 ?
From the official site
https://kivy.org/docs/installation/installation-linux.html
for python 2
sudo apt-get install python-kivy
for python 3
sudo apt-get install python3-kivy

Compiling Cx-Freeze under Ubuntu

For the entire day I have been attempting to compile cx-Freeze under Ubuntu 14.04 and had no luck. So I gave up and decided to ask experts here.
What I have
Ubuntu 14.04
Python 3.4
python-dev, python3-dev, python3.4-dev installed (I know this common issue)
Sources of cx-Freeze 4.3.3
I tried two ways:
install from the sources
install by pip
Install from the sources
sudo python3 setup.py install
What I got
a lot of
MyPath/cx_Freeze-4.3.3/source/bases/Console.c:24: undefined reference to `PyErr_Print'
MyPath/cx_Freeze-4.3.3/source/bases/Console.c:24: undefined reference to `Py_FatalError'
and then
collect2: error: ld returned 1 exit status
error: command 'i686-linux-gnu-gcc' failed with exit status 1
Install by pip
sudo pip3 install cx-Freeze
What I got
collect2: error: ld returned 1 exit status
error: command 'i686-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/cx-Freeze/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-c954v7x6-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_root/cx-Freeze
Storing debug log for failure in /home/grimel/.pip/pip.log
and in pip.log
Exception information:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 283, in run
requirement_set.install(install_options, global_options, root=options.root_path)
File "/usr/lib/python3/dist-packages/pip/req.py", line 1435, in install
requirement.install(install_options, global_options, *args, **kwargs)
File "/usr/lib/python3/dist-packages/pip/req.py", line 706, in install
cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
File "/usr/lib/python3/dist-packages/pip/util.py", line 697, in call_subprocess
% (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/cx-Freeze/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-c954v7x6-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_root/cx-Freeze
So, I expect you to help me with this issue and gonna be very thankful:)
In setup.py string
if not vars.get("Py_ENABLE_SHARED", 0):
replace by
if True:
Thanks to Thomas K
From cx_freeze/issues
Download
You need to download the source code
For python 3.3 and 3.4:
sudo apt-get install python3-dev
sudo apt-get install libssl-dev
Open setup.py and change the line
if not vars.get("Py_ENABLE_SHARED", 0):
to
if True:
python3 setup.py build
sudo python3 setup.py install
For python 2.7:
sudo apt-get install python-dev
sudo apt-get install libssl-dev
Open setup.py and change the line
if not vars.get("Py_ENABLE_SHARED", 0):
to
if True:
python setup.py build
sudo python setup.py install
GriMel's answer worked for me. Until cx_freeze releases an update for this, I'm including a set of steps that you can do to get this working using GriMels' solution.
# create and activate virtualenv (as desired)
virtualenv envs/test_cxfreeze
. ./envs/test_cxfreeze/bin/activate
# download cxfreeze; do not install yet
mkdir src/
pip install --download=./src/ cx-freeze
tar zxvf ./src/cx_Freeze-4.3.4.tar.gz -C ./src/
# fix bug in setup.py
vim src/cx_Freeze-4.3.4/setup.py
84c84
< if True:
---
> if not vars.get("Py_ENABLE_SHARED", 0):
# install cxfreeze
pip install ./src/cx_Freeze-4.3.4/
I've also opened up a bitbucket issue with what appears to be the source repo.
https://bitbucket.org/anthony_tuininga/cx_freeze/issues/153/cx_freeze-434-compile-error-on-ubuntu-1404
Looks like you might have a typo
change
sudo pip3 install cx-Freeze
to
sudo pip3 install cx_Freeze

Can't install discount with pip: error: command 'cc' failed with exit status 1

I am running into many problems when trying to install the discount package on osx 10.9 Mavericks. I'm working with the django framework.
My steps are (while having a virtualenv activated):
pip install discount
Then I get:
.
.
.
1 error generated.
error: command 'cc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /Users/KaeserMic/Sites/2013/Duotones/naturkostbar/env/bin/python -c "import setuptools;__file__='/Users/KaeserMic/Sites/2013/Duotones/naturkostbar/env/build/discount/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/4b/smwv2y_s32z044brp6t__ssh0000gp/T/pip-65i7x3-record/install-record.txt --single-version-externally-managed --install-headers /Users/KaeserMic/Sites/2013/Duotones/naturkostbar/env/include/site/python2.7 failed with error code 1 in /Users/KaeserMic/Sites/2013/Duotones/naturkostbar/env/build/discount
Storing complete log in /Users/KaeserMic/.pip/pip.log
Here is the complete log:
https://gist.github.com/MichaelKaeser/7302829
Things I did:
updating python
updating pip
updating virtualenv
updating homebrew
Any ideas how to solve this issue?
Thanks in advance.
Try:
sudo ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install discount
This seems to be an issue with LLVM in XCode 5.1 as mentioned here: Can't install mysql gem on OS X
If you enter gcc or cc in the terminal, will it output an error? Do you have Xcode installed?
xcode-select....
If so, Mac Os Mavericks should alert you to install Xcode Command Line Tools.

Categories