We Keep Coding

Symbol not found: _sqlite3_enable_load_extension - sqlite installed via homebrew

Symptom: In my Django app, when I call from pysqlite2._sqlite import * I get the traceback Symbol not found: _sqlite3_enable_load_extension when
Background:
I've installed python using homebrew (python 2.7.13), which auto installed sqlite
I am running macOS 10.12.3 with Command Line Tools macOS 10.12, Xcode 8.2.1
I've installed pysqlite using pip (pysqlite 2.8.3)
I have tried
brew uninstall sqlite and brew uninstall python and reinstalling
Adding these to my .bash_profile
export PATH="$(brew --prefix sqlite)/bin:$PATH"
LDFLAGS="-L/usr/local/opt/sqlite/lib"
CPPFLAGS="-I/usr/local/opt/sqlite/include"
export PKG_CONFIG_PATH=“/usr/local/opt/sqlite/lib/pkgconfig”
python -c "import sqlite3" doesn't return any errors
Gist of traceback: https://gist.github.com/xwchen/e9704fa34f0463d2117fe9fbb37922a1

Copied answer from here (https://github.com/Homebrew/homebrew-core/pull/3134). If you're like me, you probably installed sqlite after you install python3. Anyway, if anyone stumbles upon this question and needs an answer...
First remove SQLite, python and python3 if installed via homebrew
brew uninstall --force python
brew uninstall --force python3
brew uninstall --force sqlite
This removes all copies.
Then make sure Xcode 8 or later is installed
Important
Re-install command line tools via
xcode-select --install
sudo xcode-select --reset
and finally install command line tools via pkg file found here.
Look for command line tools in search. (I've downloaded "Command Line Tools (macOS 10.12) for Xcode 8.dmg") Then open DMG and install the pkg found there.
Now install SQLite and then python and python 3 if needed:
brew install sqlite
brew install python
brew install python3
Run brew install python --verbose and make sure there are no warnings, if none then issues should be resolved.
Don't forget to do
pip install setuptools --upgrade && pip3 install setuptools --upgrade

It may happen because it's not able to locate the .so library. As explained here:
On OS X, homebrew installs sqlite3 as “keg-only” because otherwise it interferes with the system sqlite3. This means that libsqlite3.dylib ends up in /usr/local/opt/sqlite/lib, not in /usr/local/lib
So you can simply do this:
export DYLD_LIBRARY_PATH=/usr/local/opt/sqlite/lib:/usr/lib
Then you should be able to import sqlite3.

In my conda environment simply running conda install sqlite fixed it.

unable to install JQ via PIP

I am trying to install JQ via PIP in python.
pip install jq
I am getting following error.
Failed building wheel for jq
I am facing same issue while install pyjq.
pip install pyjq
Failed building wheel for pyjq

On Ubuntu 18.04, I needed to apt-get install autoconf libtool before my pip install pyjq would complete.

It doesn't appear that jq supports Windows; it says it requires gcc & libtool, which generally means a Unix-like environment.

Like you I had a difficult time installing jq
In my attempts I had many various errors including the failed building wheel that you are getting. I assume that the problem was on my end and not that the host was temporarily down. My setup: python 2 & 3, Jupyter, brew in addition to pip. The problem was likely due to some lack of consistency in package links, brew doctor helped me identify which links were broken then brew link/unlink/overwrite.
At anyrate I was only successful after brew uninstall jq, fixing all links, then updating brew and rebooting my system (perhaps some dependency was occupied?).
Then and only then did finally pip install jq work

I have experienced same issues as OP. Although in my case (Debian Stretch 9.5) it turned out dh-autoreconf package was missing. After installing it, jq build finished successfully.

How to install psycopg2 with pg_config error?

I've tried to install psycopg2 (PostgreSQL Database adapater) from this site, but when I try to install after I cd into the package and write
python setup.py install
I get the following error:
Please add the directory containing pg_config to the PATH
or specify the full executable path with the option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
I've also tried 'sudo pip install psycopg2' and I got the same message.
After reading through the docs, it asks to look at the setup.cfg file (which is below):
[build_ext]
define=
# PSYCOPG_DISPLAY_SIZE enable display size calculation (a little slower)
# HAVE_PQFREEMEM should be defined on PostgreSQL >= 7.4
# PSYCOPG_DEBUG can be added to enable verbose debug information
# "pg_config" is required to locate PostgreSQL headers and libraries needed to
# build psycopg2. If pg_config is not in the path or is installed under a
# different name uncomment the following option and set it to the pg_config
# full path.
#pg_config=
# Set to 1 to use Python datetime objects for default date/time representation.
use_pydatetime=1
# If the build system does not find the mx.DateTime headers, try
# uncommenting the following line and setting its value to the right path.
#mx_include_dir=
# For Windows only:
# Set to 1 if the PostgreSQL library was built with OpenSSL.
# Required to link in OpenSSL libraries and dependencies.
have_ssl=0
# Statically link against the postgresql client library.
#static_libpq=1
# Add here eventual extra libraries required to link the module.
#libraries=
However, I'm not sure if I'm suppose to edit this file, since the documentation states the following:
then take a look at the setup.cfg file.
Some of the options available in setup.cfg are also available as command line arguments of the build_ext sub-command. For instance you can specify an alternate pg_config version using:
$ python setup.py build_ext --pg-config /path/to/pg_config build
Use python setup.py build_ext --help to get a list of the options supported.
I've gotten the list of options supported but I'm not sure where to go from there

Debian/Ubuntu
Python 2
sudo apt install libpq-dev python-dev
Python 3
sudo apt install libpq-dev python3-dev
Additional
If none of the above solve your issue, try
sudo apt install build-essential
or
sudo apt install postgresql-server-dev-all
With pip
Install the psycopg2-binary PyPI package instead, it has Python wheels for Linux and Mac OS.
pip install psycopg2-binary

I was getting this issue because I hadn't yet installed PostgreSQL on my machine.
For Mac
brew install postgresql
For Mac without brew
python3 -m pip install psycopg2-binary

If you need to install without compiling:
pip install psycopg2-binary
https://www.psycopg.org/docs/install.html#binary-install-from-pypi
Note: The psycopg2-binary package is meant for beginners to start
playing with Python and PostgreSQL without the need to meet the build
requirements. If you are the maintainer of a publish package depending
on psycopg2 you shouldn’t use ‘psycopg2-binary’ as a module
dependency. For production use you are advised to use the source distribution.

If you are on Ubuntu or any other debian-based distro, try
sudo apt-get install python3-psycopg2
Otherwise, you need to find and install the Postgresql client packages for your distribution. psycopg2 installation from source

On Macbook with M1 i had to install postgresql
brew install postgresql
(If you don't have brew: https://brew.sh)
then run the install again
python3 -m pip install psycopg2-binary

Upgrading pip worked for me: pip install --upgrade pip

For OSX with Macports, you can install sudo port install py38-psycopg2 for Python 3.8. You can search for your version of Python with port search psycopg2. At the time of writing, the versions range from 2.7 to 3.9 all up to date with version 2.8.6 of psycopg2.
Note: This probably will not help in a venv.
Edit: So to find your pg_config for a venv, run the command port contents postgresql13 | grep pg_config where postgresql13 is the version of the package of postgresql installed from the above port command. You are looking for a path such as /opt/local/lib/postgresql13/bin/pg_config. Export that to your PATH variable with a command such as export PATH=/opt/local/lib/postgresql13/bin/:$PATH.
Refer to this answer for Homebrew.

On Fedora (tested and built from source on Fedora 35)
pg_config is present in libpq5-devel.
Now try these steps:
sudo dnf install libpq5-devel
python setup.py build
sudo python setup.py install

If you installed Postgres.app on macOS, add its bin directory to the PATH environment variable:
export PATH=/Applications/Postgres.app/Contents/Versions/latest/bin:$PATH

For mac os
install postgresql with : brew install postgresql
then
install psycopg2 with : pip install psycopg2
worked for me

Incase you are using PyCharm/Visual Code Studio terminal, don't do it. Use the default system terminal, as it ensures correct format of brew install is done.
On M1 pro with Intel chip:
brew install postgresql
pip install psycopg2

For me just this worked on Mac:
pip install --upgrade pip
pip install psycopg2-binary

For Mac
I had this issue too. I had followed a guide that had me run
brew install postgresql#15
This worked, but I kept getting the error
Error: pg_config executable not found.
So I then tried to brew install postgres without the #15 after it, and it worked! So, if you have installed postgresql#15 or #any-number, try to brew install without a number.
brew install postgresql

Your error is becouse didn't install the prerequisites on your machine or environment.
You can install prerequisites from here.
In this case probably your missing prerequisite is postgresql you can try below instructors.
For Mac
brew install postgresql
For Linux
sudo apt-get install postgresql

For people building postgres and psycopg2 from source like me, another solution is here:
sudo su
export PATH=/usr/local/pgsql/bin:$PATH #or path to your pg_config
Now setup.py from psycopg2 could find pg_config correctly.
python3 setup.py install
or if you just want to use pip3, pip3 install psycopg2 should work too.

"Failed building wheel for psycopg2" - MacOSX using virtualenv and pip

I'm attempting to make a website with a few others for the first time, and have run into a weird error when trying to use Django/Python/VirtualEnv. I've found solutions to this problem for other operating systems, such as Ubuntu, but can't find any good solutions for Mac.
This is the relevant code being run:
virtualenv -p python3 venv
source venv/bin/activate
pip install -r requirements.txt
After running that block, I get the following errors:
AssertionError
Failed building wheel for django-toolbelt
Running setup.py bdist_wheel for psycopg2
...
AssertionError
Failed building wheel for psycopg2
Failed to build django-toolbelt psycopg2
I believe I've installed the "django-toolbelt" and "psycopg2", so I'm not sure why it would be failing.
The only difference I can think of is that I did not use the command
sudo apt-get install libpq-dev
as was instructed for Ubuntu usage as I believe that installing postgresql with brew took care of the header.
Thanks for any help or insight!

For MacOS users
After trying all the above methods (which did not work for me on MacOS 10.14), that one worked :
Install openssl with brew install openssl if you don't have it already.
add openssl path to LIBRARY_PATH :
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/
install psycopg2 with pip pip3 install psycopg2

I had the same problem on Arch linux. I think that it's not an OS dependant problem. Anyway, I fixed this by finding the outdated packages and updating then.
pip uninstall psycopg2
pip list --outdated
pip install --upgrade wheel
pip install --upgrade setuptools
pip install psycopg2

I was also getting same error.
Using Python 3.7.3 and pip 19.1.1.
I used following command.
pip install psycopg2-binary==2.8.3

TDLR
If you aren't used to installing Python C-extensions, and psycopg2 isn't a core part of your work, try
pip install psycopg2-binary
Building Locally
psycopg2 is a C-extension, so it requires compilation when being installed by pip. The Build Prerequisites section of the docs explain what must be done to make installation via pip possible. In summary (for psycopg 2.8.5):
a C compiler must be installed on the machine
the Python header files must be installed
the libpq header files must be installed
the pg_config program must be installed (it usually comes with the libpq headers) and on $PATH.
With these prerequisites satisfied, pip install psycopg2 ought to succeed.
Installing pre-compiled wheels
Alternatively, pip can install pre-compiled binaries so that compilation (and the associated setup) is not required. They can be installed like this:
pip install psycopg2-binary
The docs note that
The psycopg2-binary package is meant for beginners to start playing with Python and PostgreSQL without the need to meet the build requirements.
but I would suggest that psycopg2-binary is often good enough for local development work if you are not using psycopg2 directly, but just as a dependency.
Concluding advice
Read the informative installation documentation, not only to overcome installation issues but also to understand the impact of using the pre-compiled binaries in some scenarios.

I had same problem and this appears to be a Mojave Issue, I was able to resolve with:
sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /

For Mac OS X users:
1. First check your postgresql path by running this command in terminal:
pg_config
If this fails lookup how to add pg_config to your path.
2. Next install Xcode Tools by running this command in terminal:
xcode-select --install
If you have both those sorted out now try to install psycopg2 again

For MacOS users, this question has the correct solution:
install command line tools if necessary:
xcode-select --install
then
env LDFLAGS="-I/usr/local/opt/openssl/include -L/usr/local/opt/openssl/lib" pip install psycopg2

I was also facing the same after running all the above commands, but the following two commands worked for me:
Instead of pip, use this:
sudo apt-get install libpq-dev
then run this command:
pip install psycopg2

On OS X, I was able to solve this by simply upgrading wheel before installing psycopg2:
pip install --upgrade wheel

For OSX Sierra users, it seems that an xcode update is the solution: Can't install psycopg2 package through pip install... Is this because of Sierra?

I tried all the above solutions but they did not work for me. What I did was change the psycopg2 version in my requirements.txt file from psycopg2==2.7.4 to psycopg2==2.7.6

Is your error message complete? the most encountered reason for failing to install psycopg2 on mac from pip is pg_config is not in path.
by the way, using macports or fink to install psycopg2 is more recommended way, so you don't have to worry about pg_config, libpq-dev and python-dev.
plus, are using Python 3.5? then upgrage your wheel to > 0.25.0 using pip.

I faced the same issue, but the answers above didn't work for me.
So this is what I did in my requirements.txt
psycopg2-binary==2.7.6.1 and it worked fine

I had this issue on several packages, including psycopg2, numpy, and pandas. I simply removed the version from the requirements.txt file, and it worked.
So instead of psycopg2-binary==2.7.6.1 I just had psycopg2-binary.

I know you are asking for development environment but if you are deploying on server say, Heroku. Just add below line in the requirements.txt of your project.
django-heroku==0.3.1
As this package itself will install the required packages like psycopg2 on server deployment.So let the server(heroku) should take care of it.

sudo apt install libpq-dev python3.X-dev
where X is the sub version,
these should be followed by :
pip install --upgrade wheel
pip install --upgrade setuptools
pip install psycopg2
Enjoy !!!

I solved my problem by updating/installing vs_BuildTools. The link to the software was given in the error itself.
Error Image

Fixed by installing python3.7-dev: sudo apt install python3.7-dev, based on the link.
Python: 3.7
Ubuntu: 20.04.3 LTS

Error with pip install lxml [duplicate]

I want to install Lxml so I can then install Scrapy.
When I updated my Mac today it wouldn't let me reinstall lxml, I get the following error:
In file included from src/lxml/lxml.etree.c:314:
/private/tmp/pip_build_root/lxml/src/lxml/includes/etree_defs.h:9:10: fatal error: 'libxml/xmlversion.h' file not found
#include "libxml/xmlversion.h"
^
1 error generated.
error: command 'cc' failed with exit status 1
I have tried using brew to install libxml2 and libxslt, both installed fine but I still cannot install lxml.
Last time I was installing I needed to enable the developer tools on Xcode but since it's updated to Xcode 5 it doesn't give me that option anymore.
Does anyone know what I need to do?

You should install or upgrade the commandline tool for Xcode.
Try this in a terminal:
xcode-select --install

I solved this issue on Yosemite by both installing and linking libxml2 and libxslt through brew:
brew install libxml2
brew install libxslt
brew link libxml2 --force
brew link libxslt --force
If you have solved the problem using this method but it pops up again at a later time, you might need to run this before the four lines above:
brew unlink libxml2
brew unlink libxslt
If you are having permission errors with Homebrew, especially on El Capitan, this is a helpful document. In essence, regardless of OS X version, try running:
sudo chown -R $(whoami):admin /usr/local

You may solve your problem by running this on the commandline:
STATIC_DEPS=true pip install lxml
It sure helped me.
Explanations on docs

I tried most of the solutions above, but none of them worked for me. I'm running Yosemite 10.10, the only solution that worked for me was to type this in the terminal:
sudo CPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/libxml2 CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments pip install lxml
EDIT: If you are using virtualenv, the sudo in beginning is not needed.

This has been bothering me as well for a while. I don't know the internals enough about python distutils etc, but the include path here is wrong. I made the following ugly hack to hold me over until the python lxml people can do the proper fix.
sudo ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/libxml2/libxml/ /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/libxml

Installing globally... OS X 10.9.2
xcode-select --install
sudo easy_install pip
sudo CPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/libxml2 CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments pip install lxml

instalation instructions on http://lxml.de/installation.html explain:
To speed up the build in test environments, e.g. on a continuous integration server, disable the C compiler optimisations by setting the CFLAGS environment variable:
CFLAGS="-O0" pip install lxml

None of the above worked for me on 10.9.2, as compilation bails out with following error:
clang: error: unknown argument: '-mno-fused-madd'
Which actually lead to cleanest solution (see more details in [1]):
export CFLAGS=-Qunused-arguments
export CPPFLAGS=-Qunused-arguments
pip install lxml
or following if installing globally
sudo pip install lxml
[1] clang error: unknown argument: '-mno-fused-madd' (python package installation failure)

xcode-select --install
sudo easy_install pip
sudo pip install lxml

I solved this issue on Yosemite by running the following commands:
xcode-select install #this may take several minutes.
pip install lxml

With homebrew, libxml2 is hidden to not interfere with the system libxml2, so pip must be helped a little in order to find it.
With bash:
LDFLAGS=-L`brew --prefix libxml2`/lib CPPFLAGS=-I`brew --prefix libxml2`/include/libxml2 pip install --user lxml
With fish:
env LDFLAGS=-L(brew --prefix libxml2)/lib CPPFLAGS=-I(brew --prefix libxml2)/include/libxml2 pip install --user lxml

I tried all the answers on this page, none of them worked for me. I'm running OS X Version 10.9.2
But this definitely works....like a charm:
ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install lxml

OSX 10.9.2
sudo env ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future STATIC_DEPS=true pip install lxml

Unfortunately xcode-select --install did not work for me as I already had the latest version.
It's very strange but I solved the issue by opening XCode and accepting the Terms & Conditions. Re-running pip install lxml returned no errors after.

After successful install from pip (lxml 3.6.4) I was getting an error when importing the lxml.etree module.
I was searching endlessly to install this as a requisite for scrapy, and tried all the options, but finally this worked for me (mac osx 10.11 python 2.7):
$ STATIC_DEPS=true sudo easy_install-2.7 "lxml==2.3.5"
The older version of lxml seem to work with etree module.
Pip can often ignore the specified version of a package, for example when you have the newer version in the pip cache, thus the easy_install. The '-2.7' option is for python version, omit this if you are installing for python 3.x.

In my case, I must shutdown Kaspersky Antivirus before installing lxml by:
pip install lxml

before compiling add the path that to xmlversion.h into your environment.
$ set INCLUDE=$INCLUDE:/private/tmp/pip_build_root/lxml/src/lxml/
But make sure the path I've provided has the xmlversion.h file located inside. Then,
$ python setup.py install

pip did not work for me. I went to
https://pypi.python.org/pypi/lxml/2.3
and downloaded the macosx .egg file:
https://pypi.python.org/packages/2.7/l/lxml/lxml-2.3-py2.7-macosx-10.6-intel.egg#md5=52322e4698d68800c6b6aedb0dbe5f34
Then used command line easy_install to install the .egg file.

This post links to a solution that worked for me
Python3, lxml and "Symbol not found: _lzma_auto_decoder" on Mac OS X 10.9
hth

I met the same question and after days of working I resolved this problem on my OS X 10.9.4, with Python 3.4.1.
Here's my solution,
According to installing lxml from lxml.de,
A macport of lxml is available. Try something like port install py25-lxml
If you do not have MacPort, install it from MacPort.org. It's quite easy. You may also need a compiler, to install Xcode compiling tools, use xcode-select --install
Firstly I updated my port to the latest version via sudo port selfupdate,
Then I just type sudo port install libxml2 and several minutes later you should see libxml2 installed successfully. Probably you may also need libxslt to install lxml. To install libxslt, use:sudo port install libxslt.
Now, just type pip install lxml, it should work fine.

I am using OSX 10.9.2 and I get the same error.
Installation of the Xcode command line tools does not help for this particular version of OSX.
I think a better approach to fix this is to install with the following command:
$ CPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/libxml2 pip install lxml
This is similar to jdkoftinoff' fix, but does not alter your system in a permanent way.

After much tearing of the hair and gnashing of the teeth, I uninstalled Xcode with pip and ran:
easy_install lxml
And all was well.

Try:
% STATIC_DEPS=true pip install lxml
Or:
% STATIC_DEPS=true sudo pip install lxml
It works!