We Keep Coding

Error on import caffe

For DeepDream or other deeplearning project, constructing environment of Caffe.
I installed required packages for PyCaffe and made PYTHONPATH to caffe/python.
However when I import caffe on python:
import caffe
Error occurred as below.How to solve this problem?
Segmentation fault: 11

Are you using a mac? I had a very hard time making pycaffe on mac, until I realized that there is a native python installed on all macs and that I was using another version I had installed. While compiling, caffe was using some of the stuff from the native python, and some of the stuff from the other python. I had to make sure to change all the relevant paths in the makefile.config file and to change the python my bash was using. I recommend working in a virtual environment as well. This is a good link to help you out, good luck!

This has been discussed since 2015 in Github issue.
Mostly the reason is the conflict of homebrew python and OS X system python.
Homebrew has provided an awesome solution for OS X:
$ python -V # system Python interpreter
$ python2 -V # Homebrew installed Python 2 interpreter
$ python3 -V # Homebrew installed Python 3 interpreter (if installed)
So the solution is changing all the python paths to python2. Bellowing is my Makefile.config related:
# ...
# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
# /usr/lib/python2.7/dist-packages/numpy/core/include
# ------ For Homebrew installed python. Numpy path is added using python commands.
PYTHON_INCLUDE := /usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/include/python2.7
# We need to be able to find libpythonX.X.so or .dylib. ------ (Update Homebrew path)
# PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib
PYTHON_LIB := /usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib
# Homebrew installs numpy in a non standard path (keg only) ------ (python2 for brew instead of python for system)
PYTHON_INCLUDE += $(dir $(shell python2 -c 'import numpy.core; print(numpy.core.__file__)'))/include
PYTHON_LIB += $(shell brew --prefix numpy)/lib
# ...

Try to manually set python path in your python script if you get no module named caffe error
Ex.
import sys
sys.path.insert(0, "/home/nviso/GitHub/caffe/distribute/python")
import caffe
This generally works for me.
Adding the caffe or python path to .bashrc manually should probably solve the issue as well though not sure, don't have my Office PC now to try :)

Getting the correct version of Pmw to install

Problem:
I'd like to install Pmw 2.0.0 (project page here) so that I can use it with tkinter in python3. The setup script from the package detects which version of python you're using and installs the version that is appropriate for your system (Ubuntu 15 in my case). I can't find any references to switches to make it install the 2.0.0 instead of 1.3.3(the Python 2.7 version), nor have I been able to get the script to install to the python3 libraries.
What I've done so far:
I've changed the python version detector in the setup script from
if sys.version_info[0]<3:
version='2.0.0' # really '1.3.3'
packages=['Pmw', 'Pmw.Pmw_1_3_3', 'Pmw.Pmw_1_3_3.lib',]
to
if sys.version_info[0]<2:
version='2.0.0' # really '1.3.3'
packages=['Pmw', 'Pmw.Pmw_1_3_3', 'Pmw.Pmw_1_3_3.lib',]
to attempt to force the installer to default to the python3 version, which it does, but it installs them in the python2.7 libraries (/usr/local/lib/python2.7/distpackages).
What I want to do:
I'm looking for a way to force the installer to put the 3.4-compatible package into the python3 libraries. If that means getting it to install both packages in their respective correct directories, that's fine, too. I'm stumped about what to try next.

Answered by RazZiel on AskUbuntu:
Link here.
Instead of using the command sudo python setup.py build and then sudo python setup.py install, I should have been using python3 to execute the setup script. I've managed to outthink myself pretty badly on this one.

Pip build option to use multicore

I found that pip only use single core when it compiles packages. Since some python packages takes some time to build using pip, I'd like to utilize multicore on the machine. When using Makefile, I can do that like following command:
make -j4
How can I achieve same thing for pip?

The Ultimate Way to Resolve This Problem
Because all the c / cpp files would be compiled by using make commend, and make has an option which specify how many cpu cores shoule be used to compile the source code, we could do some tricks on make.
Backup your original make command:
sudo cp /usr/bin/make /usr/bin/make.bak
write a "fake" make command, which will append --jobs=6 to its parameter list and pass them to the original make command make.bak:
make.bak --jobs=6 $#
So after that, not even compile python with c libs, but also others contain c libs would speed up on compilation by 6 cores. Actually all files compiled by using make command will speed up.
And good luck.
Use: --install-option="--jobs=6" (pip docs).
pip3 install --install-option="--jobs=6" PyXXX
I have the same demand that use pip install to speed the compile progress. My target pkg is PySide. At first I use pip3 install pyside, it takes me nearly 30 minutes (AMD 1055T 6-cores, 10G RAM), only one core take 100% load.
There are no clues in pip3 --help, but I found lots of options like pip install -u pyXXX, but I didn't know what is '-u' and this parameter was not in pip --help too. I tried 'pip3 install --help' and there came the answer: --install-option.
I read the code of PySide's code and found another clue: OPTION_JOBS = has_option('jobs'), I put ipdb.set_trace() there and finally understand how to use multicore to compile by using pip install.
it took me about 6 minutes.
--------------------------update------------------------------
as comment below, I finally used tricks like this:
cd /usr/bin
sudo mv make make.bak
touch make
then edit make: vim make or other way you like and type this:
make.bak --jobs=6 $*
I'm not familiar with bash, so I'm not sure if this is the correcct bash code. I'm writing this comment in windows. The key is rename make into make.bak, and then create a new make, use this new make to call make.bak with added param --jobs=6

Tested this works
https://stackoverflow.com/a/57014278/6147756
Single command:
MAKEFLAGS="-j$(nproc)" pip install xxx
Enable for all commands in a script:
export MAKEFLAGS="-j$(nproc)"

From what I can tell it does not look like pip has this ability but I may be mistaken.
To do multiprocessing in python you use the multiprocessing package, [here is a guide I found] (http://pymotw.com/2/multiprocessing/basics.html) about how to do it if you are interested and this is a link to the python docs that talk about it. I also found this question useful, Multiprocessing vs Threading Python, to make sure that multiprocessing did what I thought it did, being take advantage of multiple CPUs.
I have gone through the pip source code (available here) looking for a reference to the multiprocessing package and did not find any use of the package. This would mean that pip does not use/support multiprocessing. From what I can tell the /pip/commands/install.py file is the one of interest for your question as it is called when you run pip install <package>. For this file specifically the imports are
from __future__ import absolute_import
import logging
import os
import tempfile
import shutil
import warnings
from pip.req import InstallRequirement, RequirementSet, parse_requirements
from pip.locations import virtualenv_no_global, distutils_scheme
from pip.basecommand import Command
from pip.index import PackageFinder
from pip.exceptions import (
InstallationError, CommandError, PreviousBuildDirError,
)
from pip import cmdoptions
from pip.utils.deprecation import RemovedInPip7Warning, RemovedInPip8Warning
which you can see does not have any reference to the multiprocessing package but I did check all of the other files just to be sure.
Furthermore, I checked the pip install documentation and found no reference to installing using multiple cores.
TL;DR: Pip doesn't do what you are asking. I may be wrong as I didn't look at the source that long but I'm pretty sure it just doesn't support it.

Distribution independent libpython path

Under newer Ubuntu/Debian versions, libpython2.7.so is under /usr/lib/i386-linux-gnu/libpython2.7.so or /usr/lib/x86_64-linux-gnu/libpython2.7.so, etc. Earlier, they could be found in /usr/lib/libpython2.7.so, no matter the architecture. I haven't checked for other distributions. How do I find the path of libpython2.7.so with python?

Using pkg-config is not the best option - it will not distinguish between different installations of Python, returning only the system installation. You are better off using the Python executable to discover the location of libpythonX.Y.so.
From inside Python:
from distutils import sysconfig;
print sysconfig.get_config_var("LIBDIR")
Or inside a Makefile:
PYTHON_LIBDIR:=$(shell python -c 'from distutils import sysconfig; print sysconfig.get_config_var("LIBDIR")')
This will discover the location from whatever Python executable is first in $PATH and thus will work if there are multiple Python installations on the system.
Credit to Niall Fitzgerald for pointing this out.

Here is my solution which seems to work against system wide Debian and CentOS installation, anaconda on Debian, miniconda on OSX, virtualenv on Debian... but fails for system-wide python on OSX:
from distutils import sysconfig;
import os.path as op;
v = sysconfig.get_config_vars();
fpaths = [op.join(v[pv], v['LDLIBRARY']) for pv in ('LIBDIR', 'LIBPL')];
print(list(filter(op.exists, fpaths))[0])
and here it ran on my laptop:
$> for p in python python3 ~/anaconda-4.4.0-3.6/bin/python ~datalad/datalad-master/venvs/dev/bin/python ; do $p -c "from distutils import sysconfig; import os.path as op; v = sysconfig.get_config_vars(); fpaths = [op.join(v[pv], v['LDLIBRARY']) for pv in ('LIBDIR', 'LIBPL')]; print(list(filter(op.exists, fpaths))[0])"; done
/usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so
/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.so
/home/yoh/anaconda-4.4.0-3.6/lib/libpython3.6m.so
/usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so
P.S. I had no clue that it is such a problem... bad bad bad Python

I'm assuming you're looking to link against this file. Python is usually installed with pkgconfig info to help compile against it. Specifically for the .so file, you should use pkg-config --libs python-2.7. From Python:
import subprocess
subprocess.check_output(["pkg-config", "--libs", "python-2.7"])
If the only flag shown is -lpython2.7, you might want to consider reading /etc/ld.so.conf to see default locations in which the linker looks for its libraries.

How can I update Python on Mac OS and have RubyGems use the new version?

I am having serious issues trying to download the "therubyracer" gem. I think the problem is that I do not have a working libv8 library to create the gem.
I think I do not have a working libv8 because my RubyGems keeps trying to use "Python 2.5.6", which is too old to gem install libv8. I tried countless ways to update the Python that RubyGems uses, however, I cannot figure it out.
Here is what I have done thus far. I went to http://www.python.org/download/releases/3.3.2/ and downloaded "(Mac OS X 64-bit/32-bit Installer (3.3.2) for Mac OS X 10.6 and later 2.)".
Upon completion of the download, I ran:
gem install libv8 --version 3.11.8.17
Yet, I still received the same error. I noticed the output still said:
"Using Python 2.5.4"
Despite downloading a newer Python version, it is not being used. So how do I change the version of Python that the "gem" command uses? I searched and discovered "virtualenv," which appears to be a good Python manager, however I am a bit at a dead end now. Any help or suggestion?

If libv8 uses the python command from the command line, this should apply:
If you do not care what python2 is installed, you can do this in the command line:
type python # this gives you the path of the python command
python is /usr/bin/python
type python2.7
python is /usr/bin/python2.7
Then you can copy one over the other
cp /usr/bin/python2.7 /usr/bin/python
You can safely copy python2.7 there because it is compatible with older versions.
Suggestions?