After successfully compiling Mesos 0.16.0, running the tests fails when checking the PythonFramework. All other tests pass successfully.
Steps I used for building:
./bootstrap
mkdir build
cd build
../configure CXX=g++4.7 CC=gcc-4.7
make
Then, when running the tests;
make check
The results look like this:
[...]
[ RUN ] ExamplesTest.PythonFramework
../../src/tests/script.cpp:78: Failure
Failed
python_framework_test.sh exited with status 1
[ FAILED ] ExamplesTest.PythonFramework (201 ms)
[...]
Environment:
OS X 10.9.1 (Mavericks)
Python 2.7.5 (default, Aug 25 2013, 00:04:04) [GCC 4.2.1 Compatible
Apple LLVM 5.0 (clang-500.0.68)] on darwin
gcc-4.7 (GCC) 4.7.3 Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There
is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
How do I build proper Mesos Python bindings that pass the tests within this environment?
Update:
My former answer has mostly become obsolete as of Mesos 0.17.0 due to the fact that this version does fully handle clang compilage (Yay!). So there is no need to compile it using gcc anymore - just go ahead and use Xcode's clang (Xcode commandline utilities).
In case you still get into trouble getting the Python bindings to work, please add a comment or new question here on StackOverflow or post into the Mesos Mailing list.
Mesos version 0.16.0 or lower:
How to fix the Python bindings of Mesos on OS X (10.9).
Install Python 2.7.3 via homebrew
Find out which versions are available
brew versions python
2.7.6 git checkout 3c86d2b /usr/local/Library/Formula/python.rb
2.7.5 git checkout a04b443 /usr/local/Library/Formula/python.rb
2.7.3 git checkout 865f763 /usr/local/Library/Formula/python.rb
2.7.4 git checkout 280581d /usr/local/Library/Formula/python.rb
[...]
Select Python 2.7.3
cd /usr/local/Library/Formula/
git checkout 865f763 /usr/local/Library/Formula/python.rb
brew install python
Make sure you do not force installing a universal build (32 + 64bit) as that would again cause the same issue explained below. The default is 64bit only and that is just fine.
Rebuild Mesos in connection with your custom Python installation
rm -rf build
rm -rf ~/.python-eggs
mkdir build
cd build
../configure CXX=g++-4.7 CC=gcc-4.7 PYTHON=/usr/local/bin/python
make
make check
You should now see a properly functioning test, hence a perfectly fine Mesos Python binding:
[ RUN ] ExamplesTest.PythonFramework
[ OK ] ExamplesTest.PythonFramework (1682 ms)
As asking users to install a custom Python version often is just wrong but in this case appears to be inevitable, let me draft an explanation of the issue. And maybe one of the readers knows a better workaround.
Manually executing that test using the verbosive output setting does help identifying the exact problem.
bin/mesos-tests.sh --gtest_filter="*.PythonFramework" --verbose
Traceback (most recent call last): File
"/Users/till/Documents/Development/github/mesos-master/build/../src/examples/python/test_framework.py",
line 23, in
import mesos File "build/bdist.macosx-10.9-intel/egg/mesos.py", line 26, in File
"build/bdist.macosx-10.9-intel/egg/_mesos.py", line 7, in
File "build/bdist.macosx-10.9-intel/egg/_mesos.py", line 6, in
bootstrap ImportError: dlopen(/Users/till/.python-eggs/mesos-0.16.0-py2.7-macosx-10.9-intel.egg-tmp/_mesos.so,
2): Symbol not found: __ZNSoD0Ev Referenced from:
/Users/till/.python-eggs/mesos-0.16.0-py2.7-macosx-10.9-intel.egg-tmp/_mesos.so
Expected in: flat namespace in
/Users/till/.python-eggs/mesos-0.16.0-py2.7-macosx-10.9-intel.egg-tmp/_mesos.so
The important detail is the fact that the dynamic linkage of that native Python egg failed.
The reasoning is to be found within the distutils build step of this module when building mesos 0.16. The Python distutils derive their build-settings directly from python-config. As your Python was built using clang, the distutils will try to build your native egg using clang as well.
Issues:
Mesos' autoconf phase did not propagate the compiler settings into the distutils build phase. So even though Mesos itself is being built using gcc-4.7 in the above description, the egg is being built using clang. The result is a mishmash of libc++ and stdlibc++ which are not ABI compatible.
that part is being fixed right now, Mesos will use the same compiler in the distutils build phase as well (see MESOS-798 and MESOS-799). Chances are pretty high that when you read this answer, that particular issue was already taken care of.
The default OS X Python distutils do enforce building universal binaries (i386 + x86_64) using parameters that only the gcc-frontend of clang supports. There appears to be no workaround, hence all dynamically linked dependencies of that egg will have to be built for both architectures as well (this appears to be a leftover from OS X 10.6).
Mesos itself is linked statically into that egg, hence it does not have to be built as a universal binary for the egg to build and function on a 64bit platform. It will however fail to execute on a 32bit platform.
As long as Mesos does not support clang compilage (hence being linked against libc++), the only proper workaround seems to be to install a differently compiled Python. A quick and easy solution is using homebrew to install Python 2.7.3. Note: do not install Python 2.7.6 (the current default of homebrew) as that one has issues in connection with its autoconf developer macro (see MESOS-617)
Related
I have upgraded my RHEL OS from 6.7 to 7.5. After upgrading, I found some issues when trying to run yum. Below are the details.
# yum repolist
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:
/lib64/libc.so.6: version `GLIBC_2.14' not found (required by /lib64/libgcc_s.so.1)
Please install a package which provides this module, or
verify that the module is installed correctly.
It's possible that the above module doesn't match the
current version of Python, which is:
2.6.6 (r266:84292, Aug 9 2016, 06:11:56)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)]
If you cannot solve this problem yourself, please go to
the yum faq at:
http://yum.baseurl.org/wiki/Faq
After getting this error, I just installed python2.7 and GLIBC 2.14. But when I am trying to install GLIBC 2.14 from my current GLIBC version 2.12, it is throwing some error. Below are the steps that I am using to install GLIBC 2.14:
tar xvfz glibc-2.14.tar.gz
cd glibc-2.14
mkdir build
cd build
../configure --prefix=/opt/glibc-2.14
make
sudo make install
export LD_LIBRARY_PATH=/opt/glibc-2.14/lib:$LD_LIBRARY_PATH
In step5, I am getting error. Below are the details:
# ../configure --prefix=/opt/glibc-2.14
checking for forced unwind support... no
configure: error: forced unwind support is required
I am unaware of this error "unwind support is required".Please let me know how to setup/install forced unwind in Redhat 7.5.
add libc_cv_forced_unwind=yes when configure:
../configure --prefix=/opt/glibc-2.14 libc_cv_forced_unwind=yes
and make
Below is the error I get when I do a sudo pip install imposm
#include "tcutil.h"
^
1 error generated.
error: command 'gcc' failed with exit status 1
I believe that I already have gcc (version: 4.2.1) installed under /usr/bin/gcc on my Mac (OSX 10.8.5). I don't know what else needs to be done. Other details.
gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin12.5.0
Thread model: posix
Edit: I already have the listed requirements (C/C++ compiler, the Python libraries and Google Protobuf) on my Mac. This is why the error baffles me.
You need to install tokyo-cabinet, which imposm uses for caching.
If you use homebrew, it's simply: brew install tokyo-cabinet
The documentation for that module states the following:
The PBF parser is written as a C extension and you need to have a
C/C++ compiler, the Python libraries and Google Protobuf.
At least, if this is the appropriate documentation for that module (http://imposm.org/docs/imposm.parser/latest/install.html#installation)
I can't really research how to install these components on a mac, but if you haven't fulfilled those requirements, that may be the reason why it's not working.
I'm trying to install numpy 1.7 via an RPM on an older Linux machine with Python 2.4. The numpy release notes and the RPM page say it is supposed to be compatible with 2.4 (or <= 2.7), but when I try to install it on the machine with the command
rpm -i /tmp/python-numpy-1.7.0-2.1.i586.rpm
I get a number of missing dependency notes, including:
libc.so.6(GLIBC_2.11) is needed by python-numpy-1.7.0-2.1.i586
libc.so.6(GLIBC_2.4) is needed by python-numpy-1.7.0-2.1.i586
liblapack.so.3 is needed by python-numpy-1.7.0-2.1.i586
libpython2.7.so.1.0 is needed by python-numpy-1.7.0-2.1.i586
python >= 2.7 is needed by python-numpy-1.7.0-2.1.i586
python = 2.7 is needed by python-numpy-1.7.0-2.1.i586
python(abi) = 2.7 is needed by python-numpy-1.7.0-2.1.i586
rpmlib(PayloadIsLzma) <= 4.4.6-1 is needed by python-numpy-1.7.0-2.1.i586
So now at least Python 2.7 is needed, rather than up to 2.7. Is this a real discrepancy or am I using rpm incorrectly? I'm used to higher-level Linux package managers that report dependencies correctly and install them automatically, so I'm unsure how to proceed here.
Are you sure your distribution does not provide numpy already? It looks like numpy is part of epel.
If for some reason you are unwilling to use the version in the distribution, you're likely going to have to build the RPM yourself. I was able to build 1.7.1 on CentOS 5.7 like so:
sudo yum install rpm-build gcc python-devel
wget 'https://pypi.python.org/packages/source/n/numpy/numpy-1.7.1.tar.gz'
tar -xf numpy-1.7.1.tar.gz
cd numpy-1.7.1/
python setup.py bdist_rpm
sudo yum localinstall dist/numpy-1.7.1-1.i386.rpm
the generated RPM (in ./dist) should be useable without rebuilding on all of the machines with similar hardware and OS.
If all the machines have identical versions of Python, glibc, etc., then it would probably be easier to get the numpy source and build it yourself, assuming you have gcc (and perhaps gfortran) installed, along with dependencies like BLAS and LAPACK. Once it's installed on one machine, you can copy the numpy folder (and any .egg file) from /usr/lib/python2.4/site-packages (or whichever directory) and distribute that around the world. Make sure to create static libraries when you build so you don't need all the dependencies everywhere.
I'd also get numpy 1.7.1, as it fixes some issues with 1.7.0.
I'm not having much success when attempting building pgmagick on OS X Lion with XCode 4.3.1.
I've installed both ImageMagick and GraphicsMagick, along side boost, using the following commands (via homebrew):
$ brew install graphicsmagick --with-magick-plus-plus
$ brew install imagemagick --with-magick-plus-plus
$ brew install boost --with-thread-unsafe
then I'm cloning the repo at https://bitbucket.org/hhatto/pgmagick:
$ hg clone https://bitbucket.org/hhatto/pgmagick/src
$ cd pgmagick
$ python setup.py build
However I always receive the following error:
ld: library not found for -lboost_python
collect2: ld returned 1 exit status
Based on the output on stdout, setup is looking in the right place for the boost (/usr/local/lib).
I've also tried easy_install and pip but with no luck. I'm using Pythonbrew but have also disabled this and tried using the stock python install -- still no success.
Any suggestions on how I can fix the problem, or further diagnose the issue?
According to my own reproduction of this issue in brew 0.9 and OSX 10.6.8, the problem is --with-thread-unsafe isn't being honored by the current brew formula file. You can verify this by checking the formula with brew edit boost and seeing if the option appears within the contents of the formula.
Because of this, libboost_python-mt.a and libboost_python-mt.dylib are being built instead of libboost_python.a and libboost_python.dylib.
The easiest ways to fix this are to edit your pgmagick setup.py to replace boost_lib="boost_python" with boost_lib="boost_python-mt" (as pointed out here) or to follow the instructions and patch here. It's otherwise a known issue.
The boost_python lib inside /usr/local/lib/ is named after libboost_python-mt.a and libboost_python-mt.dylib, since the default compiling is w/ multi-threads supporting enabled.
Grep boost_lib="boost_python" under ELSE condition in setup.py and replace it w/ boost_lib="boost_python-mt", will fix the "not found" issue.
Also it's OK to ln "-mt" version to libboost_python.a: as described here for linux boost which no longer appends '-mt' suffix since 1.42.
Ignore this line or you could "with-boost-python=boost_python-mt python setup.py install".
You could probably append '--with-boost-python=boost_python-mt' to extra_compile_args inside setup.py, to achieve the same goal.
Furthermore, you could install pgmagick through pip in managed envs. Refs http://rohanradio.com/blog/2011/12/02/installing-pgmagick-on-os-x/
Note that as of July 2014 the boost Python library is a separate homebrew package called boost-python.
5254f8f510fb30484f8fac8be3d38e388a4392e2
Author: Tim D. Smith <git#tim-smith.us>
Date: Sat Jul 19 15:37:25 2014 -0700
Split out Boost.Python
You need to install it separately to get the libboost_python shared library.
Does setting DYLD_FALLBACK_LIBRARY_PATH=/usr/local/lib in the environment help before the build
e.g
$ export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/lib
$ hg clone https://bitbucket.org/hhatto/pgmagick/src
$ cd pgmagick
$ python setup.py build
I've submitted a pull request to homebrew to build Boost with both mt and non mt (threaded and thread unsafe) binaries which are required to build pgmagick.
Turns out this is a rather common problem, until the patch is accepted, you can check out or use my formula for Boost to build pgmagick.
I am setting up Python and Django on os X 10.7 from a virgin install and Xcode 4.3.
I tried using the default install of Python:
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/
I normally use a Python based package manager called easy_install.
Easy_install seems to not be able to find the compiler.
EDIT: When I tried to install MySQL-python I got this error:
$ sudo easy_install MySQL-python
Password:
Searching for MySQL-python
Reading http://pypi.python.org/simple/MySQL-python/
Reading http://sourceforge.net/projects/mysql-python/
Reading http://sourceforge.net/projects/mysql-python
Best match: MySQL-python 1.2.3
Downloading http://download.sourceforge.net/sourceforge/mysql-python/MySQL-python-1.2.3.tar.gz
Processing MySQL-python-1.2.3.tar.gz
Running MySQL-python-1.2.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-P9H9WX/MySQL-python-1.2.3/egg-dist-tmp-rRTfZL
warning: no files found matching 'MANIFEST'
warning: no files found matching 'ChangeLog'
warning: no files found matching 'GPL'
unable to execute llvm-gcc-4.2: No such file or directory
error: Setup script exited with error: command 'llvm-gcc-4.2' failed with exit status 1
error: Setup script exited with error: command 'llvm-gcc-4.2' failed with exit status 1
Apparently, the system attempts to use the same compiler used to compile the installed Python framework.
For some reason Apple didn't include llvm-gcc-4.2.
Xcode 4.1 used GCC, but with Xcode 4.3 that seems to have changed.
From what I can gather, Apple wants to use Clang as the compiler vs GCC.
So I added to .bash_profile:
cc=clang
I decided I would just recompile Python with clang but first, I needed to install readline.
Fail:
Wed Feb 22 16:04:59 ~/Downloads/readline-6.2
$ ./configure
checking build system type... i386-apple-darwin11.3.0
checking host system type... i386-apple-darwin11.3.0
Beginning configuration for readline-6.2 for i386-apple-darwin11.3.0
checking whether make sets $(MAKE)... no
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in `/Users/Bryan/Downloads/readline-6.2':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details.
What the easiest way to install Django on Lion 10.7?
Did you install the command-line tools with Xcode 4.3? They are not installed by default. You can install them by going to the Downloads pane in Xcode 4.3's preferences.
Making it way too hard:
First, make sure you install Xcode (available for free in the Mac App Store). It includes all the build tools that might be necessary to compile certain Python packages.
To get easy_install just download setuptools and follow the instructions for installing on Mac OS X at that link.
Once that's done, you can easy_install virtualenv to get a nice segregated environment to work in.
For the MySQL issue you have to edit the site.cfgfile:
mysql_config = /usr/local/mysql/bin/mysql_config
And then:
$ python setup.py build
$ sudo python setup.py install
Try updating XCode and reinstall all global site-packages. You may also want to try pip instead of easy_install.
You can also try to compile with
export ARCHFLAGS='-arch i386 -arch x86_64'
This solved many of my problems in the past when upgrading OSX versions.