We Keep Coding

Mesos ExamplesTest.PythonFramework check fails on OSX

After successfully compiling Mesos 0.16.0, running the tests fails when checking the PythonFramework. All other tests pass successfully.
Steps I used for building:
./bootstrap
mkdir build
cd build
../configure CXX=g++4.7 CC=gcc-4.7
make
Then, when running the tests;
make check
The results look like this:
[...]
[ RUN ] ExamplesTest.PythonFramework
../../src/tests/script.cpp:78: Failure
Failed
python_framework_test.sh exited with status 1
[ FAILED ] ExamplesTest.PythonFramework (201 ms)
[...]
Environment:
OS X 10.9.1 (Mavericks)
Python 2.7.5 (default, Aug 25 2013, 00:04:04) [GCC 4.2.1 Compatible
Apple LLVM 5.0 (clang-500.0.68)] on darwin
gcc-4.7 (GCC) 4.7.3 Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There
is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
How do I build proper Mesos Python bindings that pass the tests within this environment?

Update:
My former answer has mostly become obsolete as of Mesos 0.17.0 due to the fact that this version does fully handle clang compilage (Yay!). So there is no need to compile it using gcc anymore - just go ahead and use Xcode's clang (Xcode commandline utilities).
In case you still get into trouble getting the Python bindings to work, please add a comment or new question here on StackOverflow or post into the Mesos Mailing list.
Mesos version 0.16.0 or lower:
How to fix the Python bindings of Mesos on OS X (10.9).
Install Python 2.7.3 via homebrew
Find out which versions are available
brew versions python
2.7.6 git checkout 3c86d2b /usr/local/Library/Formula/python.rb
2.7.5 git checkout a04b443 /usr/local/Library/Formula/python.rb
2.7.3 git checkout 865f763 /usr/local/Library/Formula/python.rb
2.7.4 git checkout 280581d /usr/local/Library/Formula/python.rb
[...]
Select Python 2.7.3
cd /usr/local/Library/Formula/
git checkout 865f763 /usr/local/Library/Formula/python.rb
brew install python
Make sure you do not force installing a universal build (32 + 64bit) as that would again cause the same issue explained below. The default is 64bit only and that is just fine.
Rebuild Mesos in connection with your custom Python installation
rm -rf build
rm -rf ~/.python-eggs
mkdir build
cd build
../configure CXX=g++-4.7 CC=gcc-4.7 PYTHON=/usr/local/bin/python
make
make check
You should now see a properly functioning test, hence a perfectly fine Mesos Python binding:
[ RUN ] ExamplesTest.PythonFramework
[ OK ] ExamplesTest.PythonFramework (1682 ms)
As asking users to install a custom Python version often is just wrong but in this case appears to be inevitable, let me draft an explanation of the issue. And maybe one of the readers knows a better workaround.
Manually executing that test using the verbosive output setting does help identifying the exact problem.
bin/mesos-tests.sh --gtest_filter="*.PythonFramework" --verbose
Traceback (most recent call last): File
"/Users/till/Documents/Development/github/mesos-master/build/../src/examples/python/test_framework.py",
line 23, in
import mesos File "build/bdist.macosx-10.9-intel/egg/mesos.py", line 26, in File
"build/bdist.macosx-10.9-intel/egg/_mesos.py", line 7, in
File "build/bdist.macosx-10.9-intel/egg/_mesos.py", line 6, in
bootstrap ImportError: dlopen(/Users/till/.python-eggs/mesos-0.16.0-py2.7-macosx-10.9-intel.egg-tmp/_mesos.so,
2): Symbol not found: __ZNSoD0Ev Referenced from:
/Users/till/.python-eggs/mesos-0.16.0-py2.7-macosx-10.9-intel.egg-tmp/_mesos.so
Expected in: flat namespace in
/Users/till/.python-eggs/mesos-0.16.0-py2.7-macosx-10.9-intel.egg-tmp/_mesos.so
The important detail is the fact that the dynamic linkage of that native Python egg failed.
The reasoning is to be found within the distutils build step of this module when building mesos 0.16. The Python distutils derive their build-settings directly from python-config. As your Python was built using clang, the distutils will try to build your native egg using clang as well.
Issues:
Mesos' autoconf phase did not propagate the compiler settings into the distutils build phase. So even though Mesos itself is being built using gcc-4.7 in the above description, the egg is being built using clang. The result is a mishmash of libc++ and stdlibc++ which are not ABI compatible.
that part is being fixed right now, Mesos will use the same compiler in the distutils build phase as well (see MESOS-798 and MESOS-799). Chances are pretty high that when you read this answer, that particular issue was already taken care of.
The default OS X Python distutils do enforce building universal binaries (i386 + x86_64) using parameters that only the gcc-frontend of clang supports. There appears to be no workaround, hence all dynamically linked dependencies of that egg will have to be built for both architectures as well (this appears to be a leftover from OS X 10.6).
Mesos itself is linked statically into that egg, hence it does not have to be built as a universal binary for the egg to build and function on a 64bit platform. It will however fail to execute on a 32bit platform.
As long as Mesos does not support clang compilage (hence being linked against libc++), the only proper workaround seems to be to install a differently compiled Python. A quick and easy solution is using homebrew to install Python 2.7.3. Note: do not install Python 2.7.6 (the current default of homebrew) as that one has issues in connection with its autoconf developer macro (see MESOS-617)

Errors installing Matplotlib - clang

I've tried to install Matplot lib but I'm still struggling to get past errors.
I've installed numpy and psipy but have the following error when I use easy_install (there is a similar error with pip).
And yes, I've installed the command line tools in Xcode in a bid to solve the 'gcc-4.2 not found' problem.
pymods ['pylab']
packages ['matplotlib', 'matplotlib.backends', 'matplotlib.backends.qt4_editor',
'matplotlib.projections', 'matplotlib.testing', 'matplotlib.testing.jpl_units',
'matplotlib.tests', 'mpl_toolkits', 'mpl_toolkits.mplot3d', 'mpl_toolkits.axes_grid',
'mpl_toolkits.axes_grid1', 'mpl_toolkits.axisartist', 'matplotlib.sphinxext',
'matplotlib.tri', 'matplotlib.delaunay', 'pytz', 'dateutil', 'dateutil.zoneinfo']
warning: no files found matching 'KNOWN_BUGS'
warning: no files found matching 'INTERACTIVE'
warning: no files found matching 'MANIFEST'
warning: no files found matching '__init__.py'
warning: no files found matching 'examples/data/*'
warning: no files found matching 'lib/mpl_toolkits'
warning: no files found matching 'LICENSE*' under directory 'license'
gcc-4.2 not found, using clang instead
In file included from src/ft2font.cpp:3:
src/ft2font.h:16:10: fatal error: 'ft2build.h' file not found
#include <ft2build.h>
^
1 error generated.
error: Setup script exited with error: command 'clang' failed with exit status 1

I recently had this issue as well, but I was using pip. If you have homebrew installed you can solve it by running the following commands:
brew install freetype
brew install libpng
brew link freetype
pip install git+https://github.com/matplotlib/matplotlib.git#egg=matplotlib-dev

theres actually an easier way! First see if X11 libraries are installed (ls /usr/X11/include)
If they aren't you may need to install X11
If they are then a simple fix is this:
cd /usr/X11/include
sudo ln -s freetype2/freetype
Basically for some reason its looking for the freetype library at /usr/X11 and its actually in the subfolder in freetype2. Odd - but it does then compile this way.

I did the following and it worked for me
sudo apt install libfreetype6-dev
pip install matplotlib

You're missing freetype2, or at least the development version. It's looking for ft2build.h, which is a freetype2 header file.
You can either try installing freetype2 from source, or just a package manager like macports, fink or homebrew to install it (for package managers, make sure you pick the development version if that's available).
But if you are going the route of package managers, you could also install matplotlib that way. And there may be binaries for OS X that include freetype2, so that may be even easier than installing from source.
The reason easy_install and pip don't install freetype2 for you, is that it's not a Python package.
== Correction ==
freetype2 should be available, not just where easy_install or pip expect it. On my Mac, I can ft2build.h in /usr/X11/include. So, you'll need to set the CFLAGS variable to that directory (and probably LDFLAGS to /usr/X11/lib`) to get the installation working. Now, with easy_install and pip, I'm not 100% sure how to do that. Try e.g.
CFLAGS=-I/usr/X11/include LDFLAGS=-L/usr/X11/lib easy_install matplotlib
Or, just download the matplotlib source, adjust setup.py and build it yourself.

What is the best way to setup Django on os X 10.7 Lion?

I am setting up Python and Django on os X 10.7 from a virgin install and Xcode 4.3.
I tried using the default install of Python:
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/
I normally use a Python based package manager called easy_install.
Easy_install seems to not be able to find the compiler.
EDIT: When I tried to install MySQL-python I got this error:
$ sudo easy_install MySQL-python
Password:
Searching for MySQL-python
Reading http://pypi.python.org/simple/MySQL-python/
Reading http://sourceforge.net/projects/mysql-python/
Reading http://sourceforge.net/projects/mysql-python
Best match: MySQL-python 1.2.3
Downloading http://download.sourceforge.net/sourceforge/mysql-python/MySQL-python-1.2.3.tar.gz
Processing MySQL-python-1.2.3.tar.gz
Running MySQL-python-1.2.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-P9H9WX/MySQL-python-1.2.3/egg-dist-tmp-rRTfZL
warning: no files found matching 'MANIFEST'
warning: no files found matching 'ChangeLog'
warning: no files found matching 'GPL'
unable to execute llvm-gcc-4.2: No such file or directory
error: Setup script exited with error: command 'llvm-gcc-4.2' failed with exit status 1
error: Setup script exited with error: command 'llvm-gcc-4.2' failed with exit status 1
Apparently, the system attempts to use the same compiler used to compile the installed Python framework.
For some reason Apple didn't include llvm-gcc-4.2.
Xcode 4.1 used GCC, but with Xcode 4.3 that seems to have changed.
From what I can gather, Apple wants to use Clang as the compiler vs GCC.
So I added to .bash_profile:
cc=clang
I decided I would just recompile Python with clang but first, I needed to install readline.
Fail:
Wed Feb 22 16:04:59 ~/Downloads/readline-6.2
$ ./configure
checking build system type... i386-apple-darwin11.3.0
checking host system type... i386-apple-darwin11.3.0
Beginning configuration for readline-6.2 for i386-apple-darwin11.3.0
checking whether make sets $(MAKE)... no
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in `/Users/Bryan/Downloads/readline-6.2':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details.
What the easiest way to install Django on Lion 10.7?

Did you install the command-line tools with Xcode 4.3? They are not installed by default. You can install them by going to the Downloads pane in Xcode 4.3's preferences.

Making it way too hard:
First, make sure you install Xcode (available for free in the Mac App Store). It includes all the build tools that might be necessary to compile certain Python packages.
To get easy_install just download setuptools and follow the instructions for installing on Mac OS X at that link.
Once that's done, you can easy_install virtualenv to get a nice segregated environment to work in.

For the MySQL issue you have to edit the site.cfgfile:
mysql_config = /usr/local/mysql/bin/mysql_config
And then:
$ python setup.py build
$ sudo python setup.py install

Try updating XCode and reinstall all global site-packages. You may also want to try pip instead of easy_install.
You can also try to compile with
export ARCHFLAGS='-arch i386 -arch x86_64'
This solved many of my problems in the past when upgrading OSX versions.

Switching gcc version on mac

I have the newest XCode (4D199) installed and in terminal when I type
new-host-2: me$ gcc -version
i686-apple-darwin11-llvm-gcc-4.2: no input files
Is that the default xcode/mac gcc compiler version? Because when I try to do a
sudo easy_install cython
I get:
Running Cython-0.15.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-qS3Kqb/Cython-0.15.1/egg-dist-tmp-Zh0Vnv
cc1: error: unrecognized command line option "-arch"
cc1: error: unrecognized command line option "-arch"
I've read that -arch is a Apple GCC compiler only function. I think when I installed a port from macports I remember it installing something called "llvm" and now I suspect that that is being used instead of the one that comes with XCODE.
Any way to switch it back?
Oh, and when I type "sudo port select gcc" I get (this might be relevent to knowing which gcc version I have):
Available versions for gcc:
apple-gcc42
gcc42
llvm-gcc42
mp-gcc44
mp-llvm-gcc42
none

Does sound like you're getting a non-apple version. If you don't need any non-standard compilers, I'd remove any that macports has installed. The apple infrastructure is different enough that using compilers from macports causes grief fairly easily.

This is not extremely related to your problem but you will find a solution here: Can't install Ruby under Lion with RVM – GCC issues
This answer was edited multiple times and now contains three alternative solutions. Skip to the end and try the simple “edit 3” solution first, it seems to work for most people.
You need a non-LLVM version of GCC, which is no longer included with XCode 4.2. Install it yourself (or downgrade to XCode 4.1 temporarily), then do CC=/usr/local/bin/gcc-4.2 rvm install 1.9.3 (substituting the path to your non-LLVM gcc).
Edit: https://github.com/kennethreitz/osx-gcc-installer/downloads may help for installing GCC.
Edit 2 (apparently the easiest solution): Alternatively you can try to add --with-gcc=clang to the arguments to configure for Ruby to use clang.
Edit 3: rvm install 1.9.3 --with-gcc=clang does that for you.

How do I stop Python install on Mac OS X from putting things in my home directory?

I'm trying to install Python from source on my Mac. (OS X 10.6.2, Python-2.6.5.tar.bz2) I've done this before and it was easy, but for some reason, this time after ./configure, and make, the sudo make install puts things some things in my home directory instead of in /usr/local/... where I expect. The .py files are okay, but not the .so files...
RobsMac Python-2.6.5 $ sudo make install
[...]
/usr/bin/install -c -m 644 ./Lib/anydbm.py /usr/local/lib/python2.6
/usr/bin/install -c -m 644 ./Lib/ast.py /usr/local/lib/python2.6
/usr/bin/install -c -m 644 ./Lib/asynchat.py /usr/local/lib/python2.6
[...]
running build_scripts
running install_lib
creating /Users/rob/Library/Python
creating /Users/rob/Library/Python/2.6
creating /Users/rob/Library/Python/2.6/site-packages
copying build/lib.macosx-10.4-x86_64-2.6/_AE.so -> /Users/rob/Library/
Python/2.6/site-packages
copying build/lib.macosx-10.4-x86_64-2.6/_AH.so -> /Users/rob/Library/
Python/2.6/site-packages
copying build/lib.macosx-10.4-x86_64-2.6/_App.so -> /Users/rob/Library/
Python/2.6/site-packages
[...]
Later, this causes imports that require those .so files to fail. For
example...
RobsMac Python-2.6.5 $ python
Python 2.6.5 (r265:79063, Apr 28 2010, 13:40:18)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import zlib
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named zlib
Any ideas what is wrong?
thanks,
Rob

Doh. I've answered my own question. Recently I created a ~/.pydistutils.cfg file, for some stupid reason. I forgot to delete that file. It's contents were:
[install]
install_lib = ~/Library/Python/$py_version_short/site-packages
install_scripts = ~/bin
make install calls setup.py, and this file was overriding the normal setup.py behavior.
Rob

In general, installing Python (or anything directly from the source) when it is already available on your system or when there are package managers that will install it for you, is not a very good idea. I strongly advise you against installing Python manually... Mac OS X 10.6 Snow Leopard comes with Python 2.6 out of the box; if you want a newer version of Python 2.6, then you should install MacPorts, and use:
sudo port install python26 python_select
You can then use the python_select to toggle between the system's version and MacPort's version.
If you are determined to install manually from the source, though, the way to do it would be to run "make distclean" (or untar the code separately again), then run "./configure --help" for a full list of configuration options. It is possible that on Mac OS X, it defaults to something other than /usr/local, in which case you could force it to install in that location by invoking configure with "./configure --prefix=/usr/local".

Have you checked the parameters or variables make expects? There probably is a make variable you can use to override that behavior. In any case, have you tried MacPorts? It may be a better solution to what you are trying to accomplish.