We Keep Coding

How does distutils determine what binary files to copy?

I have a setup command defined like this for distutils (using py2app for Mac OS X, if it matters):
setup(...,
extensions=Extension('tracking_funcs',
['tracking_funcs/tracking_funcs.pyx'],
include_dirs=[numpyincludedirs,]),
Extension('_psutil_osx',
sources = ['psutil/_psutil_osx.c',
'psutil/_psutil_common.c',
'psutil/arch/osx/process_info.c'],
define_macros=[('PSUTIL_VERSION', int(get_psutilver().replace('.', '')))],
extra_link_args=['-framework', 'CoreFoundation',
'-framework', 'IOKit']),
Extension('_psutil_posix',
sources = ['psutil/_psutil_posix.c'])],
...)
It builds all three extensions correctly:
...
building 'tracking_funcs' extension
creating build/bdist.macosx-10.6-x86_64/temp.macosx-10.6-x86_64-2.7/tracking_funcs
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c tracking_funcs/tracking_funcs.c -o build/bdist.macosx-10.6-x86_64/temp.macosx-10.6-x86_64-2.7/tracking_funcs/tracking_funcs.o
... (some compiler warnings) ...
42 warnings generated.
/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g build/bdist.macosx-10.6-x86_64/temp.macosx-10.6-x86_64-2.7/tracking_funcs/tracking_funcs.o -o build/bdist.macosx-10.6-x86_64/lib.macosx-10.6-x86_64-2.7/tracking_funcs.so
building '_psutil_osx' extension
creating build/bdist.macosx-10.6-x86_64/temp.macosx-10.6-x86_64-2.7/psutil
... (some compiler warnings) ...
2 warnings generated.
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSUTIL_VERSION=400 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c psutil/_psutil_common.c -o build/bdist.macosx-10.6-x86_64/temp.macosx-10.6-x86_64-2.7/psutil/_psutil_common.o
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSUTIL_VERSION=400 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c psutil/arch/osx/process_info.c -o build/bdist.macosx-10.6-x86_64/temp.macosx-10.6-x86_64-2.7/psutil/arch/osx/process_info.o
/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g build/bdist.macosx-10.6-x86_64/temp.macosx-10.6-x86_64-2.7/psutil/_psutil_osx.o build/bdist.macosx-10.6-x86_64/lib.macosx-10.6-x86_64-2.7/_psutil_osx.so -framework CoreFoundation -framework IOKit
building '_psutil_posix' extension
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c psutil/_psutil_posix.c -o build/bdist.macosx-10.6-x86_64/temp.macosx-10.6-x86_64-2.7/psutil/_psutil_posix.o
/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g build/bdist.macosx-10.6-x86_64/temp.macosx-10.6-x86_64-2.7/psutil/_psutil_posix.o -o build/dist.macosx-10.6-x86_64/lib.macosx-10.6-x86_64-2.7/_psutil_posix.so
...
Then it copies the built binaries into the package destination, but it only copies two of the three extensions:
...
copying file /.../build/bdist.macosx-10.6-x86_64/lib.macosx-10.6-x86_64-2.7/tracking_funcs.so -> /.../dist/my.app/Contents/Resources/lib/python2.7/lib-dynload/tracking_funcs.so
copying file /.../build/bdist.macosx-10.6-x86_64/lib.macosx-10.6-x86_64-2.7/_psutil_posix.so -> /.../dist/my.app/Contents/Resources/lib/python2.7/lib-dynload/_psutil_posix.so
...
Then my application crashes because it can't find the third extension at runtime.
How can I debug this? Where does distutils get its dependency tree from, if not from the list of extensions I define? Perhaps the bug is in py2app rather than distutils itself?

Py2app uses modulegraph to recursively build a source tree containing all the dependencies of all the dependencies of your project. Running py2app with the debug-modulegraph flag set will print a bunch of debugging information to the console and drop into a breakpoint that allow you to browse the contents of the module graph:
$ python setup.py py2app --debug-modulegraph
...
(Pdb) for item in mf.flatten(): print item
This will probably show something like (MissingModule) psutil._psutil_osx in its output, which means modulegraph isn't able to find the import path for that extension.
Modulegraph exposes a public function called addPackagePath that will allow you to give it additional hints as to where it should look for files in particular packages. In this case, adding something like this in setup.py should resolve the issue:
from modulegraph import modulegraph
modulegraph.addPackagePath('psutil', 'build/bdist.macosx-10.6-x86_64/lib.macosx-10.6-x86_64-2.7/psutil/')

Error installing PyLucene JCC on OSX

I am having trouble installing PyLucene JCC on Mac OSX.
I dowloaded the latest version of JCC from apache lucene site and fallowed the instruction to install it using the command
python setup.py build
But I am getting the following error while installing
ld: internal error: atom not found in symbolIndex(__ZN7JNIEnv_13CallIntMethodEP8_jobjectP10_jmethodIDz) for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'c++' failed with exit status 1
I have tried following the instruction given on the official site of adding the flag "-framework", "Python" to the LFLAGS value when installing on mac OSX or using the export CFLAGS=-Qunused-arguments and export CPPFLAGS=-Qunused-arguments before runnig the install command to ignore the warnings while build. But none of them are working.
The full trace of error is
found JAVAHOME = /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home
found JAVAFRAMEWORKS = /System/Library/Frameworks/JavaVM.framework
Loading source files for package org.apache.jcc...
doc/serialized-form.html...
Building index for all the packages and classes...
Generating javadoc/overview-tree.html...
Generating javadoc/index-all.html...
Generating javadoc/deprecated-list.html...
Building index for all classes...
Generating javadoc/allclasses-frame.html...
Generating javadoc/allclasses-noframe.html...
Generating javadoc/index.html...
Generating javadoc/help-doc.html...
running build
running build_py
writing /Users/harshsingh/Documents/Codes/IR/jcc/jcc/config.py
copying jcc/config.py -> build/lib.macosx-10.10-intel-2.7/jcc
copying jcc/classes/org/apache/jcc/PythonVM.class -> build/lib.macosx-10.10-intel-2.7/jcc/classes/org/apache/jcc
copying jcc/classes/org/apache/jcc/PythonException.class -> build/lib.macosx-10.10-intel-2.7/jcc/classes/org/apache/jcc
running build_ext
building 'jcc' extension
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX - DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -dynamiclib -D_jcc_lib -DJCC_VER="2. 21" -I/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/jdk1.7.0_45. jdk/Contents/Home/include/darwin -I_jcc -Ijcc/sources -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c jcc/sources/jcc.cpp - o build/temp.macosx-10.10-intel-2.7/jcc/sources/jcc.o -DPYTHON -fno-strict-aliasing -Wno-write-strings
clang: warning: argument unused during compilation: '-dynamiclib'
jcc/sources/jcc.cpp:197:16: warning: implicit conversion loses integer precision: 'long' to 'int'
[-Wshorten-64-to-32]
int hash = PyObject_Hash(arg);
~~~~ ^~~~~~~~~~~~~~~~~~
1 warning generated.
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX - DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -dynamiclib -D_jcc_lib -DJCC_VER="2. 21" -I/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/jdk1.7.0_45. jdk/Contents/Home/include/darwin -I_jcc -Ijcc/sources -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c jcc/sources/JCCEnv. cpp -o build/temp.macosx-10.10-intel-2.7/jcc/sources/JCCEnv.o -DPYTHON -fno-strict-aliasing -Wno-write-strings
clang: warning: argument unused during compilation: '-dynamiclib'
c++ -Wl,-x -dynamiclib -undefined dynamic_lookup build/temp.macosx-10.10-intel-2.7/jcc/sources/jcc.o build/temp.macosx-10.10-intel-2.7/jcc/sources/JCCEnv. o -o build/lib.macosx-10.10-intel-2.7/libjcc.dylib -L/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre/lib -ljava - L/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre/lib/server -ljvm -Wl,-rpath -Wl,/Library/Java/JavaVirtualMachines/jdk1.7.0_45. jdk/Contents/Home/jre/lib -Wl,-rpath -Wl,/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre/lib/server -Wl,-S -install_name #rpath/libjcc. dylib -current_version 2.21 -compatibility_version 2.21
ld: internal error: atom not found in symbolIndex(__ZN7JNIEnv_13CallIntMethodEP8_jobjectP10_jmethodIDz) for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'c++' failed with exit status 1
I am really lost now and any help will be really great.

If rebuilding from source still ends up giving the same errors, you could probably try the following approach.
Seems like there is a bug with running the clang++ command (below) with the -x flag, on OSX 10.9+ (referenced here)
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX - DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -dynamiclib -D_jcc_lib -DJCC_VER="2. 21" -I/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/jdk1.7.0_45. jdk/Contents/Home/include/darwin -I_jcc -Ijcc/sources -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c jcc/sources/JCCEnv. cpp -o build/temp.macosx-10.10-intel-2.7/jcc/sources/JCCEnv.o -DPYTHON -fno-strict-aliasing -Wno-write-strings
clang: warning: argument unused during compilation: '-dynamiclib'
c++ -Wl,-x -dynamiclib -undefined dynamic_lookup build/temp.macosx-10.10-intel-2.7/jcc/sources/jcc.o build/temp.macosx-10.10-intel-2.7/jcc/sources/JCCEnv. o -o build/lib.macosx-10.10-intel-2.7/libjcc.dylib -L/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre/lib -ljava - L/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre/lib/server -ljvm -Wl,-rpath -Wl,/Library/Java/JavaVirtualMachines/jdk1.7.0_45. jdk/Contents/Home/jre/lib -Wl,-rpath -Wl,/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre/lib/server -Wl,-S -install_name #rpath/libjcc. dylib -current_version 2.21 -compatibility_version 2.21
Run the clang++ command again without the -x flag and then run:
python setup.py build

Same problem here - from pyLucene-dev:
You must ensure that the compiler used to build Python is the same as the compiler you're using. If you did not build Python then it's likely you're htting a clang/gcc mismatch and you need to rebuild Python from sources, using the same compiler as you're using to build JCC.

The clang problem is fixed as given here. However, its not pushed in the PyLucene latest(as of May 2018) version(v6.5). However, you can still get the next build(v7.2) of Pylucene from this link. Using Pylucene 7.2.0 fixed the issue for me.

Upgrading scipy on Mac OS Mavericks 10.9.2

I'm trying to upgrade the scipy (currently I have version '0.11.0') on my machine, but I'm getting some compiler error that I can not work around:
When I invoke:
sudo pip install --upgrade scipy
I get:
cc: scipy/optimize/Zeros/bisect.c
clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
error: Command "cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/include -c scipy/optimize/Zeros/bisect.c -o build/temp.macosx-10.9-intel-2.7/scipy/optimize/Zeros/bisect.o" failed with exit status 1
Any help is very welcomed!!!

Try:
sudo $ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install --upgrade scipy

Error compiling pyGraphviz on Os X 10.9

Good day to everyone,
I've an error trying to compile and install Pygraphviz 1.2 in OS X 10.9
I've downloaded the pygraphviz archive from https://pypi.python.org/packages/source/p/pygraphviz/pygraphviz-1.2.zip#md5=90c728a8db276eede4e3af2f990a8985
From Terminal I've typed
sudo ./setup.py install
the output was:
Trying pkg-config
library_path=/usr/local/Cellar/graphviz/2.38.0/lib
include_path=/usr/local/Cellar/graphviz/2.38.0/include/graphviz
running install
running build
running build_py
running build_ext
building 'pygraphviz._graphviz' extension
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -I/usr/local/Cellar/graphviz/2.38.0/include/graphviz -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c pygraphviz/graphviz_wrap.c -o build/temp.macosx-10.9-intel-2.7/pygraphviz/graphviz_wrap.o
clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
error: command 'cc' failed with exit status 1
As you can see I've already installed graphviz with Homebrew.

The error is in clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
Solution from clang error: unknown argument: '-mno-fused-madd' (python package installation failure) is:
export CFLAGS=-Qunused-arguments
export CPPFLAGS=-Qunused-arguments

cython install failed (-mno-fused-madd unknown argument)

reating build/temp.macosx-10.9-intel-2.7
creating build/temp.macosx-10.9-intel-2.7/Cython
creating build/temp.macosx-10.9-intel-2.7/Cython/Plex
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c Cython/Plex/Scanners.c -o build/temp.macosx-10.9-intel-2.7/Cython/Plex/Scanners.o
clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
error: command 'cc' failed with exit status 1
I got this error while installing Cython from setup.py.
I was installing the scikit-learn package while get the same error. (-mno-fused-madd unknown argument)
Is there anything, any package needed before I install cython?
Many thanks

You need probably need to install another C compiler. Try compiling it with GCC.
clang: error: unknown argument: '-mno-fused-madd'
Alternatively, you can edit the configure script and change the compiler flags.
On a side note, this seems like it is a bug you should report.