We Keep Coding

scipy + (numpy.distutils-based) setup.py + ctypes: undefined symbol

Intro
As a follow-up to my earlier question here, i still try to:
add new functionality to one of scipy's modules (e.g. scipy.optimize)
based on C++-code
which defines an interface-function to be called with ctypes
automatically setup through setup.py in scipy/optimize
Code incorporation
To keep it simple, let's assume we got the following (dumb and) simple C++-code:
extern "C" void hallo()
{
int a = 0;
}
where we use the extern keyword to handle name-mangling.
We introduce new files and folders in scipy.optimize:
scipy/optimize/__init__py
scipy/optimize/lbfgsb/...
scipy/optimize/lbfgsb.py
...
scipy/optimize/_mylib/README.md # new
scipy/optimize/_mylib/LICENSE # new
scipy/optimize/_mylib/src/debug.cpp # new
scipy/optimize/mylib.py # new
Setup preparation
We prepare the module-setup in scipy/optimize/setup.py:
from __future__ import division, print_function, absolute_import
from os.path import join
from scipy._build_utils import numpy_nodepr_api
def configuration(parent_package='',top_path=None):
from numpy.distutils.misc_util import Configuration
from numpy.distutils.system_info import get_info
config = Configuration('optimize',parent_package, top_path)
# MODIFICATION START
# OTHER EXTENSIONS OMITTED
# INSPIRED BY scipy.spatial: ckdtree/src & setup.py
mylib_src = ['debug.cpp']
mylib_src = [join('_mylib', 'src', x) for x in mylib_src]
mylib_headers = []
mylib_headers = [join('_mylib', 'src', x) for x in mylib_headers]
mylib_dep = mylib_headers + mylib_src
config.add_extension('_mylib',
sources=mylib_src,
depends=mylib_dep,
include_dirs=[join('_mylib', 'src')])
# MODIFICATION END
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(**configuration(top_path='').todict())
Run scipy install
Installing scipy now (from base-dir) with:
python3 setup.py build_ext --inplace
works and we will see those shared-libraries in scipy/optimize/:
_lbfgsb.cpython-35m-x86_64-linux-gnu.so
...
_mylib.cpython-35m-x86_64-linux-gnu.so
Let's try to use it / a look into scipy/optimize/mylib.py
As seen in the other question, we can get the lib with (we are in scipy/optimize/mylib.py) and get the function we want to use:
import scipy as scp
import numpy.ctypeslib as ctl
lib = ctl.load_library('_mylib', scp.optimize.__file__)
myfunc = lib.hallo
Now here is the problem: this fails with:
AttributeError: /.../_mylib.cpython-35m-x86_64-linux-gnu.so: undefined symbol: hallo
Trying it per hand:
import ctypes
lib = ctypes.CDLL('full path to above so')
myfun = lib.hallo
fails too.
But, checking (on my linux OS): nm --defined-only _mylib.cpython-35m-x86_64-linux-gnu.so outputs:
...
...
0000000000000530 t hallo
...
which should be ok. (Disclaimer: i absolutely miss any knowledge about C++ linking). Edit: maybe it's not okay. See later observation in regards to t vs. T!
Doing it manually: this works
Going to scipy/optimize/_mylib/src:
g++ -shared -fPIC debug.cpp -o mylib.so
followed by nm --defined-only mylib.so shows the same functions, but some t became T.
...
...
0000000000000600 T hallo
...
Probably this is the reason, and there is some related general question here.
As mentioned, this works:
# in src
import numpy.ctypeslib as ctl
lib = ctl.load_library('mylib.so', '.')
lib.hallo
# < FuncPtr object at 0x....
But what needs to be done in setup.py or my sources to make it work?
During running scipy-install, output looks like:
building 'scipy.optimize._mylib' extension
compiling C++ sources
C compiler: x86_64-linux-gnu-g++ -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
creating build/temp.linux-x86_64-3.5/scipy/optimize/_mylib
creating build/temp.linux-x86_64-3.5/scipy/optimize/_mylib/src
compile options: '-Iscipy/optimize/_mylib/src -I/usr/local/lib/python3.5/dist-packages/numpy-1.15.0.dev0+e4d678a-py3.5-linux-x86_64.egg/numpy/core/include -I/usr/include/python3.5m -c'
x86_64-linux-gnu-g++: scipy/optimize/_mylib/src/debug.cpp
scipy/optimize/_mylib/src/debug.cpp: In function ‘void hallo()’:
scipy/optimize/_mylib/src/debug.cpp:3:9: warning: unused variable ‘a’ [-Wunused-variable]
int a = 0;
^
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/scipy/optimize/_mylib/src/debug.o -Lbuild/temp.linux-x86_64-3.5 -o scipy/optimize/_mylib.cpython-35m-x86_64-linux-gnu.so -Wl,--version-script=build/temp.linux-x86_64-3.5/link-version-scipy.optimize._mylib.map

Why am I getting this error when building the python module? [duplicate]

This question already has an answer here:
Build Python (2.7) module on GCC 4.8 fails
(1 answer)
Closed 7 years ago.
I am currently working on building a python module in C. I have followed a tutorial series to get my self to a building point, and I have a pretty good grasp of what is going on.
However, when I run the script to set everything up I have the following printout. Looks like the error is coming from one of the header files in python-dev; however, I wanted to reach out here to make sure it isn't something I am doing. I am going to include the error code, and I can provide the other files on request.
sudo python setup.py build
running build
running build_ext
building 'lidar' extension
arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/local/include -I/usr/include/python2.7 -c lidar_module.c -o build/temp.linux-armv6l-2.7/lidar_module.o
lidar_module.c: In function ‘lidar_test’:
lidar_module.c:8:6: warning: unused variable ‘sts’ [-Wunused-variable]
int sts = 0;
^
In file included from /usr/include/python2.7/Python.h:58:0,
from lidar_module.c:1:
lidar_module.c: At top level:
/usr/include/python2.7/pyport.h:802:39: error: expected ‘,’ or ‘;’ before ‘void’
# define PyMODINIT_FUNC void
^
lidar_module.c:24:1: note: in expansion of macro ‘PyMODINIT_FUNC’
PyMODINIT_FUNC initlidar(){
^
lidar_module.c:3:18: warning: ‘lidarError’ defined but not used [-Wunused-variable]
static PyObject *lidarError;
^
lidar_module.c:19:20: warning: ‘lidar_methods’ defined but not used [-Wunused-variable]
static PyMethodDef lidar_methods[] = {
^
error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1
I peeked into the header file and found this around the area of the error.
#ifndef PyMODINIT_FUNC
# if defined(__cplusplus)
# fndef PyMODINIT_FUNC
# if defined(__cplusplus)
# define PyMODINIT_FUNC extern "C" void
# else /* __cplusplus */
# define PyMODINIT_FUNC void
# endif /* __cplusplus */
#endif

I literally had to search the line number output from the console, but I did find the solution here:
Build Python (2.7) module on GCC 4.8 fails

how to link a Python C module

I have written a Python C module (just ffmpeg.c which depends on some FFmpeg libs and other libs) and I am wondering how to link.
I'm compiling with:
cc -std=c99 -c ../ffmpeg.c -I /usr/include/python2.7 -g
I'm trying to link right now with:
ld -shared -o ../ffmpeg.so -L/usr/local/lib -lpython2.7 -lavutil -lavformat -lavcodec -lswresample -lportaudio -lchromaprint ffmpeg.o -lc
There is no error. However, when I try to import ffmpeg in Python, I get:
ImportError: ./ffmpeg.so: undefined symbol: avio_alloc_context
Maybe this is already correct. I checked the resulting ffmpeg.so with ldd and it partly links to a wrong FFmpeg. This is strange however because of the -L/usr/local/lib which should take precedence over the default. Maybe because my custom installed FFmpeg (in /usr/local/lib) has for some reason only installed static *.a libs and *.so files have precedence over *.a files.

You should put the libraries that you're linking to after the .o file; i.e.:
ld -shared -o ../ffmpeg.so ffmpeg.o -L/usr/local/lib -lpython2.7 -lavutil -lavformat -lavcodec -lswresample -lportaudio -lchromaprint -lc
The linker is dumb, and will not link in code from static libraries that it doesn't think are needed until a dependency arises i.e. the use of avio_alloc_context happens in ffmpeg.o, and because it's not listed after the use of the library, then the linker will not consider the code in the library as needed, so it doesn't get linked in - this is the biggest reason why linking using .a files fails.
You can also use --start-group and --end-group around all the files that you are linking - this allows you to link static libraries that have cross dependencies that just seem impossible to resolve through other means:
ld -shared -o ../ffmpeg.so -L/usr/local/lib -lpython2.7 --start-group -lavutil -lavformat -lavcodec -lswresample -lportaudio -lchromaprint ffmpeg.o --end-group -lc
using .a files is a little bit trickier than .so files, but these two items generally will work around any issues you have when linking.

What is Python's equivalent of "perl -V"

The output produced by running perl -V is packed with useful information (see example below). Is there anything like it for Python?
Example output:
% perl -V
Summary of my perl5 (revision 5 version 10 subversion 1) configuration:
Platform:
osname=linux, osvers=2.6.32-5-amd64, archname=x86_64-linux-gnu-thread-multi
uname='linux brahms 2.6.32-5-amd64 #1 smp tue jun 14 09:42:28 utc 2011 x86_64 gnulinux '
config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.1 -Dsitearch=/usr/local/lib/perl/5.10.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.10.1 -Dd_dosuid -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -g',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.4.5', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64
libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt
libc=/lib/libc-2.11.2.so, so=so, useshrplib=true, libperl=libperl.so.5.10.1
gnulibc_version='2.11.2'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector'
Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_64_BIT_ALL
USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
USE_PERLIO USE_REENTRANT_API
Locally applied patches:
DEBPKG:debian/arm_thread_stress_timeout - http://bugs.debian.org/501970 Raise the timeout of ext/threads/shared/t/stress.t to accommodate slower build hosts
DEBPKG:debian/cpan_config_path - Set location of CPAN::Config to /etc/perl as /usr may not be writable.
<snip-- iow patches galore --you get the picture>
DEBPKG:fixes/safe-reval-rdo-cve-2010-1447 - [PATCH] Wrap by default coderefs returned by rdo and reval
DEBPKG:patchlevel - http://bugs.debian.org/567489 List packaged patches for 5.10.1-17squeeze2 in patchlevel.h
Built under linux
Compiled at Jun 30 2011 22:28:00
#INC:
/etc/perl
/usr/local/lib/perl/5.10.1
/usr/local/share/perl/5.10.1
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.10
/usr/share/perl/5.10
/usr/local/lib/site_perl
/usr/local/lib/perl/5.10.0
/usr/local/share/perl/5.10.0
.
Not to be confused with the much less informative perl -v:
% perl -v
This is perl, v5.10.1 (*) built for x86_64-linux-gnu-thread-multi
(with 53 registered patches, see perl -V for more detail)
Copyright 1987-2009, Larry Wall
Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.
Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl". If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

python -c 'import sysconfig, pprint; pprint.pprint(sysconfig.get_config_vars())'

Although this is incredibly hackish, impractical, and not as detailed as perl -V, this is a one-liner that can get decent information about the environment.
python -c "import platform as p;exec('for x in vars(p):\n try:\n print ({x:vars(p)[x]()})\n except:\n pass')"
Since this is not your typical easy-to-remember command, you could save this line to Python's Lib directory as sys_info.py and then you could just run:
python -m sys_info

How to compile .c code from Cython with gcc

Now that I've successfully installed Cython on Windows 7, I try to compile some Cython code using Cython, but gcc makes my life hard.
cdef void say_hello(name):
print "Hello %s" % name
Using gcc to compile the code throws dozens of undefined reference to -erros, and I'm pretty sure the libpython.a is available (as the installation tutorial said, undefined reference to -errors are thrown if this file is missing).
$ cython ctest.pyx
$ gcc ctest.c -I"C:\Python27\include"
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1038): undefined reference to `_imp__PyString_FromStringAndSize'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1075): undefined reference to `_imp___Py_TrueStruct'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1086): undefined reference to `_imp___Py_ZeroStruct'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1099): undefined reference to `_imp___Py_NoneStruct'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x10b8): undefined reference to `_imp__PyObject_IsTrue'
c:/program files/mingw/bin/../lib/gcc/mingw32/4.5.2/../../../libmingw32.a(main.o):main.c:(.text+0xd2): undefined reference to `WinMain#16'
collect2: ld returned 1 exit status
The weird thing is, using pyximport* or a setup-script works pretty fine, but it's both not very handy when still working on a module.
How to compile those .c files generated with Cython using gcc ?
or any other compiler, important is that it will work !
*pyximport: Is it normal that only python-native functions and classes are contained in the imported module and not cdef-functions and classes ?
like:
# filename: cython_test.pyx
cdef c_foo():
print "c_foo !"
def foo():
print "foo !"
c_foo()
import pyximport as p; p.install()
import cython_test
cython_test.foo()
# foo !\nc_foo !
cython_test.c_foo()
# AttributeError, module object has no attribute c_foo
UPDATE
Calling $ gcc ctest.c "C:\Python27\libs\libpython27.a" kills the undefined reference to -erros, but this one:
c:/program files/mingw/bin/../lib/gcc/mingw32/4.5.2/../../../libmingw32.a(main.o):main.c:(.text+0xd2): undefined reference to `WinMain#16'

Try:
gcc -c -IC:\Python27\include -o ctest.o ctest.c
gcc -shared -LC:\Python27\libs -o ctest.pyd ctest.o -lpython27
-shared creates a shared library. -lpython27 links with the import library C:\Python27\libs\libpython27.a.

That is a linker (ld) error and not a compiler error. You should provide the path to the library (-l and -L) and not only to the headers (-I).