We Keep Coding

Cross compiling python for ARM from source

I am cross compiling python 3.7 for ARM using TI-SDK.
I have downloaded the source code on ubuntu and followed the below instructions
Set the path of SDK (export PATH = $PATH:)
./confgure
make
When I run make, I get the errors below. Can you please tell me how to resolve the linker errors?
*ar rc libpython3.5m.a Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o
ar rc libpython3.5m.a Modules/_threadmodule.o Modules/signalmodule.o Modules/posixmodule.o Modules/errnomodule.o Modules/pwdmodule.o Modules/_sre.o Modules/_codecsmodule.o Modules/_weakref.o Modules/_functoolsmodule.o Modules/_operator.o Modules/_collectionsmodule.o Modules/itertoolsmodule.o Modules/atexitmodule.o Modules/_stat.o Modules/timemodule.o Modules/_localemodule.o Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o Modules/zipimport.o Modules/faulthandler.o Modules/_tracemalloc.o Modules/hashtable.o Modules/symtablemodule.o Modules/xxsubtype.o
ranlib libpython3.5m.a
gcc -pthread -Xlinker -export-dynamic -o python Programs/python.o libpython3.5m.a -lpthread -ldl -lutil -lm
libpython3.5m.a(fileutils.o): In function `set_inheritable':
/home/pcadmin/work/OTIS/Python-3.5.9/Python/fileutils.c:892: undefined reference to `fcntl64'
/home/pcadmin/work/OTIS/Python-3.5.9/Python/fileutils.c:903: undefined reference to `fcntl64'
libpython3.5m.a(fileutils.o): In function `get_inheritable':
/home/pcadmin/work/OTIS/Python-3.5.9/Python/fileutils.c:777: undefined reference to `fcntl64'
/home/pcadmin/work/OTIS/Python-3.5.9/Python/fileutils.c:777: undefined reference to `fcntl64'
libpython3.5m.a(fileutils.o): In function `set_inheritable':
/home/pcadmin/work/OTIS/Python-3.5.9/Python/fileutils.c:892: undefined reference to `fcntl64'
libpython3.5m.a(fileutils.o):/home/pcadmin/work/OTIS/Python-3.5.9/Python/fileutils.c:903: more undefined references to `fcntl64' follow
collect2: error: ld returned 1 exit status
Makefile:555: recipe for target 'python' failed
make: *** [python] Error 1*
Please comment if more information is required to help me out with this issue.

I found the solution. Thought it may help others if i post it.
The linker error is due to the dependency on libxml2. I have cross compiled libxml2 and linked to python and it worked.
The configure script includes the path of libxml along with other options.
./configure CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar LD=arm-linux-gnueabihf-ld RANLIB=arm-linux-gnueabihf-ranlib --host=arm-linux-gnueabihf --target=arm -prefix=/home/sagar/otis/python3_install --without-sqlite3 --without-pdo-sqlite --without-pear --enable-simplexml --disable-mbregex --enable-sockets --enable-fpm --disable-opcache --enable-libxml --without-zlib --with-libxml-dir=/home/sagar/otis/libxml/_install/ --build=x86_64-linux-gnu --disable-all --disable-ipv6 ac_cv_have_long_long_format=yes ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no

Python / C++ binding, how to link agains static c++ library (portaudio) with distutils?

I am trying to staticaly link the "c++ portaudio library" against my "C++ demo module" which is a python callable library (module).
I'm doing this with distutils, and in order to perform the static linking, I've added the libportaudio to the extra_objects argument, as follows:
module1 = Extension(
"demo",
sources=cppc,
# TODO remove os dependency
extra_compile_args=gccArgs,
# link against shared libraries
#libraries=[""]
# link against static libraries
extra_objects=["./clib-3rd-portaudio/libportaudio.a"]) # << I've added the static lib here
Compiling with "python setup.py build" results in the following linker error:
/usr/bin/ld: ./clib-3rd-portaudio/libportaudio.a(pa_front.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
./clib-3rd-portaudio/libportaudio.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
So at this point I've tried the obvious, I've added the -fPIC flagg to gccArgs (note extra_compile_args=gccArgs above), as follows:
gccArgs = [
"-Icsrc",
"-Icsrc/paExamples",
"-Icinc-3rd-portaudio",
"-Icinc-3rd-portaudio/common",
"-Icinc-3rd-portaudio/linux",
"-fPIC"] # << I've added the -fPIC flag here
However this results in the exact same error, so I guess the -fPIC flag is not the root cause. I'm probably missing something trivial, but I'm a bit lost here, hope somebody can help.

As the error message said, you should recompile the external library libportaudio.a with -fPIC argument, NOT your own codes. That's why it doesn't help to add -fPIC to your extra_compile_args.
Several other posts suggest that the file libportaudio.a cannot be used to build shared library, probably because the default build settings of portaudio don't include -fPIC.
To recompile portaudio correctly, download the source and try to run ./configure with -shared option (or something similar). If you cannot find the proper option, then modify the Makefile and append -fPIC to the extra compile options. You can also compile each object file manually and pack them into libportaudio.a.
Since your target file (libdemo.so) is a shared library, you must make sure ANY object codes included inside are compiled with -fPIC option. To understand why you need this option, please refer to:
What does -fPIC mean when building a shared library? and Position Independent Code (PIC) in shared libraries

undefined symbol: PyOS_InputHook, from shared library

I wrote a C++ "python plugin" for an a non-python C++ application.
At some point, this plugin, which is a .so, initializes the python interpreter and opens a python console.
For convenience the "readline" module is then imported and we get this error:
ImportError: /usr/lib/python2.7/lib-dynload/readline.so: undefined symbol: PyOS_InputHook
The link command (generated by cmake) goes:
/usr/bin/c++ -fPIC -Wall -Wextra -O3 -DNDEBUG -Xlinker -export-dynamic -Wl,-fwhole-program /usr/lib/libpython2.7.a -shared -Wl,-soname,libMyplugin.so -o libMyplugin.so [sources] [qt libs] -lGLU -lGL -lX11 -lXext -lc -lc -lpython2.7 -Wl,-rpath,/src:/usr/local/Trolltech/Qt-4.8.4/lib:
nm libMyplugin.so gives the following python-related symbols:
U Py_Finalize
U Py_Initialize
00000000002114a8 B PyOS_InputHook
U PyRun_InteractiveLoopFlags
U PyRun_SimpleStringFlags
We observe that PyOS_InputHook is defined in the BSS section of the plugin. Yet, python's readline.so fails to find it.
The question is why, and how to fix it.

The issue is with how the main application loads the plugin: it uses dlopen() without the flag RTLD_GLOBAL.
This implies that the symbols present in the plugin that are not currently needed (like PyOS_InputHook in this instance) are not resolved and will not be resolved for other shared libraries that will be loaded afterwards (like readline.so in this instance).
To fix this, the flag RTLD_GLOBAL should be used when loading the plugin.
If there is no control over the main application (as in this instance) and on how it uses dlopen(), it is still possible to "reload" the plugin from within the plugin itself using dlopen() with flags RTLD_NOLOAD | RTLD_GLOBAL, so as to resolve all previously unresolved symbols in the currently loaded library.
Doing this solves the issue.

how to link a Python C module

I have written a Python C module (just ffmpeg.c which depends on some FFmpeg libs and other libs) and I am wondering how to link.
I'm compiling with:
cc -std=c99 -c ../ffmpeg.c -I /usr/include/python2.7 -g
I'm trying to link right now with:
ld -shared -o ../ffmpeg.so -L/usr/local/lib -lpython2.7 -lavutil -lavformat -lavcodec -lswresample -lportaudio -lchromaprint ffmpeg.o -lc
There is no error. However, when I try to import ffmpeg in Python, I get:
ImportError: ./ffmpeg.so: undefined symbol: avio_alloc_context
Maybe this is already correct. I checked the resulting ffmpeg.so with ldd and it partly links to a wrong FFmpeg. This is strange however because of the -L/usr/local/lib which should take precedence over the default. Maybe because my custom installed FFmpeg (in /usr/local/lib) has for some reason only installed static *.a libs and *.so files have precedence over *.a files.

You should put the libraries that you're linking to after the .o file; i.e.:
ld -shared -o ../ffmpeg.so ffmpeg.o -L/usr/local/lib -lpython2.7 -lavutil -lavformat -lavcodec -lswresample -lportaudio -lchromaprint -lc
The linker is dumb, and will not link in code from static libraries that it doesn't think are needed until a dependency arises i.e. the use of avio_alloc_context happens in ffmpeg.o, and because it's not listed after the use of the library, then the linker will not consider the code in the library as needed, so it doesn't get linked in - this is the biggest reason why linking using .a files fails.
You can also use --start-group and --end-group around all the files that you are linking - this allows you to link static libraries that have cross dependencies that just seem impossible to resolve through other means:
ld -shared -o ../ffmpeg.so -L/usr/local/lib -lpython2.7 --start-group -lavutil -lavformat -lavcodec -lswresample -lportaudio -lchromaprint ffmpeg.o --end-group -lc
using .a files is a little bit trickier than .so files, but these two items generally will work around any issues you have when linking.

How to compile .c code from Cython with gcc

Now that I've successfully installed Cython on Windows 7, I try to compile some Cython code using Cython, but gcc makes my life hard.
cdef void say_hello(name):
print "Hello %s" % name
Using gcc to compile the code throws dozens of undefined reference to -erros, and I'm pretty sure the libpython.a is available (as the installation tutorial said, undefined reference to -errors are thrown if this file is missing).
$ cython ctest.pyx
$ gcc ctest.c -I"C:\Python27\include"
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1038): undefined reference to `_imp__PyString_FromStringAndSize'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1075): undefined reference to `_imp___Py_TrueStruct'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1086): undefined reference to `_imp___Py_ZeroStruct'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x1099): undefined reference to `_imp___Py_NoneStruct'
C:\Users\niklas\AppData\Local\Temp\cckThGrF.o:ctest.c:(.text+0x10b8): undefined reference to `_imp__PyObject_IsTrue'
c:/program files/mingw/bin/../lib/gcc/mingw32/4.5.2/../../../libmingw32.a(main.o):main.c:(.text+0xd2): undefined reference to `WinMain#16'
collect2: ld returned 1 exit status
The weird thing is, using pyximport* or a setup-script works pretty fine, but it's both not very handy when still working on a module.
How to compile those .c files generated with Cython using gcc ?
or any other compiler, important is that it will work !
*pyximport: Is it normal that only python-native functions and classes are contained in the imported module and not cdef-functions and classes ?
like:
# filename: cython_test.pyx
cdef c_foo():
print "c_foo !"
def foo():
print "foo !"
c_foo()
import pyximport as p; p.install()
import cython_test
cython_test.foo()
# foo !\nc_foo !
cython_test.c_foo()
# AttributeError, module object has no attribute c_foo
UPDATE
Calling $ gcc ctest.c "C:\Python27\libs\libpython27.a" kills the undefined reference to -erros, but this one:
c:/program files/mingw/bin/../lib/gcc/mingw32/4.5.2/../../../libmingw32.a(main.o):main.c:(.text+0xd2): undefined reference to `WinMain#16'

Try:
gcc -c -IC:\Python27\include -o ctest.o ctest.c
gcc -shared -LC:\Python27\libs -o ctest.pyd ctest.o -lpython27
-shared creates a shared library. -lpython27 links with the import library C:\Python27\libs\libpython27.a.

That is a linker (ld) error and not a compiler error. You should provide the path to the library (-l and -L) and not only to the headers (-I).