When one uses pybind to create python-c++ bindings, upon compilation pybind creates a *.so file. AFAIK the compilation step in pybind just uses the c++ compiler, so this should be no different from just regular shared libs that one would create for a normal c++ code. How does the python interpreter introspect into these *.so files to notice that there are python-compatible modules in them?
Ultimately, you'll want to look at the CPython docs for how C extensions work. From the docs: https://docs.python.org/3/extending/building.html
A C extension for CPython is a shared library (e.g. a .so file on Linux, .pyd on Windows), which exports an initialization function.
As it says here, the primary difference is that it defines its initialization / entry point function.
All pybind does is wrap this entry point via PYBIND11_MODULE:
https://pybind11.readthedocs.io/en/stable/basics.html#creating-bindings-for-a-simple-function
https://github.com/pybind/pybind11/blob/25abf7e/include/pybind11/detail/common.h#L283
Related
Is it possible to modify Ctypes src code to support static libraries for Python (eg. I can modify Ctypes to add some additional funcs to invoke static library)?
Since for special reasons, I have to compile everything into one giant Python blob (including Python interpreter, and user Python .py programs(zip section)). Therefore, I cannot use separate .so libraries during my Python blob running.
BTW: Python C extension is not considered here, since my clients' code uses Ctypes to bridge there C code for Python. I don't want to change too much in their side.
I need to compile my c extension that is invokable by python in kodi. Can anyone please list the steps involved. I think I have actually cross compiled the c extension but it wouldn't work with kodi.
Sorry that I cannot provide you specific steps, but from what I know building binary modules for Kodi-Android is not a trivial task. Here's what I know:
You need to use Python.h from the Python sources used for Kodi's built-in interpreter.
You need to link against libkodi.so to find necessary Python symbols.
This is important: import mechanism for binary nodules in Kodi-Android is broken.
If you use:
import foo
Kodi-Android will actually search for libfoo.so because it automatically appends lib- when searches for shared library files and your import will fail. Simple renaming foo.so to libfoo.so won't help because the name must match the module declaration, for example:
PyMODINIT_FUNC
initspam(void)
{
(void) Py_InitModule("libfoo", SpamMethods);
}
Only if the module declaration "libfoo" matches the filename libfoo.so ("lib-" part is mandatory for Kodi-Android), then the import should succeed provided there are no other pitfall. As I've said, it's not a trivial task.
BTW, you can build a pure C shared library and use ctypes, and don't mess with all this broken Python-C modules stuff. Naturally, your library name must start with "lib-" (again, this is mandatory for Kodi-Android) but using shared libs via ctypes is easier provided your library doesn't have external dependencies.
UPD: I forgot about permission issues. Android does not allow to import binary modules from everywhere. Kodi's temporary directory is known to work but not always. Again, as far as binary Python modules concerned, Kodi-Android is a totall mess.
I am trying to import pygr:
It fails on:
>>> import seqfmt
ImportError: No module named seqfmt
The program that uses this works fine in Python. However its calling a C library called seqfmt (which has a C file and a PYX files). Is this possible to import over to Jython or since its C am I out of luck?
.PYX is the file extension used by cython, a tool for writing C extensions for python in a python-like syntax. Cython creates an intermediate file (that's presumably the .C file you see, at least it's not in the git repository) and compiles it into a python extension.
Jython does not support CPython extensions yet. From its homepage:
There are a number of differences. First, Jython programs cannot currently use CPython
extension modules written in C. These modules usually have files with the extension .so,
.pyd or .dll. If you want to use such a module, you should look for an
equivalent written in pure Python or Java. However, it is technically
feasible to support such extensions, as demonstrated by IronPython.
For the next release of Jython, we plan to support the C Python
Extension API.
Some cython modules can be easily translated into python, and seqfmt is one of them, but pygr has a second cython module, cnestedlist, which involves C calls: The lines
cdef extern from "apps/maf2nclist.h":
[..]
int readMAFrecord(IntervalMap im[],int n,SeqIDMap seqidmap[],int nseq,
int lpoStart,int *p_block_len,FILE *ifile,int maxseq,
long long linecode_count[],int *p_has_continuation)
define an external library call. You'd have to translate this library to Python, too.
Just as a side-note regarding the translation: cython can not only be used to wrap C libraries, but also to simply speed up certain parts of a program. In these cases it is pretty straight-forward to translate them into a python module. Have a look into the seqfmt.pyx source file, its pretty self-explanatory if you know python.
That all being said, there is a project related to Jython, JyNI, which aims to support CPython extensions from Jython. It is work-in-progress, so I can't tell if your libraries are supported by it. There are some examples in the github repository, maybe you can get it to work. The readme file claims binary compatibility, so with JyNI enabled you should be able to run your code without any recompiling.
I currently have an executable compiled from C++ that embeds python. The embedded executable runs a python script which load several Cython modules. Both the Cython modules and the executable are linked against a shared library.
I want to move the shared library into the executable by statically linking the shared library against the executable.
Can I statically link the Cython modules into the executable which embeds python? What is the best way to handle this situation?
Yes it's possible, but if you have an hand on the python interpreter. What i'm going to describe have been done for python on IOS platform. You need to check more how to let python known about your module if you don't want to touch on the original python interpreter (Replace TEST everywhere with your own tag/libname)
One possible way to do it is:
Compile your own python with a dynload patch that prefer to not dlopen() your module, but use directly dlsym() to check if the module is already in memory.
Create an libTEST.a, including all the .o generated during the build process (not the .so). You can found it usually in the build/temp.*, and do something like this:
ar rc libTEST.a build/temp.*/*.o
ranlib libTEST.a
When compiling the main executable, you need to add a dependency to that new libTEST.a by appending in the compilation command line:
-lTEST -L.
The result will give you an executable with all the symbol from your cython modules, and python will be able to search them in memory.
(As an example, I'm using an enhanced wrapper that redirect ld during compilation to not produce .so, and create a .a at the end. On the kivy-ios project, you can grab liblink that is used to produce .o, and biglink that is used to grab all the .o in directories and produce .a. You can see how it's used in build_kivy.sh)
I'm trying to compile python source code foo.py to C using cython.
In foo.py:
print "Hello World"
The command I'm running is cython foo.py.
The problem is that when compiling foo.c using gcc, I get the error:
undefined reference to 'main'.
when converting the code from python to c (using Cython) it converts it to c code which can be compiled into a shared object.
in order to make it executable, you should add "--embed" to cython conversion command. this flag adds the 'main' function you need, so you could compile the c code into executable file.
please notice you'll need the python .so runtime libraries in order to run the exec.
Read the Cython documentation. This will also (hopefully) teach you what Cython is and what it isn't. Cython is for creating python extensions (not a general-purpose Python-to-C-compiler), which are shared objects/dlls. Dynamically loaded libraries don't have a main function like standalone programs, but compilers assume that they are ultimately linking an executable. You have to tell them otherwise via flags (-shared methinks, but again, refer to the Cython documentation) - or even better, don't compile yourself, use a setup.py for this (yet again, read the Cython documentation).
The usual way is to use distutils to compile the cython-generated file. This also gives you all the include directories you need in a portable way.