We Keep Coding

Python Initialization fails with dynamically loaded DLL [duplicate]

I have a C++ program and it has sort of plugin structure: when program starts up, it's looking for dll in the plugin folder with certain exported function signatures, such as:
void InitPlugin(FuncTable* funcTable);
Then the program will call the function in the dll to initialize and pass function pointers to the dll. From that time on, the dll can talk to the program.
I know Cython let you call C function in Python, but I'm not sure can I write a Cython code and compile it to a dll so my C++ program can initialize with it. An example code would be great.

Using cython-module in a dll is not unlike using a cython-module in an embeded python interpreter.
The first step would be to mark cdef-function which should be used from external C-code with public, for example:
#cyfun.pyx:
#doesn't need python interpreter
cdef public int double_me(int me):
return 2*me;
#needs initialized python interpreter
cdef public void print_me(int me):
print("I'm", me);
cyfun.c and cyfun.h can be generated with
cython -3 cyfun.pyx
These files will be used for building of the dll.
The dll will need one function to initialize the python interpreter and another to finalize it, which should be called only once before double_me and print_me can be used (Ok, double_me would work also without interpreter, but this is an implementation detail). Note: the initialization/clean-up could be put also in DllMain - see such a version further bellow.
The header-file for the dll would look like following:
//cyfun_dll.h
#ifdef BUILDING_DLL
#define DLL_PUBLIC __declspec(dllexport)
#else
#define DLL_PUBLIC __declspec(dllimport)
#endif
//return 0 if everything ok
DLL_PUBLIC int cyfun_init();
DLL_PUBLIC void cyfun_finalize();
DLL_PUBLIC int cyfun_double_me(int me);
DLL_PUBLIC void cyfun_print_me(int me);
So there are the necessary init/finalize-functions and the symbols are exported via DLL_PUBLIC (which needs to be done see this SO-post) so it can be used outside of the dll.
The implementation follows in cyfun_dll.c-file:
//cyfun_dll.c
#define BUILDING_DLL
#include "cyfun_dll.h"
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "cyfun.h"
DLL_PUBLIC int cyfun_init(){
int status=PyImport_AppendInittab("cyfun", PyInit_cyfun);
if(status==-1){
return -1;//error
}
Py_Initialize();
PyObject *module = PyImport_ImportModule("cyfun");
if(module==NULL){
Py_Finalize();
return -1;//error
}
return 0;
}
DLL_PUBLIC void cyfun_finalize(){
Py_Finalize();
}
DLL_PUBLIC int cyfun_double_me(int me){
return double_me(me);
}
DLL_PUBLIC void cyfun_print_me(int me){
print_me(me);
}
Noteworthy details:
we define BUILDING_DLL so DLL_PUBLIC becomes __declspec(dllexport).
we use cyfun.h generated by cython from cyfun.pyx.
cyfun_init inizializes python interpreter and imports the built-in module cyfun. The somewhat complicated code is because since Cython-0.29, PEP-489 is default. More information can be found in this SO-post. If the Python-interpreter isn't initialized or if the module cyfun is not imported, the chances are high, that calling the functionality from cyfun.h will end in a segmentation fault.
cyfun_double_me just wraps double_me so it becomes visible outside of the dll.
Now we can build the dll!
:: set up tool chain
call "<path_to_vcvarsall>\vcvarsall.bat" x64
:: build cyfun.c generated by cython
cl /Tccyfun.c /Focyfun.obj /c <other_coptions> -I<path_to_python_include>
:: build dll-wrapper
cl /Tccyfun_dll.c /Focyfun_dll.obj /c <other_coptions> -I<path_to_python_include>
:: link both obj-files into a dll
link cyfun.obj cyfun_dll.obj /OUT:cyfun.dll /IMPLIB:cyfun.lib /DLL <other_loptions> -L<path_to_python_dll>
The dll is now built, but the following details are noteworthy:
<other_coptions> and <other_loptions> can vary from installation to installation. An easy way is to see them is to run cythonize some_file.pyx` and to inspect the log.
we don't need to pass python-dll, because it will be linked automatically, but we need to set the right library-path.
we have the dependency on the python-dll, so later on it must be somewhere where it can be found.
Were you go from here depends on your task, we test our dll with a simple main:
//test.c
#include "cyfun_dll.h"
int main(){
if(0!=cyfun_init()){
return -1;
}
cyfun_print_me(cyfun_double_me(2));
cyfun_finalize();
return 0;
}
which can be build via
...
:: build main-program
cl /Tctest.c /Focytest.obj /c <other_coptions> -I<path_to_python_include>
:: link the exe
link test.obj cyfun.lib /OUT:test_prog.exe <other_loptions> -L<path_to_python_dll>
And now calling test_prog.exe leads to the expected output "I'm 4".
Depending on your installation, following things must be considered:
test_prog.exe depends on pythonX.Y.dll which should be somewhere in the path so it can be found (the easiest way is to copy it next to the exe)
The embeded python interpreter needs an installation, see this and/or this SO-posts.
IIRC, it is not a great idea to initialize, then to finalize and then to initialize the Python-interpreter again (that might work for some scenarios, but not all , see for example this) - the interpreter should be initialized only once and stay alive until the programs ends.
Thus, it may make sense to put initialization/clean-up code into DllMain (and make cyfun_init() and cyfun_finalize() private), e.g.
BOOL WINAPI DllMain(
HINSTANCE hinstDLL, // handle to DLL module
DWORD fdwReason, // reason for calling function
LPVOID lpReserved ) // reserved
{
// Perform actions based on the reason for calling.
switch( fdwReason )
{
case DLL_PROCESS_ATTACH:
return cyfun_init()==0;
case DLL_PROCESS_DETACH:
cyfun_finalize();
break;
case DLL_THREAD_ATTACH:
// Do thread-specific initialization.
break;
case DLL_THREAD_DETACH:
// Do thread-specific cleanup.
break;
}
return TRUE;
}
If your C/C++-program already has an initialized Python-interpreter it would make sense to offer a function which only imports the module cyfun and doesn't initialize the python-interpreter. In this case I would define CYTHON_PEP489_MULTI_PHASE_INIT=0, because PyImport_AppendInittab must be called before Py_Initialize, which might be already too late when the dll is loaded.

I'd imagine it'd be difficult to call it directly, with Cython depending so much on the Python runtime.
Your best bet is to embed a Python interpreter directly inside your app, e.g. as described in this answer, and call your Cython code from the interpreter. That's what I would do.

Cython with Scikit-build: dynamic module does not define module export function

I have been doing some tests with the cython hello world example in scikit-build-sample-projects, and noticed that if I change the name of some files and their corresponding fields in CMakeLists.txt the compilation works with no error but when trying the result, the following errors can come:
For python3:ImportError: dynamic module does not define module export function (PyInit_...
For python2:ImportError: dynamic module does not define module export function (Init_...
and looking more into the c++ file I found these parts:
#if PY_MAJOR_VERSION < 3
PyMODINIT____FUNC init_hello(void)
{
(void) Py_InitModule("_rectangle", hello_methods);
}
#else /* PY_MAJOR_VERSION >= 3 */
static struct PyModuleDef hello_module_def = {
PyModuleDef_HEAD_INIT,
"_rectangle",
"Internal \"_rectangle\" module",
-1,
hello_methods
};
PyMODINIT_FUNC PyInit__rectangle(void)
{
return PyModule_Create(&hello_module_def);
}
#endif /* PY_MAJOR_VERSION >= 3 */
where if I change the "_hello" parts to the new name I had defined ion the project everything works again.
Now I want to try the original Cython's rectangle class example and am not sure how to make it work because of the above part. It seems to me that I should change the c++ code of rectangle.
Does anyone know how we can reproduce the Cython's rectangle example with scikit build taking into account the above issue?

Wrapper DLL for Python: "fatal error LNK1127: library is corrupt"

Brief description
I have a DLL programmed in ADA with GNAT. I want to compile with MSVC another DLL in C as a wrapper to the ADA_DLL in order to use it with Python.
I have compiled the ada_DLL, then I have generated the .lib file according to gnat documentation about MSVC. And finally I tried to compile the C_DLL with Visual-Studio, getting the error:
libmath.lib : fatal error LNK1127: library is corrupt
Update: In the case of compiling with gcc as suggested by #Brian, I get the following output:
>"C:\GNAT\2015\bin\gcc.exe" -c -IC:\Python27\include -o libmath_c.o libmath_c.c
>"C:\GNAT\2015\bin\gcc.exe" -shared -LC:\Python27\libs -L./ -l libmath -o DIVISION_CPP.pyd libmath_c.o -lpython27
.//libmath.lib: error adding symbols: Malformed archive
collect2.exe: error: ld returned 1 exit status
Things I tried & more data:
I have tried importing the ADA_DLL directly with ctypes in Python and it works, so I believe that the ADA_DLL is correctly compiled. Also, forgetting about the C_DLL is not really an option.
I did a small example with a division example module. My .def file looks something like:
; dlltool -z libmath.def --export-all-symbols libmath.dll
EXPORTS
[...]
div # 259
[...]
The libmath_c.c:
#include "libmath_c.h"
PyObject* _wrap_DIVISION(PyObject *self, PyObject *args){
div(10, 2);
return Py_None;
}
__declspec(dllexport) void __cdecl initDIVISION_CPP(void){
Py_InitModule("DIVISION_CPP", LIB_METHODS_methods);
}
The libmath_c.h:
#include <windows.h>
#include <stdio.h>
#include <Python.h>
PyObject* _wrap_DIVISION(PyObject *self, PyObject *args);
static PyMethodDef LIB_METHODS_methods[] = {
{ "CPP_DIVISION", _wrap_DIVISION, METH_VARARGS },
{NULL, NULL, 0, NULL} //Added as indicated by #Brian. Thanks!
};
__declspec(dllexport) void __cdecl initDIVISION_CPP(void);
Any idea of what is happening? Any help would be really appreciated. Thanks!

Preamble: Apologies if this turns out to be a non-answer; I want to be able to come back to this and find the links again, and comments tend to rot...
First, gcc (in the version matching Gnat) may work as an alternative C compiler, and if it does, it may eliminate difficulties with incompatible library versions.
GCC can be used for building Windows DLLs so the result should be usable from other Windows executables.
Following comments; gcc does appear to allow compilation, but the result is not currently usable from Python - here, my Python knowledge is shallow, and we don't have an MCVE, so this is speculative:
This Q&A addresses the same error message between Python and pure C, with no Ada, suggesting this error may not be specific to C-wrapped Ada.
You have already bypassed the asker's specific error,
static PyMethodDef* _npfindmethods = { ... };
which was using a pointer; you are (correctly according to the answer) statically allocating an array. However, the accepted answer terminates the list of methods
static PyMethodDef _npfindmethods[] = {
{"add", py_add, METH_VARARGS, py_add_doc},
{NULL, NULL, 0, NULL}
};
with a NULL method; your example does not:
static PyMethodDef LIB_METHODS_methods[] = {
{ "CPP_DIVISION", _wrap_DIVISION, METH_VARARGS }
};
So my hypothesis is that when you run setup() on this module, it finds CPP_DIVISION successfully, then in the absence of a NULL method it runs off into the weeds, producing the same symptoms despite the difference in cause.
I could test this hypothesis using the MCVE in that question by deleting the NULL method; however I don't have a Windows system handy, only Linux.
Alternatively, I see no reason for a C layer. If there isn't one, this Q&A addresses direct interaction between Python and Ada with no C layer, though it appears to use a different method, getattr() to import the external method. Might be an alternative?

Finally I managed to compile with gcc+gnat but not with MSVC+gnat.
With gcc+gnat, I was getting .//libmath.lib: error adding symbols: Malformed archive. The solution consists on using libmath.dll instead of building the .lib from the .dll.
So, in summary:
If you have a .dll generated by gnat, use it with gcc. You don't need to build a .lib.
If you have a .lib (for example python27.lib) or a .dll not generated by gnat, convert it to a .a using a tool like "pexport" (DO NOT USE SED!).
If you really need to compile using the MSVC... I'm sorry, I could not manage to make it work. Your princess is in another castle.

Compiler can't find Py_InitModule() .. is it deprecated and if so what should I use?

I am attempting to write a C extension for python. With the code (below) I get the compiler warning:
implicit declaration of function ‘Py_InitModule’
And it fails at run time with this error:
undefined symbol: Py_InitModule
I have spent literally hours searching for a solution with no joy. I have tried multiple minor changes to syntax, I even found a post suggesting the method has been deprecated. However I find no replacement.
Here is the code:
#include <Python.h>
//a func to calc fib numbers
int cFib(int n)
{
if (n<2) return n;
return cFib(n-1) + cFib(n-2);
}
static PyObject* fib(PyObject* self,PyObject* args)
{
int n;
if (!PyArg_ParseTuple(args,"i",&n))
return NULL;
return Py_BuildValue("i",cFib(n));
}
static PyMethodDef module_methods[] = {
{"fib",(PyCFunction) fib, METH_VARARGS,"calculates the fibonachi number"},
{NULL,NULL,0,NULL}
};
PyMODINIT_FUNC initcModPyDem(void)
{
Py_InitModule("cModPyDem",module_methods,"a module");
}
If it helps here is my setup.py :
from distutils.core import setup, Extension
module = Extension('cModPyDem', sources=['cModPyDem.c'])
setup(name = 'packagename',
version='1.0',
description = 'a test package',
ext_modules = [module])
And the test code in test.py :
import cModPyDem
if __name__ == '__main__' :
print(cModPyDem.fib(200))
Any help would be much, much appreciated.

The code you have would work fine in Python 2.x, but Py_InitModule is no longer used in Python 3.x. Nowadays, you create a PyModuleDef structure and then pass a reference to it to PyModule_Create.
The structure would look like:
static struct PyModuleDef cModPyDem =
{
PyModuleDef_HEAD_INIT,
"cModPyDem", /* name of module */
"", /* module documentation, may be NULL */
-1, /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */
module_methods
};
And then your PyMODINIT_FUNC function would look like:
PyMODINIT_FUNC PyInit_cModPyDem(void)
{
return PyModule_Create(&cModPyDem);
}
Note that the name of the PyMODINIT_FUNC function must be of the form PyInit_<name> where <name> is the name of your module.
I think it would be worthwhile if you read Extending in the Python 3.x documentation. It has a detailed description of how to build extension modules in modern Python.

I ran into the same problem with Py_InitModule(). I started with the aforementioned Python 3 docs, specifically the "Extending and Embedding the Python Interpreter" doc. But that doc's chapter entitled "A Simple Example" leaves out details. So. I googled this scipy lecture:
http://www.scipy-lectures.org/advanced/interfacing_with_c/interfacing_with_c.html
which is in many ways more suitable for someone new to Python-C API extensions ... except it has not been updated for Python v3. So ... consult the scipy lecture, and the Python 3 docs, and this StackOverflow discussion, culling the pertinent information from each for your needs.

How to get Python to use Assembly

I am a beginner in assembly, but a master in Python. I have just recently started to learn x86_64 NASM for windows, and I wish to combine the power of assembly, and the flexibility of Python. I have looked all over, and I have not found a way to use a NASM assembly procedure from within Python. By this I do not mean in-line assembly. I wish to write an assembly program, compile it, and then somehow extract the procedure to use in my Python program. Can someone illustrate a simple example of how to do this, as I am completely lost.

You could create a C extension wrapper for the functions implemented in assembly and link it to the OBJ file created by nasm.
A dummy example (for 32 bit Python 2; not tested):
myfunc.asm:
;http://www.nasm.us/doc/nasmdoc9.html
global _myfunc
section .text
_myfunc:
push ebp
mov ebp,esp
sub esp,0x40 ; 64 bytes of local stack space
mov ebx,[ebp+8] ; first parameter to function
; some more code
leave
ret
myext.c:
#include <Python.h>
void myfunc(void);
static PyObject*
py_myfunc(PyObject* self, PyObject* args)
{
if (!PyArg_ParseTuple(args, ""))
return NULL;
myfunc();
Py_RETURN_NONE;
}
static PyMethodDef MyMethods[] =
{
{"myfunc", py_myfunc, METH_VARARGS, NULL},
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC initmyext(void)
{
(void) Py_InitModule("myext", MyMethods);
}
setup.py:
from distutils.core import setup, Extension
setup(name='myext', ext_modules=[
Extension('myext', ['myext.c'], extra_objects=['myfunc.obj'])])
Build and run:
nasm -fwin32 myfunc.asm
python setup.py build_ext --inplace
python -c"import myext;myext.myfunc()"

You can also embed assembly directly inside your Python program:
https://github.com/Maratyszcza/PeachPy
https://github.com//pycca/pycca
http://codeflow.org/entries/2009/jul/31/pyasm-python-x86-assembler/
https://github.com/AmihaiN/pyAsm
These work by compiling the assembly and loading it into executable memory at runtime. The first three projects implement x86 assemblers in Python, whereas the last calls out to an external compiler.

Not sure about the "power" of assembly, really.
You can start here:
https://docs.python.org/2/extending/extending.html
It's about extending python with compiled code, written in C or C++, but the principle should be the same (C is really just a portable macro-assembler).