I have been trying to convert a C-code to DLL and load it from Python but I have never succeeded. I am using Windows10
(64bit). The original C code is quite simple:
//Filename: my_functions.c
# include <stdio.h>
int square(int i){
return i*i
}
I summarize the two approaches I have tried so far.
Method1
First, I built the 64-bit dll by cl.exe on Command Prompt:
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\Tools\VsDevCmd.bat" -arch=x64
cl.exe /D_USRDL /D_WINDLL my_functions.c /MT /link /DLL /OUT:my_functions.dll /MACHINE:X64
As a result, I got a file my_functions.dll. Subsequently, I tried to load it by the following Python code (Python 3.8, Spyder):
# Filename: calling_c_functions.py
from ctypes import *
my_functions = CDLL("./my_functions.dll")
print(type(my_functions))
print(my_functions.square(10))
print("Done")
However, I got the error:
AttributeError: function 'square' not found.
Why does this happen? According to the document in Microsoft: Exporting from a DLL, a DLL file contains an exports table including the name of every function that the DLL exports to other executables. These functions need to be accessed either by ordinal number or by name (different from the original name 'square').
This brought me to Method 2 below.
Method2
Based on Python: accessing DLL function using ctypes -- access by function name fails, I first created a DLL file by myself using the keyword __declspec(dllexport):
//Filename: my_functionsDLL2.dll
#include <stdio.h>
extern "C"
{
__declspec(dllexport) int square(int i);
}
__declspec(dllexport) int square(int i)
{
return i * i;
}
The next step is to find a real function name in the my_functionsDLL2.dll by link.exe. I did the following on the Command Prompt.
call "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\link.exe"
link /dump /exports my_functionsDLL2.dll
I should be able to see a list of function names but ended up getting the following output:
Microsoft (R) COFF/PE Dumper Version 14.29.30037.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file myfunctionsDLL2.dll
myfunctionsDLL2.dll : warning LNK4048: Invalid format file; ignored
Summary
Any idea what I'm doing wrong?
You're on the right track but got a bit confused along the way - taking a code file and giving it a .dll extension isn't enough to turn it into a DLL, you do still need to compile it. Your linker is telling you that it doesn't know how to parse your file:
myfunctionsDLL2.dll : warning LNK4048: Invalid format file; ignored
because it only knows how to parse real executables.
You should rename your second code file (the one you named my_functionsDLL2.dll) back to my_functions.c - and compile it the same way you did the first time. But this time, because you added __declspec(dllexport) to your function definition, your compiler (cl.exe) will know that these functions are meant to be exported from the DLL and will export them, making them available through ctypes.
I needed to make 2 modifications to my current approach.
The first modification is the one pointed out by #Daniel Kleinstein. But only with that, I stil got an error: C2059: syntax error: 'string'. This error is due to extern "C" { guards that are only understood by C++ (reference: C2059 syntax error 'string' ? ). For a C source file, I needed to add __cplusplus ifdef.
I summarize the overall procedures that led me to the correct result (there may be some redundancies though).
I want to call the following C file from Python.
//Filename: my_functions.c
#include <stdio.h>
int square(int i){
return i * i;
}
Create a C file using __declspec(dllexport) and __cplusplus ifdef.
//Filename: my_functions2.c
#include <stdio.h>
#ifdef __cplusplus
extern "C" {
#endif
__declspec(dllexport) int square(int i);
#ifdef __cplusplus
}
#endif
__declspec(dllexport) int square(int i)
{
return i * i;
}
Create a 64-bit dll file my_functions2.dll by running the following commands on Command Prompt. You need to locate the cl.exe carefully on your computer.
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\Tools\VsDevCmd.bat" -arch=x64
cl.exe /D_USRDL /D_WINDLL my_functions2.c /MT /link /DLL /OUT:my_functions2.dll /MACHINE:X64
Call the my_functions2.dll from Python using ctypes. In this example, I test the function square and obtain a correct result.
#Filename: call_my_functions2.py
import ctypes as ctypes
my_functions = ctypes.CDLL("./my_functions2.dll")
print(type(my_functions))
print(my_functions.square(10))
print("Done")
I get the following output:
<class 'ctypes.CDLL'>
100
Done
Related
I have a C++ program and it has sort of plugin structure: when program starts up, it's looking for dll in the plugin folder with certain exported function signatures, such as:
void InitPlugin(FuncTable* funcTable);
Then the program will call the function in the dll to initialize and pass function pointers to the dll. From that time on, the dll can talk to the program.
I know Cython let you call C function in Python, but I'm not sure can I write a Cython code and compile it to a dll so my C++ program can initialize with it. An example code would be great.
Using cython-module in a dll is not unlike using a cython-module in an embeded python interpreter.
The first step would be to mark cdef-function which should be used from external C-code with public, for example:
#cyfun.pyx:
#doesn't need python interpreter
cdef public int double_me(int me):
return 2*me;
#needs initialized python interpreter
cdef public void print_me(int me):
print("I'm", me);
cyfun.c and cyfun.h can be generated with
cython -3 cyfun.pyx
These files will be used for building of the dll.
The dll will need one function to initialize the python interpreter and another to finalize it, which should be called only once before double_me and print_me can be used (Ok, double_me would work also without interpreter, but this is an implementation detail). Note: the initialization/clean-up could be put also in DllMain - see such a version further bellow.
The header-file for the dll would look like following:
//cyfun_dll.h
#ifdef BUILDING_DLL
#define DLL_PUBLIC __declspec(dllexport)
#else
#define DLL_PUBLIC __declspec(dllimport)
#endif
//return 0 if everything ok
DLL_PUBLIC int cyfun_init();
DLL_PUBLIC void cyfun_finalize();
DLL_PUBLIC int cyfun_double_me(int me);
DLL_PUBLIC void cyfun_print_me(int me);
So there are the necessary init/finalize-functions and the symbols are exported via DLL_PUBLIC (which needs to be done see this SO-post) so it can be used outside of the dll.
The implementation follows in cyfun_dll.c-file:
//cyfun_dll.c
#define BUILDING_DLL
#include "cyfun_dll.h"
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "cyfun.h"
DLL_PUBLIC int cyfun_init(){
int status=PyImport_AppendInittab("cyfun", PyInit_cyfun);
if(status==-1){
return -1;//error
}
Py_Initialize();
PyObject *module = PyImport_ImportModule("cyfun");
if(module==NULL){
Py_Finalize();
return -1;//error
}
return 0;
}
DLL_PUBLIC void cyfun_finalize(){
Py_Finalize();
}
DLL_PUBLIC int cyfun_double_me(int me){
return double_me(me);
}
DLL_PUBLIC void cyfun_print_me(int me){
print_me(me);
}
Noteworthy details:
we define BUILDING_DLL so DLL_PUBLIC becomes __declspec(dllexport).
we use cyfun.h generated by cython from cyfun.pyx.
cyfun_init inizializes python interpreter and imports the built-in module cyfun. The somewhat complicated code is because since Cython-0.29, PEP-489 is default. More information can be found in this SO-post. If the Python-interpreter isn't initialized or if the module cyfun is not imported, the chances are high, that calling the functionality from cyfun.h will end in a segmentation fault.
cyfun_double_me just wraps double_me so it becomes visible outside of the dll.
Now we can build the dll!
:: set up tool chain
call "<path_to_vcvarsall>\vcvarsall.bat" x64
:: build cyfun.c generated by cython
cl /Tccyfun.c /Focyfun.obj /c <other_coptions> -I<path_to_python_include>
:: build dll-wrapper
cl /Tccyfun_dll.c /Focyfun_dll.obj /c <other_coptions> -I<path_to_python_include>
:: link both obj-files into a dll
link cyfun.obj cyfun_dll.obj /OUT:cyfun.dll /IMPLIB:cyfun.lib /DLL <other_loptions> -L<path_to_python_dll>
The dll is now built, but the following details are noteworthy:
<other_coptions> and <other_loptions> can vary from installation to installation. An easy way is to see them is to run cythonize some_file.pyx` and to inspect the log.
we don't need to pass python-dll, because it will be linked automatically, but we need to set the right library-path.
we have the dependency on the python-dll, so later on it must be somewhere where it can be found.
Were you go from here depends on your task, we test our dll with a simple main:
//test.c
#include "cyfun_dll.h"
int main(){
if(0!=cyfun_init()){
return -1;
}
cyfun_print_me(cyfun_double_me(2));
cyfun_finalize();
return 0;
}
which can be build via
...
:: build main-program
cl /Tctest.c /Focytest.obj /c <other_coptions> -I<path_to_python_include>
:: link the exe
link test.obj cyfun.lib /OUT:test_prog.exe <other_loptions> -L<path_to_python_dll>
And now calling test_prog.exe leads to the expected output "I'm 4".
Depending on your installation, following things must be considered:
test_prog.exe depends on pythonX.Y.dll which should be somewhere in the path so it can be found (the easiest way is to copy it next to the exe)
The embeded python interpreter needs an installation, see this and/or this SO-posts.
IIRC, it is not a great idea to initialize, then to finalize and then to initialize the Python-interpreter again (that might work for some scenarios, but not all , see for example this) - the interpreter should be initialized only once and stay alive until the programs ends.
Thus, it may make sense to put initialization/clean-up code into DllMain (and make cyfun_init() and cyfun_finalize() private), e.g.
BOOL WINAPI DllMain(
HINSTANCE hinstDLL, // handle to DLL module
DWORD fdwReason, // reason for calling function
LPVOID lpReserved ) // reserved
{
// Perform actions based on the reason for calling.
switch( fdwReason )
{
case DLL_PROCESS_ATTACH:
return cyfun_init()==0;
case DLL_PROCESS_DETACH:
cyfun_finalize();
break;
case DLL_THREAD_ATTACH:
// Do thread-specific initialization.
break;
case DLL_THREAD_DETACH:
// Do thread-specific cleanup.
break;
}
return TRUE;
}
If your C/C++-program already has an initialized Python-interpreter it would make sense to offer a function which only imports the module cyfun and doesn't initialize the python-interpreter. In this case I would define CYTHON_PEP489_MULTI_PHASE_INIT=0, because PyImport_AppendInittab must be called before Py_Initialize, which might be already too late when the dll is loaded.
I'd imagine it'd be difficult to call it directly, with Cython depending so much on the Python runtime.
Your best bet is to embed a Python interpreter directly inside your app, e.g. as described in this answer, and call your Cython code from the interpreter. That's what I would do.
Brief description
I have a DLL programmed in ADA with GNAT. I want to compile with MSVC another DLL in C as a wrapper to the ADA_DLL in order to use it with Python.
I have compiled the ada_DLL, then I have generated the .lib file according to gnat documentation about MSVC. And finally I tried to compile the C_DLL with Visual-Studio, getting the error:
libmath.lib : fatal error LNK1127: library is corrupt
Update: In the case of compiling with gcc as suggested by #Brian, I get the following output:
>"C:\GNAT\2015\bin\gcc.exe" -c -IC:\Python27\include -o libmath_c.o libmath_c.c
>"C:\GNAT\2015\bin\gcc.exe" -shared -LC:\Python27\libs -L./ -l libmath -o DIVISION_CPP.pyd libmath_c.o -lpython27
.//libmath.lib: error adding symbols: Malformed archive
collect2.exe: error: ld returned 1 exit status
Things I tried & more data:
I have tried importing the ADA_DLL directly with ctypes in Python and it works, so I believe that the ADA_DLL is correctly compiled. Also, forgetting about the C_DLL is not really an option.
I did a small example with a division example module. My .def file looks something like:
; dlltool -z libmath.def --export-all-symbols libmath.dll
EXPORTS
[...]
div # 259
[...]
The libmath_c.c:
#include "libmath_c.h"
PyObject* _wrap_DIVISION(PyObject *self, PyObject *args){
div(10, 2);
return Py_None;
}
__declspec(dllexport) void __cdecl initDIVISION_CPP(void){
Py_InitModule("DIVISION_CPP", LIB_METHODS_methods);
}
The libmath_c.h:
#include <windows.h>
#include <stdio.h>
#include <Python.h>
PyObject* _wrap_DIVISION(PyObject *self, PyObject *args);
static PyMethodDef LIB_METHODS_methods[] = {
{ "CPP_DIVISION", _wrap_DIVISION, METH_VARARGS },
{NULL, NULL, 0, NULL} //Added as indicated by #Brian. Thanks!
};
__declspec(dllexport) void __cdecl initDIVISION_CPP(void);
Any idea of what is happening? Any help would be really appreciated. Thanks!
Preamble: Apologies if this turns out to be a non-answer; I want to be able to come back to this and find the links again, and comments tend to rot...
First, gcc (in the version matching Gnat) may work as an alternative C compiler, and if it does, it may eliminate difficulties with incompatible library versions.
GCC can be used for building Windows DLLs so the result should be usable from other Windows executables.
Following comments; gcc does appear to allow compilation, but the result is not currently usable from Python - here, my Python knowledge is shallow, and we don't have an MCVE, so this is speculative:
This Q&A addresses the same error message between Python and pure C, with no Ada, suggesting this error may not be specific to C-wrapped Ada.
You have already bypassed the asker's specific error,
static PyMethodDef* _npfindmethods = { ... };
which was using a pointer; you are (correctly according to the answer) statically allocating an array. However, the accepted answer terminates the list of methods
static PyMethodDef _npfindmethods[] = {
{"add", py_add, METH_VARARGS, py_add_doc},
{NULL, NULL, 0, NULL}
};
with a NULL method; your example does not:
static PyMethodDef LIB_METHODS_methods[] = {
{ "CPP_DIVISION", _wrap_DIVISION, METH_VARARGS }
};
So my hypothesis is that when you run setup() on this module, it finds CPP_DIVISION successfully, then in the absence of a NULL method it runs off into the weeds, producing the same symptoms despite the difference in cause.
I could test this hypothesis using the MCVE in that question by deleting the NULL method; however I don't have a Windows system handy, only Linux.
Alternatively, I see no reason for a C layer. If there isn't one, this Q&A addresses direct interaction between Python and Ada with no C layer, though it appears to use a different method, getattr() to import the external method. Might be an alternative?
Finally I managed to compile with gcc+gnat but not with MSVC+gnat.
With gcc+gnat, I was getting .//libmath.lib: error adding symbols: Malformed archive. The solution consists on using libmath.dll instead of building the .lib from the .dll.
So, in summary:
If you have a .dll generated by gnat, use it with gcc. You don't need to build a .lib.
If you have a .lib (for example python27.lib) or a .dll not generated by gnat, convert it to a .a using a tool like "pexport" (DO NOT USE SED!).
If you really need to compile using the MSVC... I'm sorry, I could not manage to make it work. Your princess is in another castle.
The situation is simple. I open file in one DLL (DLL1) and close it in another (DLL2). I invoke both DLLs one after another in python script, the code is as following:
from ctypes import *
DLLFP1=r"C:\Workspaces\CPP\Test\build\files1.dll"
DLLFP2=r"C:\Workspaces\CPP\Test\build\files2.dll"
DLL1=cdll.LoadLibrary(DLLFP1)
DLL2=cdll.LoadLibrary(DLLFP2)
cppobj=c_void_p()
DLL1.open( byref(cppobj) )
DLL2.close(cppobj)
DLL1 code:
#define DLLEXPORT extern "C" __declspec(dllexport)
DLLEXPORT void open(ofstream* &f){
f = new ofstream();
string FP("c:\\teest.log");
f->open(FP);
}
and DLL2 code:
#define DLLEXPORT extern "C" __declspec(dllexport)
DLLEXPORT void close(ofstream* f){
f->close();
}
In the result I have "access violation writing". I also can normally write text to file from DLL2 and if I move close function to DLL1 it works fine. I use intel compiler. OS is Windows7.
The question is what is wrong? What is the right way to share file between DLLs?
Thank you very much in advance.
This may be related unless DLL1 and DLL2 are built for the same runtime: Potential Errors Passing CRT Objects Across DLL Boundaries
When you pass C Run-time (CRT) objects such as file handles, locales,
and environment variables into or out of a DLL (function calls across
the DLL boundary), unexpected behavior can occur if the DLL, as well
as the files calling into the DLL, use different copies of the CRT
libraries.
I am writing a program in python. now i want to use ctypes to use some functions of a class i have in c++.
so basically , i have an array of data in python. i have another program in c++ which is supposed to use that array and transform it to an image class. here is the program i have in c++. i have already made the makefile in linux and it compiles.
the name of the file is 'pclink.cpp'
#include <stdio.h>
#include "MImage.h"
#include<stdlib.h>
#ifdef __cplusplus
extern "C" {
#endif
int conv(double* data, int x,int y, int z, int len) {
MImage M;
//M.MLoadFromArray(data,x,y,z,len);
return 0;
}
#ifdef __cplusplus
}
#endif
I have also made the .so and .o files every thing seems fine. but when in python i type
import numpy as np
import ctypes
import os
pclink = np.ctypeslib.load_library('pclink','.')
this is the message i get
Traceback (most recent call last):
File "<pyshell#10>", line 1, in <module>
pclink=np.ctypeslib.load_library('pclink','.')
File "/usr/lib/python2.7/dist-packages/numpy/ctypeslib.py", line 131, in load_library
raise exc
OSError: /home/lucy94/pclink.so: undefined symbol: _ZN6MImage14MLoadFromArrayEPdiiii
previously i tried to link c++ and python with a simple program which does not have any classes and it works fine. it seems the problem is when i try to define an object from another class. so any one knows how to fix the problem?
thanks
Looks like your built .so file depends on another shareable library containing the MImage stuff. I don’t think Python ctypes will automatically load dependent libraries for you, you have to pull them in yourself. And you’ll need to specify the RTLD_GLOBAL flag so the symbols from that MImage library are available to pclink.so when you load that.
All, I found a piece of information on how to call c files in python, in these examples: there is a c file, which includes many other header files, the very beginning of this c files is #include Python.h, then I found that #include Python.h actually involves many many other header files, such as pystate.h, object.h, etc, so I include all the required header files. In an cpp IDE environment, it did not show errors. What I am trying to do is call this c code in python, so from ctypes import *, then it seems that a dll should be generated by code such as: cl -LD test.c -test.dll, but how to use the cl in this case? I used the cygwin: gcc, it worked fine. Could anyone help me with this i.e.: Call the C in python? Do I make myself clear? Thank you in advance!!
Well, Now I feel it important to tell me what I did:
The ultimate goal I wanna achieve is:
I am lazy, I do not want to re-write those c codes in python, (which is very complicated for me in some cases), so I just want to generate dll
files that python could call. I followed an example given by googleing "python call c", there are two versions in this examples: linux and windows:
The example test.c:
#include <windows.h>
BOOL APIENTRY
DllMain(HANDLE hModule, DWORD dwReason, LPVOID lpReserved) {
return TRUE;
}
__declspec(dllexport) int
multiply(int num1, int num2) {
return num1 * num2;
}
Two versions:
1, Complie under linux
gcc -c -fPIC test.c
gcc -shared test.o -o test.so
I did this in cygwin on my vista system, it works fine; :)
2, Compile under windows:
cl -LD test.c -test.dll
I used the cl in windows command line prompt, it won't work!
These are the python codes:
from ctypes import *
import os
libtest = cdll.LoadLibrary(os.getcwd() + '/test.so')
print test.multiply(2, 2)
Could anyone try this and tell me what you get? thank you!
You will find the command line options of Microsoft's C++ compiler here.
Consider the following switches for cl:
/nologo /GS /fp:precise /Zc:forScope /Gd
...and link your file using
/NOLOGO /OUT:"your.dll" /DLL <your lib files> /SUBSYSTEM:WINDOWS /MACHINE:X86 /DYNAMICBASE
Please have a look at what those options mean in detail, I just listed common ones. You should be aware of their effect nonetheless, so try to avoid copy&paste and make sure it's really what you need - the documentation linked above will help you. This is just a setup I use more or less often.
Be advised that you can always open Visual Studio, configure build options, and copy the command line invokations from the project configuration dialog.
Edit:
Ok, here is some more advice, given the new information you've edited into your original question. I took the example code of your simple DLL and pasted it into a source file, and made two changes:
#include <windows.h>
BOOL APIENTRY DllMain(HANDLE hModule, DWORD dwReason, LPVOID lpReserved)
{
return TRUE;
}
extern "C" __declspec(dllexport) int __stdcall multiply(int num1, int num2)
{
return num1 * num2;
}
First of all, I usually expect functions exported from a DLL to use stdcall calling convention, just because it's a common thing in Windows and there are languages who inherently cannot cope with cdecl, seeing as they only know stdcall. So that's one change I made.
Second, to make exports more friendly, I specified extern "C" to get rid of name mangling. I then proceeded to compile the code from the command line like this:
cl /nologo /GS /Zc:forScope /Gd c.cpp /link /OUT:"foobar.dll" /DL kernel32.lib /SUBSYSTEM:WINDOWS /MACHINE:X86
If you use the DUMPBIN tool from the Visual Studio toolset, you can check your DLL for exports:
dumpbin /EXPORTS foobar.dll
Seeing something like this...
ordinal hint RVA name
1 0 00001010 ?multiply##YGHHH#Z
...you can notice the exported name got mangled. You'll usually want clear names for exports, so either use a DEF file to specify exports in more details, or the shortcut from above.
Afterwards, I end up with a DLL that I can load into Python like this:
In [1]: import ctypes
In [2]: dll = ctypes.windll.LoadLibrary("foobar.dll")
In [3]: dll.multiply
Out[3]: <_FuncPtr object at 0x0928BEF3>
In [4]: dll.multiply(5, 5)
Out[4]: 25
Note that I'm using ctypes.windll here, which implies stdcall.