I have dll, that builded in Delphi, and I try to call function from it. Declaration of function looks like this:
function GetUid(UID:Pointer):Integer; stdcall;
This is equivalent to this C function signature:
int GetUID(void *pointer);
Library handled using ctypes:
from ctypes import *
lib = cdll.LoadLibrary("mylib.dll")
But i stuck here:
res = lib.GetUid(?)
What I need to pass in this function?
Pointer is void *, but how make this rightly in python?
Python is a high level language. You do not typically import a DLL from a C or Pascal native library and invoke it and pass variables from Python into a C or Pascal function taking a void * type raw pointer and then manipulate raw memory this way.
In short if you knew what you were doing you would know better than to try to do what you're doing here.
Let's suppose that your implementation is like this:
function GetUid(UID:Pointer):Integer; stdcall;
var
P2:^Integer;
begin
P2 := UID;
P2^ := 0;
end;
Then, what you would want to do is pass in an address to a 32 bit integer. Of course my example above is absurd, because what would have made sense above is to just declare the parameter as an "int *pointer" (in C terms) rather than as a "void *pointer".
Whatever it is you're doing, the next thing that will likely happen is that you will corrupt your python interpreter's heap, and cause lots of fun crashes and errors.
A far more sensible approach is to read the Python documentation on writing C extensions that can manipulate native Python types (PyObject), and doing the same thing but in pascal, if you like.
p4d appears to be a workable way of writing extension DLLs in delphi:
https://code.google.com/p/python4delphi/source/list
Related
I have a module called test.go that contains two simple Go functions which accept string types:
package main
import (
"fmt"
"C"
)
//export TestConcat
func TestConcat(testArg string, testArg2 string) (string) {
retval := testArg + testArg2
return retval
}
//export TestHello
func TestHello(testArg string) {
fmt.Println("%v\n", testArg)
}
func main(){}
I compile it as a shared library with go build -o test.so -buildmode=c-shared test.go
Then I have a Python module called test.py
import ctypes
from ctypes import cdll
test_strings = [
"teststring1",
"teststring2"
]
if __name__ == '__main__':
lib = cdll.LoadLibrary("./test.so")
lib.TestConcat.argtypes = [ctypes.c_wchar_p, ctypes.c_wchar_p]
lib.TestHello.argtypes = [ctypes.c_wchar_p]
for test_string in test_strings:
print(
lib.TestConcat("hello", test_string)
)
lib.TestHello(test_string)
Then I run test.py and get a nasty segfault
runtime: out of memory: cannot allocate 279362762964992-byte block (66781184 in use)
fatal error: out of memory
I've tried wrapping the arguments in ctypes.c_wchar_p to no avail.
What am I doing wrong here? And specifically, how does one interact with Go functions that accept string arguments in Python?
The Go's string type is actually something like
type string {
ptr *byte
size int
}
so that is what the Test{Hello|Concat} actually expect—not a pair of pointers but a pair of struct-typed values.
In other words, cgo performs just enough magic to gateway calls from Go to C and back, but it does not perform automatic conversions of values.
You have two options:
Explicitly work with this from your ctypes bindings, if possible.
When compiling your package, cgo generates a header file which contains a C definition for the struct representing a Go string; you could use it right away.
Make the functions exported to C compatible with the C's "type system".
For this, cgo offers helper functions C.CString and C.GoString.
Basically, you can define your API like this:
func TestHello(a, b *C.char) *C.char {
testArg1, testArg2 := C.GoString(a), C.GoString(b)
return C.CString(testArg + TestArg2)
}
Note few caveats here:
Both of these helpers copy the memory of their argument, so the silly example above would work just fine but it would first duplicate the memory blocks pointed to by a and b, then eat up twice as much memory to produce the concatenated string and then copy the memory of the resulting string once again to produce the returned pointer.
IOW, this approach is fine if you're trying to export to C some big chunk of Go code so that these allocations are dwarfed by whatever that chunk does.
Using *C.char is the same as *char in C, so the string is expected to be NUL-terminated; if it's not, use C.GoStringN.
Every memory block allocated by C.CString have to be freed by a call to C.free. And here's a twist: C.free is basically a thin shim to call free() from the linked in libc, so if you can guarantee the complete product (the code fully loaded into memory and (inter)linked using the dymanic linker) has only a single copy of libc linked in, you can call free() from the non-Go code on the memory blocks produced by calls to C.Cstring in the Go code.
A few more random pointers:
I'm not well-versed in Python's ctypes but I'd speculate using ctypes.c_wchar_p is not correct: in C (and C++, FWIW) wchar_t is a type to denote a single fixed-sized "wide character", which is usually a UCS-2/UTF-16 code point, and Go's strings are not composed of these—they may contain arbitrary bytes, and when they are used to contain Unicode text, they are encoded using UTF-8 which is a multi-byte ecoding (a single Unicode code point may be represented by 1 to 4 bytes in the string).
In either case, wchar_t cannot be used for UTF-8 (and actually many seasoned devs beleive it's an abomination).
Please read the docs on cmd/cgo completely before embarking on this project. Really, please do!
I am creating Python bindings for a C library.
In C the code to use the functions would look like this:
Ihandle *foo;
foo = MethFunc();
SetArribute(foo, 's');
I am trying to get this into Python. Where I have MethFunc() and SetAttribute() functions that could be used in my Python code:
import mymodule
foo = mymodule.MethFunc()
mymodule.SetAttribute(foo)
So far my C code to return the function looks like this:
static PyObject * _MethFunc(PyObject *self, PyObject *args) {
return Py_BuildValue("O", MethFunc());
}
But that fails by crashing (no errors)
I have also tried return MethFunc(); but that failed.
How can I return the function foo (or if what I am trying to achieve is completely wrong, how should I go about passing MethFunc() to SetAttribute())?
The problem here is that MethFunc() returns an IHandle *, but you're telling Python to treat it as a PyObject *. Presumably those are completely unrelated types.
A PyObject * (or any struct you or Python defines that starts with an appropriate HEAD macro) begins with pointers to a refcount and a type, and the first thing Python is going to do with any object you hand it is deal with those pointers. So, if you give it an object that instead starts with, say, two ints, Python is going to end up trying to access a type at 0x00020001 or similar, which is almost certain to segfault.
If you need to pass around a pointer to some C object, you have to wrap it up in a Python object. There are three ways to do this, from hackiest to most solid.
First, you can just cast the IHandle * to a size_t, then PyLong_FromSize_t it.
This is dead simple to implement. But it means these objects are going to look exactly like numbers from the Python side, because that's all they are.
Obviously you can't attach a method to this number; instead, your API has to be a free function that takes a number, then casts that number back to an IHandle* and calls a method.
It's more like, e.g., C's stdio, where you have to keep passing stdin or f as an argument to fread, instead of Python's io, where you call methods on sys.stdin or f.
But even worse, because there's no type checking, static or dynamic, to protect you from some Python code accidentally passing you the number 42. Which you'll then cast to an IHandle * and try to dereference, leading to a segfault…
And if you were hoping Python's garbage collector would help you know when the object is still referenced, you're out of luck. You need to make your users manually keep track of the number and call some CloseHandle function when they're done with it.
Really, this isn't that much better than accessing your code from ctypes, so hopefully that inspires you to keep reading.
A better solution is to cast the IHandle * to a void *, then PyCapsule_New it.
If you haven't read about capsules, you need to at least skim the main chapter. But the basic idea is that it wraps up a void* as a Python object.
So, it's almost as simple as passing around numbers, but solves most of the problems. Capsules are opaque values which your Python users can't accidentally do arithmetic on; they can't send you 42 in place of a capsule; you can attach a function that gets called when the last reference to a capsule goes away; you can even give it a nice name to show up in the repr.
But you still can't attach any behavior to capsules.
So, your API will still have to be a MethSetAttribute(mymodule, foo) instead of mymeth.SetAttribute(foo) if mymodule is a capsule, just as if it's an int. (Except now it's type-safe.)
Finally, you can build a new Python extension type for a struct that contains an IHandle *.
This is a lot more work. And if you haven't read the tutorial on Defining Extension Types, you need to go thoroughly read through that whole chapter.
But it means that you have an actual Python type, with everything that goes with it.
You can give it a SetAttribute method, and Python code can just call that method. You can give it whatever __str__ and __repr__ you want. You can give it a __doc__. Python code can do isinstance(mymodule, MyMeth). And so on.
If you're willing to use C++, or D, or Rust instead of C, there are some great libraries (PyCxx, boost::python, Pyd, rust-python, etc.) that can do most of the boilerplate for you. You just declare that you want a Python class and how you want its attributes and methods bound to your C attributes and methods and you get something you can use like a C++ class, except that it's actually a PyObject * under the covers. (And it'll even takes care of all the refcounting cruft for you via RAII, which will save you endless weekends debugging segfaults and memory leaks…)
Or you can use Cython, which lets you write C extension modules in a language that's basically Python, but extended to interface with C code. So your wrapper class is just a class, but with a special private cdef attribute that holds the IHandle *, and your SetAttribute(self, s) can just call the C SetAttribute function with that private attribute.
Or, as suggested by user, you can also use SWIG to generate the C bindings for you. For simple cases, it's pretty trivial—just feed it your C API, and it gives you back the code to build your Python .so. For less simple cases, I personally find it a lot more painful than something like PyCxx, but it definitely has a lower learning curve if you don't already know C++.
I'm trying to write a small, modular program in Python that will dynamically load C functions and use them to execute computationally intensive code. In this program I am creating a couple of large matrices that I will be passing back and forth between my Python code to different C functions. I would prefer to pass these matrices by reference to avoid additional computational overhead.
I've tried reading through the Python docs for ctypes, but it doesn't seem to explain how to do this. I understand, for instance, that I can use byref() or pointer() to pass a pointer from Python to a C function, but how to I pass a pointer from an external C function back to Python? Given that variables are names in Python, is this just done "automatically" (for lack of a better term) when Python receives a value from a C function?
As a concrete example, this is what I'm trying to do (in pseudo-code):
foo = ctypes.CDLL("pathToFoo")
bar = ctypes.CDLL("pathToBar")
# Generate a large matrix by calling a C function.
reallyBigMatrix = foo.generateReallyBigMatrix()
# Pass reallyBigMatrix to another function and perform some operation
# on it. Since the matrix is really big, I would prefer to pass a
# reference to this matrix to my next C function rather than passing
# the matrix by value.
modifiedReallyBigMatrix = bar.modifyReallyBigMatrix(reallBigMatrix)
Alternatively, I'm using Python and C in conjunction as I need an easy way to dynamically load C functions in my program. I may pass paths to different C files to my Python program so that the Python program will execute the same code on different functions. As an example, I may want to run my program two different ways: keep the same "generateReallyBigMatrix" function in both runs, but used a different "modifyReallyBigMatrix" program between run 1 and run 2. If there is an easy, cross-platform way to do this in C or C++ I would be happy to implement that solution rather than using ctypes and Python. However, I haven't been able to find a simple, cross-platform solution.
You've mentioned that you are writing all the code, both Python and C, from yourself. I suggest not using ctypes for this, as ctypes is best suited for using C libraries that cannot be modified.
Instead, write a module in C using the Python C API. It will expose a single function to start with, like this:
PyObject* generateReallyBigMatrix(void);
Now, instead of trying to return a raw C pointer, you can return any Python object that you like. Good choices here would be to return a NumPy array (using the NumPy C API), or to return a Python "buffer" (from which a NumPy array can be constructed in Python if desired).
Either way, once this function is written in C using the appropriate APIs, your Python code will be simple:
import foo
reallyBigMatrix = foo.generateReallyBigMatrix()
To do it using the NumPy C API, your C code will look like this:
PyObject* generateReallyBigMatrix(void)
{
npy_intp dimension = 100;
PyArray_Descr* descr;
PyArray_DescrAlignConverter2("float64", &descr); // use any dtype
PyObject* array = PyArray_Empty(1, &dimension, descr, 0/*fortran*/);
Py_DECREF(descr);
void* data = PyArray_DATA(array);
// TODO: populate data
return array;
}
static PyMethodDef methods[] = {
{"generateReallyBigMatrix", generateReallyBigMatrix, METH_VARARGS, "doc"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
PyMODINIT_FUNC initfoo(void)
{
import_array(); // enable NumPy C API
Py_InitModule("foo", methods);
}
Note that the NumPy C API requires a slightly strange initialization ritual. See also Numpy C API: Link several object files
You then compile the code as a shared library called foo.so (no lib prefix).
I started on my first Python extension today and was only creating a very small wrapper around a C library as an exercise. As is typical with C libraries, you start of with an initialization function that yields a handler. You can pass that handler to functions and later you pass it to the cleanup function that frees memory.
When I started writing the wrapper I basically wanted to have a way to call each native C function from python. Quickly I hit the problem that I need to return an arbitrary pointer from C to Python only to give it from there to C again in another function. I doesn't matter how it looks as I don't use it in Python, I just store it and pass it around.
So how do you pass around a void pointer between Python and C?
Please note: I know it is not recommended to write such small wrappers using the extension system but rather ctypes and friends. This is just for practice right now.
PyLong_FromVoidPtr() and PyLong_AsVoidPtr() can be abused to inject malicious data into your program. I recommend against them.
Python has PyCapsule for exactly that job. Capsules provide a safe way to exchange void ptr between modules or Python space and C space. The capsules are type-safe, too. If you need some example, the socket / ssl modules and pyexpat / _elementtree modules use capsules to exchange CAPI structs.
http://docs.python.org/3/c-api/capsule.html
After some searching I found the functions PyLong_AsVoidPtr and PyLong_FromVoidPtr. This yields a nice way to convert between a void * and a PyObject:
# in init function
return PyLong_FromVoidPtr(handle);
# in function using handle
handle = PyLong_AsVoidPtr(python_handle);
The one problem now might be how to retrieve python_handle from the typical *args given to a function:
PyObject *python_handle;
PyArg_ParseTuple(args, "O", &python_handle);
Careful here: The argument given for the "O" object must be a pointer to a PyObject pointer: PyObject **. The "O" itself only denotes to pass this PyObject through without any handling and converting. And with this, you can pass around any pointers any way you like.
Note: I think this solution is not really pretty, because you now have to variables, one that is only needed for a short time.
I've been looking for a simple answer to this question, but it seems that I can't find one. I would prefer to stay away from any external libraries that aren't already included in Python 2.6/2.7.
I have 2 c header files that resemble the following:
//constants_a.h
const double constant1 = 2.25;
const double constant2 = -0.173;
const int constant3 = 13;
...
//constants_b.h
const double constant1 = 123.25;
const double constant2 = -0.12373;
const int constant3 = 14;
...
And I have a python class that I want to import these constants into:
#pythonclass.py
class MyObject(object):
def __init(self, mode):
if mode is "a":
# import from constants_a.h, like:
# self.constant1 = constant1
# self.constant2 = constant2
elif mode is "b":
# import from constants_b.h, like:
# self.constant1 = constant1
# self.constant2 = constant2
...
I have c code which uses the constants as well, and resembles this:
//computations.c
#include <stdio.h>
#include <math.h>
#include "constants_a.h"
// do some calculations, blah blah blah
How can I import the constants from the header file into the Python class?
The reason for the header files constants_a.h and constants_b.h is that I am using python to do most of the calculations using the constants, but at one point I need to use C to do more optimized calculations. At this point I am using ctypes to wrap the c code into Python. I want to keep the constants away from the code just in case I need to update or change them, and make my code much cleaner as well. I don't know if it helps to note I am also using NumPy, but other than that, no other non-standard Python extensions. I am also open to any suggestions regarding the design or architecture of this program.
In general, defining variables in C header file is poor style. The header file should only declare objects, leaving their definition for the appropriate ".c" source code file.
One thing you may want to do is to declare the library-global constants like extern const whatever_type_t foo; and define (or "implement") them (i.e. assigning values to them) somewhere in your C code (make sure you do this only once).
Anyway, let's ignore how you do it. Just suppose you've already defined the constants and made their symbols visible in your shared object file "libfoo.so". Let us suppose you want to access the symbol pi, defined as extern const double pi = 3.1415926; in libfoo, from your Python code.
Now you typically load your object file in Python using ctypes like this:
>>> import ctypes
>>> libfoo = ctypes.CDLL("path/to/libfoo.so")
But then you'll see ctypes thinks libfoo.pi is a function, not a symbol for constant data!
>>> libfoo.pi
<_FuncPtr object at 0x1c9c6d0>
To access its value, you have to do something rather awkward -- casting what ctypes thinks is a function back to a number.
>>> pi = ctypes.cast(foo.pi, ctypes.POINTER(ctypes.c_double))
>>> pi.contents.value
3.1415926
In C jargon, this vaguely corresponds to the following thing happening: You have a const double pi, but someone forces you to use it only via a function pointer:
typedef int (*view_anything_as_a_function_t)(void);
view_anyting_as_a_function_t pi_view = π
What do you do with the pointer pi_view in order to use the value of pi? You cast it back as a const double * and dereference it: *(const double *)(pi_view).
So this is all very awkward. Maybe I'm missing something but this I believe is by design of the ctypes module -- it's there chiefly for making foreign function calls, not for accessing "foreign" data. And exporting pure data symbol in a loadable library is arguably rare.
And this will not work if the constants are only C macro definitions. There's in general no way you can access macro-defined data externally. They're macro-expanded at compile time, leaving no visible symbol in the generated library file, unless you also export their macro values in your C code.
I recommend using regular expressions (re module) to parse the information you want out of the files.
Building a full C parser would be huge, but if you only use the variables and the file is reasonably simple/predictable/under control, then what you need to write is straightforward.
Just watch out for 'gotcha' artifacts such as commented-out code!
I would recommend using some kind of configuration file readable by both Python and C program, rather than storing constant values in headers. E.g. a simple csv, ini-file, or even your own simple format of 'key:value' pairs. And there will be no need to recompile the C program every time you'd like to change one of the values :)
I'd up-vote emilio, but I'm lacking rep!
Although you have requested to avoid other non-standard libraries, you may wish to take a look at Cython (Cython: C-Extensions for Python www.cython.org/), which offers the flexibility of Python coding and the raw speed of execution of C/C++-compiled code.
This way you can use regular Python for everything, but handle the expensive elements of code using its built-in C-types. You can then convert your Python code into .c files too (or just wrap external C-libraries themselves. ), which can then be compiled into a binary. I've achieved up to 10x speed-ups doing so for numerical routines. I also believe NumPy uses it.