I have created the following class using pybind11:
py::class_<Raster>(m, "Raster")
.def(py::init<double*, std::size_t, std::size_t, std::size_t, double, double, double>());
However I have no idea how I would call this constructor in Python.. I see that Python expects a float in the place of the double*, but I cannot seem to call it.
I have tried, ctypes.data_as(ctypes.POINTER(ctypes.c_double)) but this does not work...
Edit:
I have distilled the answer from #Sergei answer.
py::class_<Raster>(m, "Raster", py::buffer_protocol())
.def("__init__", [](Raster& raster, py::array_t<double> buffer, double spacingX, double spacingY, double spacingZ) {
py::buffer_info info = buffer.request();
new (&raster) Raster3D(static_cast<double*>(info.ptr), info.shape[0], info.shape[1], info.shape[2], spacingX, spacingY, spacingZ);
})
Pybind does automatic conversions. When you bind f(double *) the argument is assumed to be a pointer to a singe value, not a pointer to the array begin, because it would be quite unnatural to expect such input from python side. So pybind will convert argument using this logic.
If you need to pass raw array to c++ use py::buffer like here:
py::class_<Matrix>(m, "Matrix", py::buffer_protocol())
.def("__init__", [](Matrix &m, py::buffer b) {
typedef Eigen::Stride<Eigen::Dynamic, Eigen::Dynamic> Strides;
/* Request a buffer descriptor from Python */
py::buffer_info info = b.request();
/* Some sanity checks ... */
if (info.format != py::format_descriptor<Scalar>::format())
throw std::runtime_error("Incompatible format: expected a double array!");
if (info.ndim != 2)
throw std::runtime_error("Incompatible buffer dimension!");
auto strides = Strides(
info.strides[rowMajor ? 0 : 1] / (py::ssize_t)sizeof(Scalar),
info.strides[rowMajor ? 1 : 0] / (py::ssize_t)sizeof(Scalar));
auto map = Eigen::Map<Matrix, 0, Strides>(
static_cast<Scalar *>(info.ptr), info.shape[0], info.shape[1], strides);
new (&m) Matrix(map);
});
To make it work you need to pass a type which follows python buffer protocol.
Related
Context:
I have written a Red Black tree implementation in C language. To allow it to use variable types, it only handles const void * elements, and initialisation of a tree must be given a comparison function with a signature int (*comp)(const void *, const void *). So far, so good, but I now try to use that C code to build an extension module for Python. It looks simple as first sight, because Python languages always pass references to objects which are received as pointers by C routines.
Problem:
Python objects come with rich comparison operators. That means that from a C extension module, comparing 2 arbitrary objects is trivial: just a matter of using int PyObject_RichCompareBool(PyObject *o1, PyObject *o2, int opid).
But the comparison may return -1 to indicate that the objects are not comparable. In Python or C++ it would be simple enough to throw an exception to signal an abnormal condition. Unfortunately C has no notion of exception, and I could not find a way using setjmp-longjmp because:
the environment buffer has do be known to both the englobing function and the internal one
I should free any allocated memory at longjmp time, when the internal function does not know what has been allocated
First idea:
A simple solution is to give a third parameter to the comparison function for it to signal an abnormal condition. But when the library is used in a plain C environment, that third parameter just does not make sense. I then remembered that in the 80', I had learned that in C language, parameters were passed in the stack in reversed order and unstacked by the caller to allow functions with a variable number of parameters. That means that provided the first 2 parameters are correct passing a third parameter to a function expecting 2 should be harmless.
Demo code:
#include <stdio.h>
// declares a type for the comparison functions
typedef int (*func)();
// A simple function for comparing integers - only 2 params
int f1(int a, int b) {
return a - b;
}
/* Inserts a value into an increasing array
* By convention 0 denotes the end of the array
* No size control implemented for brievety
* The comp function recieves a pointer to an int
* to be able to signal abnormal conditions
* */
int insert(int* arr, int val, func comp) {
int err = 0;
while ((0 != *arr) && (comp(*arr, val, &err) < 0)) { // 1
if (err) return 0;
++arr;
}
do {
int tmp = *arr;
*arr = val;
val = tmp;
} while (0 != *arr++);
return 1;
}
int main() {
func f = &f1;
// a simple test with 3 parameters
int cr = f(3, 1, 5); // 2
printf("%d\n", cr);
// demo usage of the insert function
int arr[10] = {0};
int data[] = { 1,5,3,2,4 };
for (int i = 0; i < sizeof(data) / sizeof(*data); i++) {
insert(arr, data[i], f1);
}
for (int i = 0; i < sizeof(data) / sizeof(*data); i++) {
printf("%d ", arr[i]);
}
return 0;
}
At (1) and (2) the 2 parameter function is called with 3 parameters. Of course, this code compiles without even a warning in Clang or MSVC, and runs fine giving the expected result.
Question:
While this code works fine on common implementations, I wonder whether actually passing a third parameter to a function expecting only two is really legit or does it invokes Undefined Behaviour?
Current research
Is it safe to invoke a C function with more parameters than it expects? : the accepted answer suggests that it should be safe when the C calling convention is used (which is my use case) while other answers show that the MSVC stdcall calling convention would not allow it
6.7.6.3 Function declarators (including prototypes) and 6.5.2.2 Function calls in draft n1570 for C11, but as English is not my first language, I could not understand where it was or not allowed
Remark:
The originality of this question is that it uses function pointers conversions.
I think it invokes Undefined Behavior.
From 6.5.2.2p6:
If the expression that denotes the called function has a type that does not include a prototype, the integer promotions are performed on each argument, and arguments that have type float are promoted to double. These are called the default argument promotions. If the number of arguments does not equal the number of parameters, the behavior is undefined.
The proper solution is redesign the Red Black tree implementation to allow passing a context as a third parameter.
int (*comp)(const void *, const void *, void *);
It is highly recommended to add a context argument to any function pointer type to allow emulate closures.
As a workaround, you could use a global variable.
static int err;
int f1(int a, int b) {
err = 0;
return a - b;
}
int insert(int* arr, int val, int comp(int,int)) {
err = 0;
while ((0 != *arr) && (comp(*arr, val) < 0)) { // 1
if (err) return 0;
++arr;
}
...
}
It is not the best solution because it is not re-entrant. Only a single instance of insert()/f1() can run at a time.
This is a complement to the accepted answer. The shown code uses function pointers to solve the compilation errors that would arise when calling a prototyped function with a wrong number of parameters.
But the draft n1570 for C11 says at 6.3.2.3 [Language/Conversions/Other operands/] Pointers ยง8:
... If a converted
pointer is used to call a function whose type is not compatible with the referenced type,
the behavior is undefined.
And it fully applies here because the referenced type is a function taking 2 parameters and the converted pointer type is a function taking 3 parameters. Per the accepted answer and 6.5.2.2p6 those two function type are not compatible, hence the code does invoke UB.
After finding that, I haved decided to give up with that way, and instead choosed to use wrapper functions that call the function passed to the library with their expected number of arguments to avoid UB.
Let's say I have the following C# function
public byte AddOne(byte r){
return r + 1
}
In my python code, I am able to call the function by doing something like
x = AddOne(3)
When I check the type of x, it always returns an int. This is true even if I change my C# function to return a long, a short, or an int. I want x to be a numpy.int8 if a byte is returned, or numpy.int64 if long is returned. How can I go about doing this since currently an int is always returned?
In Python.NET 2.5+ you can define custom codecs. In 2.5 they can not be used to override primitive type conversion, however, on the C# side you can define custom types corresponding to numpy.int64, numpy.int8, etc, then register codecs for them.
public struct I8 {
public byte Value;
static I8() {
var i8 = Py.Import("numpy").GetAttr("int8");
PyObjectConversions.RegisterEncoder(new I8Codec(i8));
}
}
public class I8Codec: IPyObjectEncoder {
readonly PyObject i8;
public I8Codec(PyObject i8) => this.i8 = i8;
public PyObject TryEncode(object value)
// call numpy.int8 constructor
=> i8.Invoke(((I8)value).Value.ToPython());
public PyObject CanEncode(Type type) => type == typeof(I8);
}
I8 DoubleIt(byte val) => new I8 { Value = val*2 };
Alternatively, you can modify this line to allow codecs on primitive types, and implement I8Codec for byte. However, trying to do it for int might cause unintended consequences. E.g. ICollection.Count will start returning numpy.int32, which you probably don't want.
I only recently started using cppyy and ctypes, so this may be a bit of a silly question. I have the following C++ function:
float method(const char* args[]) {
...
}
and from Python I want to pass args as a list of strings, i.e.:
args = *magic*
x = cppyy.gbl.method(args)
I have previously found this, so I used
def setParameters(strParamList):
numParams = len(strParamList)
strArrayType = ct.c_char_p * numParams
strArray = strArrayType()
for i, param in enumerate(strParamList):
strArray[i] = param
lib.SetParams(numParams, strArray)
and from Python:
args = setParameters([b'hello', b'world'])
c_types.c_char_p expects a bytes array. However, when calling x = cppyy.gbl.method(args) I get
TypeError: could not convert argument 1 (could not convert argument to buffer or nullptr)
I'm not entirely sure why this would be wrong since the args is a <__main__.c_char_p_Array_2> object, which I believe should be converted to a const char* args[].
For the sake of having a concrete example, I'll use this as the .cpp file:
#include <cstdlib>
extern "C"
float method(const char* args[]) {
float sum = 0.0f;
const char **p = args;
while(*p) {
sum += std::atof(*p++);
}
return sum;
}
And I'll assume it was compiled with g++ method.cpp -fPIC -shared -o method.so. Given those assumptions, here's an example of how you could use it from Python:
#!/usr/bin/env python3
from ctypes import *
lib = CDLL("./method.so")
lib.method.restype = c_float
lib.method.argtypes = (POINTER(c_char_p),)
def method(args):
return lib.method((c_char_p * (len(args) + 1))(*args))
print(method([b'1.23', b'45.6']))
We make a C array to hold the Python arguments. len(args) + 1 makes sure there's room for the null pointer sentinel.
ctypes does not have a public API that is usable from C/C++ for extension writers, so the handling of ctypes by cppyy is by necessity somewhat clunky. What's going wrong, is that the generated ctypes array of const char* is of type const char*[2] not const char*[] and since cppyy does a direct type match for ctypes types, that fails.
As-is, some code somewhere needs to do a conversion of the Python strings to low-level C ones, and hold on to that memory for the duration of the call. Me, personally, I'd use a little C++ wrapper, rather than having to think things through on the Python side. The point being that an std::vector<std::string> can deal with the necessary conversions (so no bytes type needed, for example, but of course allowed if you want to) and it can hold the temporary memory.
So, if you're given some 3rd party interface like this (putting it inline for cppyy only for the sake of the example):
import cppyy
cppyy.cppdef("""
float method(const char* args[], int len) {
for (int i = 0; i < len; ++i)
std::cerr << args[i] << " ";
std::cerr << std::endl;
return 42.f;
}
""")
Then I'd generate a wrapper:
# write a C++ wrapper to hide C code
cppyy.cppdef("""
namespace MyCppAPI {
float method(const std::vector<std::string>& args) {
std::vector<const char*> v;
v.reserve(args.size());
for (auto& s : args) v.push_back(s.c_str());
return ::method(v.data(), v.size());
}
}
""")
Then replace the original C API with the C++ version:
# replace C version with C++ one for all Python users
cppyy.gbl.method = cppyy.gbl.MyCppAPI.method
and things will be as expected for any other person downstream:
# now use it as expected
cppyy.gbl.method(["aap", "noot", "mies"])
All that said, obviously there is no reason why cppyy couldn't do this bit of wrapping automatically. I created this issue: https://bitbucket.org/wlav/cppyy/issues/235/automatically-convert-python-tuple-of
I have an an array class, Array1D, defined in c++ which essentially wraps the STL vector class. I extended this class so that I can display individual elements of the array vector. Here is the relevant code in my SWIG interface file:
namespace std{
%template(dblVector) vector<double>;
}
%extend Array1D{
double __getitem__(int index) {
return (*self)[index];
}
}
This allows me to access individual elements of the array in python:
>>> a = Array1D(10) # creates a c++ vector of length 10 with zeros
>>> a[0]
>>> 0
I want to be able to call a[1:3] for example, however, I get a TypeError when I try this:
TypeError: in method 'Array1D___getitem__', argument 2 of type 'int'
The problem is that python passes a Slice object when calling the slice variant of the getitem and your function definition is expecting an int. You will need to write a version of getitem that takes PyObject* as a parameter and then you'd have to implement the slicing of the vector there.
I am writing this without being setup to actually test it, so take it with a grain of salt. But I would do something like the following.
%extend Array1D
{
Array1D* __getitem__(PyObject *param)
{
if (PySlice_Check(param))
{
/* Py_ssize_t might be needed here instead of ints */
int len = 0, start = 0, stop = 0, step = 0, slicelength = 0;
len = this->size(); /* Or however you get the size of a vector */
PySlice_GetIndicesEx((PySliceObject*)param, len, &start, &stop, &step, &slicelength);
/* Here do stuff in order to return an Array1D that is the proper slice
given the start/stop/step defined above */
}
/* Unexpected parameter, probably should throw an exception here */
}
}
python code
for b in range(4):
for c in range(4):
print myfunc(b/0x100000000, c*8)
c code
unsigned int b,c;
for(b=0;b<4;b++)
for(c=0;c<4; c++)
printf("%L\n", b/0x100000000);
printf("%L\n" , myfunc(b/0x100000000, c*8));
I am getting an error saying:
error: integer constant is too large for "long" type at both printf statement in c code.
'myfunc' function returns a long.
This can be solved by defining 'b' a different type. I tried defining 'b' as 'long' and 'unsigned long' but no help.
Any pointers?
My bad...This is short version of problem
unsigned int b;
b = 1;
printf("%L", b/0x100000000L);
I am getting error and warnings:
error: integer constant is too large for "long" type
warning: conversion lacks type at end of format
warning: too many arguments for format
Your C code needs braces to create the scope that Python does by indentation, so it should look like this:
unsigned int b,c;
for(b=0;b<4;b++)
{
for(c=0;c<4; c++)
{
printf("%L\n", b/0x100000000);
printf("%L\n" , myfunc(b/0x100000000, c*8));
}
}
Try long long. Python automatically uses number representation which fits your constants, but C does not. 0x100000000L simply does not fit in 32-bit unsigned int, unsigned long and so on. Also, read your C textbook on long long data type and working with it.
unsigned int b,c;
const unsigned long d = 0x100000000L; /* 33 bits may be too big for int */
for(b=0;b<4;b++) {
for(c=0;c<4; c++) { /* use braces and indent consistently */
printf("%ud\n", b/d); /* "ud" to print Unsigned int in Decimal */
printf("%ld\n", myfunc(b/d, c*8)); /* "l" is another modifier for "d" */
}
}