I've been searching around the web with no luck. I have the following Python code:
class LED(Structure):
_fields_ = [
('color', c_char_p),
('id', c_uint32)
]
class LEDConfiguration(Structure):
_fields_ = [
('daemon_user', c_char_p),
('leds', POINTER(LED)),
('num_leds', c_uint32)
]
Here is a simplified example function that uses these structures and returns an LEDConfiguration.
def parseLedConfiguration(path, board):
lc = LEDConfiguration()
for config in configs:
if( config.attributes['ID'].value.lstrip().rstrip() == board ):
lc.daemon_user = c_char_p('some_name')
leds = []
#Imagine this in a loop
ld = LED()
ld.color = c_char_p('red')
ld.id = int(0)
leds.append(ld)
#end imagined loop
lc.num_leds = len(leds)
lc.leds = (LED * len(leds))(*leds)
return lc
Now this the C code I am using (I've stripped out everything involved with setting up python/calling the "parseLedConfiguration" function/etc but I can add it in if it is helpful).
/*Calling the python function "parseLedConfiguration"
pValue is the returned "LEDConfiguration" python Structure*/
pValue = PyObject_CallObject(pFunc, pArgs);
Py_DECREF(pArgs);
if (pValue != NULL)
{
int i, num_leds;
PyObject *obj = PyObject_GetAttr(pValue, PyString_FromString("daemon_user"));
daemon_user = PyString_AsString(obj);
Py_DECREF(obj);
obj = PyObject_GetAttr(pValue, PyString_FromString("num_leds"));
num_leds = PyInt_AsLong(obj);
Py_DECREF(obj);
obj = PyObject_GetAttr(pValue, PyString_FromString("leds"));
PyObject_Print(obj, stdout, 0);
My problem is figuring out how to access what is returned to the final "obj". The "PyObject_Print" on the "obj" shows this output:
<ConfigurationParser.LP_LED object at 0x7f678a06fcb0>
I want to get into a state where I can access that LP_LED object in the same way I'm accessing the above "LEDConfiguration" object.
EDIT 1
I guess another maybe more important question, is my python code correct? Is that how I should be storing a list or array of "Structure" inside another "Structure" so it can be accessed from the Python C API?
Thanks!
Since your EDIT 1 clarifies the underlying question, let me put that at top:
guess another maybe more important question, is my python code correct? Is that how I should be storing a list or array of "Structure" inside another "Structure" so it can be accessed from the Python C API?
No, that's how you should be storing an array of Structure inside another Structure so it can be accessed from non-Python-C-API C code. If you want it to be accessed from the Python C API, just use a Python list.
In general, if you're writing code in both Python and C, only one side has to bend over backward to work with the other one. The point of using ctypes Structures and POINTERs and the like in Python is to allow them to work directly in C, without having to go through the C API. Conversely, the point of using functions like PyList_GetItem is to allow you to use normal Python code, not ctypes Python code.
So, if you want to store a list inside a Structure to be accessed via the Python C API, just store a Python list—and you really don't need the Structure in the first place; use a normal Python class (possibly with __slots__). You can write this code without importing ctypes at all.
Conversely, if you want to store structures that can be used directly in C, you can do that with ctypes; then, in the C code, once you've gotten into Structure guts of the PyObject * you don't need the Python API anymore, because the structure is all C. This is usually the way you go when you have existing C code, and want to interface with it from Python, rather than when you're designing the C code from scratch, but there's no rule that says you can't use it the other way.
Meanwhile, if this is your first attempt at writing C and Python code that talk to each other, I'd suggest you use Cython. Then, once you're comfortable with that, if you want to learn ctypes, do a different project that uses Python with ctypes to talk to C code that knows nothing at all about Python. And then, a third project that uses the C API to talk to Python code that knows nothing about ctypes (like most C extension modules). Once you're familiar with all three, you'll be able to pick the right one for most projects in the future.
Now, to answer the specific problem:
First, when PyList_GetItem (or most other functions in the C API) returns NULL, this means there's an exception, so you should check the exception and log it. Trying to debug NULL return values in the C API without looking at the set exception is like trying to debug Python code without looking at the tracebacks.
Anyway, there are a few obvious reasons this function could fail: Maybe you're calling it with an out-of-bounds index, or maybe you're calling it on something that isn't a list at all.
In fact, the second one seems pretty obvious here. If printing out obj gives you this:
<ConfigurationParser.LP_LED object at 0x7f678a06fcb0>
Then you've got a (pointer to an) LED object, and LED objects aren't lists.
And if you look at your code, you don't seem to have a list of LED objects anywhere, at least not in the code you show us. You do have a POINTER(LED), which could hold a C-array-decayed C array of LEDs, but that's not the same thing as a Python list of them. It's just a C array, which you use C array syntax to dereference:
PyObject *led = ledarray[i];
Related
I am new to programming, I defined a function in c++ with tuple method for two returned variables, after I compiled the files, In the python file, I try to access the two returned variables inside the dynamic library which have been compiled, but it is not working, there is segmentation error happened when I tried to run the python program. But I actually success with single return variable from c++, I think there is might be special trick for accessing the two returned variable with tuple method from python.
The following is the c++ code with two returned variables with tuple method
std::tuple<double, double> Cassie2d::Step(ControllerTorque* action)
{
dyn_model_.setState(mj_data_->qpos, mj_data_->qvel);
dyn_state_.UpdateDynamicState(&dyn_model_);
mju_copy(mj_data_->ctrl, action->torques, nU);
mj_step(mj_model_, mj_data_);
return std::make_tuple(mj_data_->qacc,mj_data_->time);
Render();
}
The following is the python method I was applied, due to both of the returned variables are double type.
lib.StepTorque.argtypes = [ctypes.c_void_p, ctypes.POINTER(ControllerTorque)]
lib.StepTorque.restype = ctypes.c_double
I guess the restype is not just equal to ctypes.c_double, because it worked for one returned variable and it might not work for two returned variables.
Really appreciate for the help!
With return std::make_tuple(mj_data_->qacc,mj_data_->time) your creating a structure which holds two variables, defined in parenthesis. If the function would be call from native C++ code you should be able to unpack it using std:tie, eq:
double a, b;
std::tie(a,b) = Step(arg);
However you need to call it directly from python, which has totally different syntax for returning multiple variables (C++ pair or tuple is kind of bypass missing functionality). I see two option in python.
Give your C++ function a two callbacks to python, and send variables separately this way:
void Cassie2d::Step(ControllerTorque* action, callback1, callback2)
{
dyn_model_.setState(mj_data_->qpos, mj_data_->qvel);
dyn_state_.UpdateDynamicState(&dyn_model_);
mju_copy(mj_data_->ctrl, action->torques, nU);
mj_step(mj_model_, mj_data_);
callback1(mj_data_->qacc);
callback2(mj_data_->time);
Render();
}
(Recommened) You can try to use native python option to serve multiple return. Left you C++ function in its initial form and call in python this way:
VarA, VarB = Step(action)
I assume you have a reference to Cassie2d::Step(ControllerTorque* action) method in your python code, however you provided very small snippet.
I am creating Python bindings for a C library.
In C the code to use the functions would look like this:
Ihandle *foo;
foo = MethFunc();
SetArribute(foo, 's');
I am trying to get this into Python. Where I have MethFunc() and SetAttribute() functions that could be used in my Python code:
import mymodule
foo = mymodule.MethFunc()
mymodule.SetAttribute(foo)
So far my C code to return the function looks like this:
static PyObject * _MethFunc(PyObject *self, PyObject *args) {
return Py_BuildValue("O", MethFunc());
}
But that fails by crashing (no errors)
I have also tried return MethFunc(); but that failed.
How can I return the function foo (or if what I am trying to achieve is completely wrong, how should I go about passing MethFunc() to SetAttribute())?
The problem here is that MethFunc() returns an IHandle *, but you're telling Python to treat it as a PyObject *. Presumably those are completely unrelated types.
A PyObject * (or any struct you or Python defines that starts with an appropriate HEAD macro) begins with pointers to a refcount and a type, and the first thing Python is going to do with any object you hand it is deal with those pointers. So, if you give it an object that instead starts with, say, two ints, Python is going to end up trying to access a type at 0x00020001 or similar, which is almost certain to segfault.
If you need to pass around a pointer to some C object, you have to wrap it up in a Python object. There are three ways to do this, from hackiest to most solid.
First, you can just cast the IHandle * to a size_t, then PyLong_FromSize_t it.
This is dead simple to implement. But it means these objects are going to look exactly like numbers from the Python side, because that's all they are.
Obviously you can't attach a method to this number; instead, your API has to be a free function that takes a number, then casts that number back to an IHandle* and calls a method.
It's more like, e.g., C's stdio, where you have to keep passing stdin or f as an argument to fread, instead of Python's io, where you call methods on sys.stdin or f.
But even worse, because there's no type checking, static or dynamic, to protect you from some Python code accidentally passing you the number 42. Which you'll then cast to an IHandle * and try to dereference, leading to a segfault…
And if you were hoping Python's garbage collector would help you know when the object is still referenced, you're out of luck. You need to make your users manually keep track of the number and call some CloseHandle function when they're done with it.
Really, this isn't that much better than accessing your code from ctypes, so hopefully that inspires you to keep reading.
A better solution is to cast the IHandle * to a void *, then PyCapsule_New it.
If you haven't read about capsules, you need to at least skim the main chapter. But the basic idea is that it wraps up a void* as a Python object.
So, it's almost as simple as passing around numbers, but solves most of the problems. Capsules are opaque values which your Python users can't accidentally do arithmetic on; they can't send you 42 in place of a capsule; you can attach a function that gets called when the last reference to a capsule goes away; you can even give it a nice name to show up in the repr.
But you still can't attach any behavior to capsules.
So, your API will still have to be a MethSetAttribute(mymodule, foo) instead of mymeth.SetAttribute(foo) if mymodule is a capsule, just as if it's an int. (Except now it's type-safe.)
Finally, you can build a new Python extension type for a struct that contains an IHandle *.
This is a lot more work. And if you haven't read the tutorial on Defining Extension Types, you need to go thoroughly read through that whole chapter.
But it means that you have an actual Python type, with everything that goes with it.
You can give it a SetAttribute method, and Python code can just call that method. You can give it whatever __str__ and __repr__ you want. You can give it a __doc__. Python code can do isinstance(mymodule, MyMeth). And so on.
If you're willing to use C++, or D, or Rust instead of C, there are some great libraries (PyCxx, boost::python, Pyd, rust-python, etc.) that can do most of the boilerplate for you. You just declare that you want a Python class and how you want its attributes and methods bound to your C attributes and methods and you get something you can use like a C++ class, except that it's actually a PyObject * under the covers. (And it'll even takes care of all the refcounting cruft for you via RAII, which will save you endless weekends debugging segfaults and memory leaks…)
Or you can use Cython, which lets you write C extension modules in a language that's basically Python, but extended to interface with C code. So your wrapper class is just a class, but with a special private cdef attribute that holds the IHandle *, and your SetAttribute(self, s) can just call the C SetAttribute function with that private attribute.
Or, as suggested by user, you can also use SWIG to generate the C bindings for you. For simple cases, it's pretty trivial—just feed it your C API, and it gives you back the code to build your Python .so. For less simple cases, I personally find it a lot more painful than something like PyCxx, but it definitely has a lower learning curve if you don't already know C++.
I'm trying to write a small, modular program in Python that will dynamically load C functions and use them to execute computationally intensive code. In this program I am creating a couple of large matrices that I will be passing back and forth between my Python code to different C functions. I would prefer to pass these matrices by reference to avoid additional computational overhead.
I've tried reading through the Python docs for ctypes, but it doesn't seem to explain how to do this. I understand, for instance, that I can use byref() or pointer() to pass a pointer from Python to a C function, but how to I pass a pointer from an external C function back to Python? Given that variables are names in Python, is this just done "automatically" (for lack of a better term) when Python receives a value from a C function?
As a concrete example, this is what I'm trying to do (in pseudo-code):
foo = ctypes.CDLL("pathToFoo")
bar = ctypes.CDLL("pathToBar")
# Generate a large matrix by calling a C function.
reallyBigMatrix = foo.generateReallyBigMatrix()
# Pass reallyBigMatrix to another function and perform some operation
# on it. Since the matrix is really big, I would prefer to pass a
# reference to this matrix to my next C function rather than passing
# the matrix by value.
modifiedReallyBigMatrix = bar.modifyReallyBigMatrix(reallBigMatrix)
Alternatively, I'm using Python and C in conjunction as I need an easy way to dynamically load C functions in my program. I may pass paths to different C files to my Python program so that the Python program will execute the same code on different functions. As an example, I may want to run my program two different ways: keep the same "generateReallyBigMatrix" function in both runs, but used a different "modifyReallyBigMatrix" program between run 1 and run 2. If there is an easy, cross-platform way to do this in C or C++ I would be happy to implement that solution rather than using ctypes and Python. However, I haven't been able to find a simple, cross-platform solution.
You've mentioned that you are writing all the code, both Python and C, from yourself. I suggest not using ctypes for this, as ctypes is best suited for using C libraries that cannot be modified.
Instead, write a module in C using the Python C API. It will expose a single function to start with, like this:
PyObject* generateReallyBigMatrix(void);
Now, instead of trying to return a raw C pointer, you can return any Python object that you like. Good choices here would be to return a NumPy array (using the NumPy C API), or to return a Python "buffer" (from which a NumPy array can be constructed in Python if desired).
Either way, once this function is written in C using the appropriate APIs, your Python code will be simple:
import foo
reallyBigMatrix = foo.generateReallyBigMatrix()
To do it using the NumPy C API, your C code will look like this:
PyObject* generateReallyBigMatrix(void)
{
npy_intp dimension = 100;
PyArray_Descr* descr;
PyArray_DescrAlignConverter2("float64", &descr); // use any dtype
PyObject* array = PyArray_Empty(1, &dimension, descr, 0/*fortran*/);
Py_DECREF(descr);
void* data = PyArray_DATA(array);
// TODO: populate data
return array;
}
static PyMethodDef methods[] = {
{"generateReallyBigMatrix", generateReallyBigMatrix, METH_VARARGS, "doc"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
PyMODINIT_FUNC initfoo(void)
{
import_array(); // enable NumPy C API
Py_InitModule("foo", methods);
}
Note that the NumPy C API requires a slightly strange initialization ritual. See also Numpy C API: Link several object files
You then compile the code as a shared library called foo.so (no lib prefix).
I am using the PyObject functionality to call c functions, and
return Py_BuildValue("theTypeToConvert", myCVariable);
to return things back to my python program, this all works fine.
However I have a custom C type
extern HANDLE pascal
how do I pass an instance of this back to python so I can give it to other c functions later, the closest I could think of was to use
Py_BuildValue("O&", etc)
but this apparently mangles the variable as I am not getting the correct results later on.
If I understand correctly that you want the object to be "opaque" from the Python perspective, i.e. just a pointer value that you can pass around in Python but not operate on the object it points to, then you might be after the Capsule object.
Official Python docs on capsules:
https://docs.python.org/2/c-api/capsule.html#capsules
See also:
Passing a C pointer around with the Python/C API
I am using Python C API 2.7.2 with my C++ console application. There is one doubt regarding Python C API Boolean Objects
I am using:
PyObject* myVariable = Py_True;
Do I need to deference myVariable with Py_DECREF(myVariable)?
The Python C API documentation says:-
The Python True object. This object has no methods. It needs to be
treated just like any other object with respect to reference counts.
I searched the questions but could not find a clear answer for it.
Thanks.
Although it isn't dynamically created, it must be reference counted because PyObject variables can hold ANY Python object. Otherwise there would need to be checks for Py_True and other special cases scattered throughout the Python runtime as well as any C/C++ code that uses the API. That would be messy and error prone.
It needs to be treated just like any other object with respect to reference counts.
This means that you must incref it when you take its reference
{
Py_INCREF(Py_True);
PyObject* myVariable = Py_True;
and you must decref it when you dispose of it.
Py_DECREF(myVariable);
}