I have a custom Python module written in C, and I want to add an attribute to the module which is dynamically populated. E.g.:
import mymod
print(mymod.x) # At this point, the value of x is computed
The name of the attribute is known in advance.
From what I understand, this should be possible using descriptors, but it is not working as expected. I implemented a custom type, implemented the tp_descr_get function for the type, and assigned an instance of the type to my module, but the tp_descr_get function is never called.
Here is my test module:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <stdio.h>
static struct PyModuleDef testmod = {
PyModuleDef_HEAD_INIT,
"testmod",
NULL,
-1
};
typedef struct testattrib_s {
PyObject_HEAD
} testattrib;
static PyObject *testattrib_descr_get(PyObject *self, PyObject *obj, PyObject *type);
static int testattrib_descr_set(PyObject *self, PyObject *obj, PyObject *value);
PyTypeObject testattribtype = {
PyVarObject_HEAD_INIT (NULL, 0)
"testattrib", /* tp_name */
sizeof (testattrib), /* tp_basicsize */
/* lots of zeros omitted for brevity */
testattrib_descr_get, /* tp_descr_get */
testattrib_descr_set /* tp_descr_set */
};
PyMODINIT_FUNC
PyInit_testmod(void)
{
if (PyType_Ready(&testattribtype)) {
return NULL;
}
testattrib *attrib = PyObject_New(testattrib, &testattribtype);
if (attrib == NULL) {
return NULL;
}
PyObject *m = PyModule_Create(&testmod);
if (m == NULL) {
return NULL;
}
if (PyModule_AddObject(m, "myattrib", (PyObject *) attrib)) {
return NULL;
}
return m;
}
static PyObject *testattrib_descr_get(PyObject *self, PyObject *obj, PyObject *type)
{
printf("testattrib_descr_get called\n");
Py_INCREF(self);
return self;
}
static int testattrib_descr_set(PyObject *self, PyObject *obj, PyObject *value)
{
printf("testattrib_descr_set called\n");
return 0;
}
I test it like this:
import testmod
print(testmod.myattrib) # should call tp_descr_get
testmod.myattrib = 1 # should call tp_descr_set
The getter/setter functions are never called. What am I doing wrong?
I am running Python 3.8.5 on macOS 12.0.1 with a build from Anaconda:
>>> sys.version
'3.8.5 (default, Sep 4 2020, 02:22:02) \n[Clang 10.0.0 ]'
Descriptors operate only as attributes on a type. You would have to create your module as an instance of a module subclass equipped with the descriptor. The easiest way to do that is to use the Py_mod_create slot (not to be confused with __slots__).
Related
I am trying to create a Python extension module with multi-phase initialization, following the advice I got from a previous question. PEP 489 suggests that it is preferable for the Py_mod_create function to return a module subclass, which presumably means a subclass of PyModule, but I cannot figure out how to do this. In all my attempts, the module segfaults when it is imported. It works fine if Py_mod_create returns some other object, (one which is not a subclass of PyModule), but I am not sure if this will cause problems in future, since isinstance(mymodule, types.ModuleType) returns false in this case.
Following the docs on subclassing built-in types, I set tp_base to PyModule_Type, and my tp_init function calls PyModule_Type.tp_init. The docs also suggest that my structure should contain the superclass structure at the beginning, which in this case is PyModuleObject. This structure is not in the public Python header files, (it is defined in moduleobject.c in the Python sources), so for now I copied and paste the definitions of the PyModuleObject fields at the start of my structure. The complete code looks like this:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <stdio.h>
struct testmod_s {
// Fields copied from PyModuleObject in moduleobject.c
PyObject_HEAD
PyObject *md_dict;
struct PyModuleDef *md_def;
void *md_state;
PyObject *md_weaklist;
PyObject *md_name;
};
static int testmod_init(PyObject *self, PyObject *args, PyObject *kwds);
static PyObject *testmod_create(PyObject *spec, PyModuleDef *def);
static PyModuleDef_Slot testmod_slots[] = {
{Py_mod_create, testmod_create},
{0, 0} /* Sentinel */
};
static struct PyModuleDef testmod_def = {
PyModuleDef_HEAD_INIT, /* m_base */
"testmod", /* m_name */
NULL, /* m_doc */
sizeof(struct testmod_s), /* m_size */
NULL, /* m_methods */
testmod_slots, /* m_slots */
NULL, /* m_traverse */
NULL, /* m_clear */
NULL /* m_free */
};
static PyTypeObject testmodtype = {
PyVarObject_HEAD_INIT (NULL, 0)
"testmodtype", /* tp_name */
sizeof (struct testmod_s), /* tp_basicsize */
/* fields omitted for brevity, all set to zero */
Py_TPFLAGS_DEFAULT |
Py_TPFLAGS_BASETYPE, /* tp_flags */
/* fields omitted for brevity, all set to zero */
testmod_init, /* tp_init */
0, /* tp_alloc */
0, /* tp_new */
};
PyMODINIT_FUNC
PyInit_testmod(void)
{
testmodtype.tp_base = &PyModule_Type;
if (PyType_Ready(&testmodtype)) {
return NULL;
}
PyObject *moduledef = PyModuleDef_Init(&testmod_def);
if (moduledef == NULL) {
return NULL;
}
return moduledef;
}
static int testmod_init(PyObject *self, PyObject *args, PyObject *kwds)
{
if (PyModule_Type.tp_init((PyObject *)self, args, kwds) < 0) {
return -1;
}
return 0;
}
static PyObject *testmod_create(PyObject *spec, PyModuleDef *def)
{
struct testmod_s *module = PyObject_New(struct testmod_s, &testmodtype);
if (module == NULL) {
return NULL;
}
return (PyObject *) module;
}
Importing this module causes a segfault. What am I doing wrong?
I am running Python 3.8.5 on macOS 12.0.1 with a build from Anaconda:
>>> sys.version
'3.8.5 (default, Sep 4 2020, 02:22:02) \n[Clang 10.0.0 ]'
After some tests I could build a custom module type by copying parts of code from moduleobject.c
Your problem is that your code does create an instance of a subclass of module, but never initializes it and gets random values in key members. Additionaly, modules are expected to be gc collectables, so you have to create your custom module with PyObject_GC_New.
The following code replaces your initial testmod_create function with a full initialization of the module:
...
// copied from moduleobject.c
static int
module_init_dict(struct testmod_s* mod, PyObject* md_dict,
PyObject* name, PyObject* doc)
{
_Py_IDENTIFIER(__name__);
_Py_IDENTIFIER(__doc__);
_Py_IDENTIFIER(__package__);
_Py_IDENTIFIER(__loader__);
_Py_IDENTIFIER(__spec__);
if (md_dict == NULL)
return -1;
if (doc == NULL)
doc = Py_None;
if (_PyDict_SetItemId(md_dict, &PyId___name__, name) != 0)
return -1;
if (_PyDict_SetItemId(md_dict, &PyId___doc__, doc) != 0)
return -1;
if (_PyDict_SetItemId(md_dict, &PyId___package__, Py_None) != 0)
return -1;
if (_PyDict_SetItemId(md_dict, &PyId___loader__, Py_None) != 0)
return -1;
if (_PyDict_SetItemId(md_dict, &PyId___spec__, Py_None) != 0)
return -1;
if (PyUnicode_CheckExact(name)) {
Py_INCREF(name);
Py_XSETREF(mod->md_name, name);
}
return 0;
}
static PyObject* testmod_create(PyObject* spec, PyModuleDef* def)
{
struct testmod_s* module = PyObject_GC_New(struct testmod_s, &testmodtype);
if (module == NULL) {
return NULL;
}
PyObject* name = PyUnicode_FromString("testmod");
if (name == NULL) {
Py_DECREF(module);
return 0;
}
module->md_def = NULL;
module->md_state = NULL;
module->md_weaklist = NULL;
module->md_name = NULL;
module->md_dict = PyDict_New();
int cr = module_init_dict(module, module->md_dict, name, NULL);
Py_DECREF(name);
if (cr != 0) {
Py_DECREF(module);
return NULL;
}
return (PyObject*)module;
}
I was writing a C extension function, which was supposed to accept a str object as argument. The code is shown below:
static PyObject *py_print_chars(PyObject *self, PyObject *o) {
PyObject *bytes;
char *s;
if (!PyUnicode_Check(o)) {
PyErr_SetString(PyExc_TypeError, "Expected string");
return NULL;
}
bytes = PyUnicode_AsUTF8String(o);
s = PyBytes_AsString(bytes);
print_chars(s);
Py_DECREF(bytes);
Py_RETURN_NONE;
}
But as I test the module in python3 console, I find str objects can't pass the PyUnicode_Check:
>>> from sample2 import *
>>> print_chars('Hello world')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Expected string
As far as I know, Python 3’s str() type is called PyUnicode in C and the C code above was written in refer to "python cookbook3" Char15.13. I just can't work out the problem. Can anybody tell me what's wrong with my code.
Here is what "python cookbook3" said:
If for some reason, you are working directly with a PyObject * and can’t use PyArg_ParseTuple(), the following code samples show how you can check and extract a suitable char * reference, from both a bytes and string object:
/* Some Python Object (obtained somehow) */
PyObject *obj;
/* Conversion from bytes */
{
char *s;
s = PyBytes_AsString(o);
if (!s) {
return NULL; /* TypeError already raised */
}
print_chars(s);
}
/* Conversion to UTF-8 bytes from a string */
{
PyObject *bytes;
char *s;
if (!PyUnicode_Check(obj)) {
PyErr_SetString(PyExc_TypeError, "Expected string");
return NULL;
}
bytes = PyUnicode_AsUTF8String(obj);
s = PyBytes_AsString(bytes);
print_chars(s);
Py_DECREF(bytes);
}
And the whole code:
#include "Python.h"
#include "sample.h"
static PyObject *py_print_chars(PyObject *self, PyObject *o) {
PyObject *bytes;
char *s;
if (!PyUnicode_Check(o)) {
PyErr_SetString(PyExc_TypeError, "Expected string");
return NULL;
}
bytes = PyUnicode_AsUTF8String(o);
s = PyBytes_AsString(bytes);
print_chars(s);
Py_DECREF(bytes);
Py_RETURN_NONE;
}
/* Module method table */
static PyMethodDef SampleMethods[] = {
{"print_chars", py_print_chars, METH_VARARGS, "print character"},
{ NULL, NULL, 0, NULL}
};
/* Module structure */
static struct PyModuleDef samplemodule = {
PyModuleDef_HEAD_INIT,
"sample",
"A sample module",
-1,
SampleMethods
};
/* Module initialization function */
PyMODINIT_FUNC
PyInit_sample2(void) {
return PyModule_Create(&samplemodule);
}
If the goal is to accept exactly one argument, the function should be declared as METH_O, not METH_VARARGS; the former passes along the single argument without wrapping, the latter wraps in a tuple which would need to be unpacked or parsed to get the PyUnicode* inside.
I'm creating my own python module in C++. The module source code :
#include <Python.h>
#include <numpy/arrayobject.h>
#include "FMM.h"
struct module_state {
PyObject *error;
};
#define GETSTATE(m) ((struct module_state*)PyModule_GetState(m))
static PyObject* FMM(PyObject* self, PyObject* args)
{
PyObject *model_obj;
PyObject *time_obj;
PyObject *accepted_obj;
PyObject *lat_obj;
PyObject *lon_obj;
PyObject *h_obj;
int N;
if (!PyArg_ParseTuple(args, "OOOOOOi", &model_obj, &time_obj, &accepted_obj, &lat_obj, &lon_obj, &h_obj, &N))
{
Py_INCREF(Py_None);
return Py_None;
}
PyObject *model = PyArray_FROM_OTF(model_obj, NPY_FLOAT, NPY_INOUT_ARRAY);
PyObject *time = PyArray_FROM_OTF(time_obj, NPY_FLOAT, NPY_INOUT_ARRAY);
PyObject *accepted = PyArray_FROM_OTF(accepted_obj, NPY_BOOL, NPY_INOUT_ARRAY);
PyObject *lat = PyArray_FROM_OTF(lon_obj, NPY_DOUBLE, NPY_INOUT_ARRAY);
PyObject *lon = PyArray_FROM_OTF(lon_obj, NPY_DOUBLE, NPY_INOUT_ARRAY);
PyObject *h = PyArray_FROM_OTF(h_obj, NPY_DOUBLE, NPY_INOUT_ARRAY);
float *MODEL = static_cast<float *>(PyArray_DATA(model_obj));
float *TIME = static_cast<float *>(PyArray_DATA(time_obj));
bool *ACCEPTED = static_cast<bool *>(PyArray_DATA(accepted_obj));
double *LAT = static_cast<double *>(PyArray_DATA(lat_obj));
double *LON = static_cast<double *>(PyArray_DATA(lon_obj));
double *H = static_cast<double *>(PyArray_DATA(h_obj));
_FMM(MODEL, TIME, ACCEPTED, LAT, LON, H, N);
Py_INCREF(Py_None);
return Py_None;
}
static PyObject* SetModelSize(PyObject* self, PyObject* args)
{
int x, y, z;
if (!PyArg_ParseTuple(args, "iii", &x, &y, &z))
{
Py_INCREF(Py_None);
return Py_None;
}
_SaveModelSize(x, y, z);
// Rerutn values
Py_INCREF(Py_None);
return Py_None;
}
static PyMethodDef FMMMethods[] = {
{"FMM", FMM, METH_VARARGS, "Some Text"},
{"SetModelSize", SetModelSize, METH_VARARGS, "Some Text"},
{NULL, NULL, 0, NULL}
};
static int FMM_traverse(PyObject *m, visitproc visit, void *arg) {
Py_VISIT(GETSTATE(m)->error);
return 0;
}
static int FMM_clear(PyObject *m) {
Py_CLEAR(GETSTATE(m)->error);
return 0;
}
static struct PyModuleDef moduledef = {
PyModuleDef_HEAD_INIT,
"FMM",
NULL,
sizeof(struct module_state),
FMMMethods,
NULL,
FMM_traverse,
FMM_clear,
NULL
};
extern "C" PyObject * PyInit_FMM(void)
{
PyObject *module = PyModule_Create(&moduledef);
if (module == NULL)
return NULL;
struct module_state *st = GETSTATE(module);
st->error = PyErr_NewException("FMM.Error", NULL, NULL);
if (st->error == NULL)
{
Py_DECREF(module);
return NULL;
}
import_array();
Py_INCREF(module);
return module;
}
My function _FMM does numerical computation on large data sets (two floats and bool has size of 2e9 elements.
The module is compiled using setup.py:
from distutils.core import setup, Extension
import numpy.distutils.misc_util
import os
os.environ["CC"] = "g++"
os.environ["CXX"] = "g++"
module1 = Extension('FMM', sources = ['FMMmodule.c', 'FMM.c'])
setup (name = 'PackageName',
version = '1.0',
description = 'This is a demo package',
ext_modules = [module1],
include_dirs=numpy.distutils.misc_util.get_numpy_include_dirs())
When I run it from python I get Segmentation fault after about 60 seconds of calculation.
Every time I run in different step of calculation. I have read that it might be related to misuse of Py_INCREF() and Py_DECREF() - tried to play with that with no luck.
Anybody has experience how to deal with such behavior?
I'm testing using python 3.4.
I have finally sorted that out. I had to modify the code to make it compatible with newer numpy c-api: see more here: Migrating to numpy api 1.7
My problem is as follows:
I would like to call a C function from my Python file and return a value back to that Python file.
I have tried the following method of using embedded C in Python (the following code is the C code called "mod1.c). I am using Python3.4 so the format follows that given in the documentation guidelines. The problem comes when I call my setup file (second code below).
#include
#include "sum.h"
static PyObject*
mod_sum(PyObject *self, PyObject *args)
{
int a;
int b;
int s;
if (!PyArg_ParseTuple(args,"ii",&a,&b))
return NULL;
s = sum(a,b);
return Py_BuildValue("i",s);
}
/* DECLARATION OF METHODS */
static PyMethodDef ModMethods[] = {
{"sum", mod_sum, METH_VARARGS, "Descirption"}, // {"methName", modName_methName, METH_VARARGS, "Description.."}, modName is name of module and methName is name of method
{NULL,NULL,0,NULL}
};
// Module Definition Structure
static struct PyModuleDef summodule = {
PyModuleDef_HEAD_INIT,
"sum",
NULL,
-1,
ModMethods
};
/* INITIALIZATION FUNCTION */
PyMODINIT_FUNC initmod(void)
{
PyObject *m;
m = PyModule_Create(&summodule);
if (m == NULL)
return m;
}
Setup.py
from distutils.core import setup, Extension
setup(name='buildsum', version='1.0', \
ext_modules=[Extension('buildsum', ['mod1.c'])])
The result that I get when I compile my code using gcc is the following error: Cannot export PyInit_buildsum: symbol not defined
I would greatly appreciate any insight or help on this problem, or any suggestion in how to call C from Python. Thank you!
---------------------------------------EDIT ---------------------------------
Thank you for the comments:
I have tried the following now:
static PyObject*
PyInit_sum(PyObject *self, PyObject *args)
{
int a;
int b;
int s;
if (!PyArg_ParseTuple(args,"ii",&a,&b))
return NULL;
s = sum(a,b);
return Py_BuildValue("i",s);
}
For the first function; however, I still get the same error of PyInit_sum: symbol not defined
The working code from above in case anyone runs into the same error: the answer from #dclarke is correct. The initialization function in python 3 must have PyInit_(name) as its name.
#include <Python.h>
#include "sum.h"
static PyObject* mod_sum(PyObject *self, PyObject *args)
{
int a;
int b;
int s;
if (!PyArg_ParseTuple(args,"ii",&a,&b))
return NULL;
s = sum(a,b);
return Py_BuildValue("i",s);
}
/* DECLARATION OF METHODS*/
static PyMethodDef ModMethods[] = {
{"modsum", mod_sum, METH_VARARGS, "Descirption"},
{NULL,NULL,0,NULL}
};
// Module Definition Structure
static struct PyModuleDef summodule = {
PyModuleDef_HEAD_INIT,"modsum", NULL, -1, ModMethods
};
/* INITIALIZATION FUNCTION*/
PyMODINIT_FUNC PyInit_sum(void)
{
PyObject *m;
m = PyModule_Create(&summodule);
return m;
}
I'm trying to write a C-Extension for python. What I'd like to write is a ModPolynomial class which represents a polynomial on (Z/nZ)[x]/x^r-1[even though you may answer to my question without knowing anything of such polynomials].
I've written some code, which seems to work. Basically I just store three PyObject* in my ModPoly structure. Now I'd like to add the storage for the coefficients of the polynomial.
Since I want the coefficients to be read-only, I'd like to add a getter/setter pair of functions through PyGetSetDef. But when I access the getter from python(e.g print pol.coefficients) I receive a Segmentation Fault.
The original code, without the "coefficients" can be found here.
The code with the coefficients is here.
I hope someone of you can tell me where I'm doing wrong here.
By the way, also comments on the code are welcome. This is my first extension and I know that I'm probably doing things quite badly.
As ecatmur says in the comments PyVarObject store a certain number of "slots" at the end of the struct. So I've decided to avoid them.
The relevant code is:
typedef struct {
PyObject_HEAD
/* Type specific fields */
Py_ssize_t ob_size;
PyObject **ob_item;
Py_ssize_t allocated;
PyObject *r_modulus;
PyObject *n_modulus;
PyObject *degree;
} ModPoly;
static PyObject *
ModPoly_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
ModPoly *self;
self = (ModPoly *)type->tp_alloc(type, 0);
if (self != NULL) {
[...]
self->ob_size = 0;
self->ob_item = NULL;
self->allocated = 0;
}
return (PyObject *)self;
}
static int
ModPoly_init(ModPoly *self, PyObject *args, PyObject *kwds)
{
PyObject *r_modulus=NULL, *n_modulus=NULL, *coefs=NULL, *tmp;
PyObject **tmp_ar;
static char *kwlist[] = {"r_modulus", "n_modulus", "coefficients", NULL};
if (!PyArg_ParseTupleAndKeywords(args, kwds, "OO|O", kwlist,
&r_modulus, &n_modulus, &coefs))
return -1;
[...]
// The polynomial defaults to "x", so the coefficients should be [0, 1].
tmp_ar = (PyObject **)malloc(2 * sizeof(PyObject*));
if (tmp_ar == NULL) {
Py_DECREF(self->r_modulus);
Py_DECREF(self->n_modulus);
Py_DECREF(self->degree);
return -1;
}
tmp_ar[0] = PyInt_FromLong(0);
if (tmp_ar[0] != NULL) {
tmp_ar[1] = PyInt_FromLong(1);
}
if (tmp_ar[0] == NULL || tmp_ar[0] == NULL) {
Py_DECREF(self->r_modulus);
Py_DECREF(self->n_modulus);
Py_DECREF(self->degree);
Py_XDECREF(tmp_ar[0]);
Py_XDECREF(tmp_ar[1]);
free(tmp_ar);
return -1;
}
self->ob_size = 2;
self->allocated = 2;
return 0;
}
[...]
static PyObject *
ModPoly_getcoefs(ModPoly *self, void *closure)
{
printf("here"); // "here" is never printed
PyTupleObject *res=(PyTupleObject*)PyTuple_New(self->ob_size);
Py_ssize_t i;
PyObject *tmp;
if (res == NULL)
return NULL;
for (i=0; i < self->ob_size; i++) {
tmp = self->ob_item[i];
Py_INCREF(tmp);
PyTuple_SET_ITEM(res, i, tmp);
}
return (PyObject *)res;
}
static PyObject *
ModPoly_setcoefs(ModPoly *self, PyObject *value, void* closure)
{
PyErr_SetString(PyExc_AttributeError,
"Cannot set the coefficients of a polynomial.");
return NULL;
}
[...]
static PyGetSetDef ModPoly_getsetters[] = {
{"coefficients",
(getter)ModPoly_getcoefs, (setter)ModPoly_setcoefs,
"The polynomial coefficients.", NULL},
{NULL, 0, 0, NULL, NULL}
};
static PyTypeObject ModPolyType = {
PyObject_HEAD_INIT(NULL)
0, /* ob_size */
[...]
ModPoly_members, /* tp_members */
ModPoly_getsetters, /* tp_getset */
0, /* tp_base */
[...]
};
[...]
edit
I tried to reimplement the getter instruction by instruction, and I understood what I wasn't doing. In the ModPoly_init function I create the tmp_ar where I store the coefficients, but I do not assign it to self->ob_item.
-facepalm-
You only seem to be assigning to ModPoly.ob_item in ModPoly_new() (setting it to NULL).
ModPoly_getcoefs() then dereferences the null pointer, which would give you your segfault. It looks like you intended to assign to ob_item in ModPoly_init(), but don't actually get around to doing so.