This is my code for Python Wrapper for C function doing some simple calculation with Rotation matrix and Vectors. My function is transform_object, but it's not causing the problem (I was debuging it also without executing this func).
static PyObject* method_transform(PyObject* self, PyObject* args) {
double or1, or2, or3;
double cd1, cd2, cd3;
double ang1, ang2, ang3;
if (!PyArg_ParseTuple(args, "(ddd)(ddd)(ddd)", &or1, &or2, &or3, &cd1, &cd2, &cd3, &ang1, &ang2, &ang3)) {
return NULL;
}
double or[3] = { or1, or2, or3 };
double cd[3] = { cd1, cd2, cd3 };
double ang[3] = { ang1, ang2, ang3 };
double* vector = transform_object(or , cd, ang);
PyObject* list = PyList_New(0);
int len = 3;
for (int i = 0; i < len; i++) {
PyObject* the_object = PyFloat_FromDouble(vector[i]);
PyList_Append(list, the_object);
}
return list;
}
And I have a problem with memory leak, I supose. It's going to infinity.
I tried to commenting line by line and found that problem is in this line:
PyObject* the_object = PyFloat_FromDouble(vector[i]);
If I repleace vector[i] with for eg. or1 it's the same problem.
But I don't know why, there are any limitations to making PyObjects from arrays or something?
The problem is that you're creating a new object (this set it refcount to 1), appending it to a list and returning the list, this set refcount to 2
When that list goes out of scope it's collected by GC, and floats refcount drop to 1, but now there is no access to it but refcount is still not 0 so they live in the heap forever, so you have your memory leak.
As I said, just drop the refcount of the floats after appending to the list
Related
I have an numpy-array (arr) of type integer which is passed to a C-function with ctypes. I narrowed it down to a simple example. I have this three functions:
int fun(void *arr, void *arr_out){
fun2(arr,arr_out);
return(1);
}
void fun2(int *arr, double *arr_out){
for(int i = 0; i<10; i++){
*(arr_out+i) = calc((arr+i));
}
}
double calc(int *in){
return (double)((log(*in)*-0.15)/1.76);
}
Do I use the pointers in right way? My outputs looks kind of random. As if the pointer itself is in someway part of the calculation and not the value?
I have a custom python module for fuzzy string search, implementing Levenshtein distance calculation, it contains a python type, called levtree which has two members a pointer to a wlevtree C type (called tree) which does all the calculations and a PyObject* pointing to a python-list of python-strings, called wordlist. Here is what I need:
-when I create a new instance of levtree I use a constructor which takes a tuple of strings as its only input (and it is the dictionary in which the instance will perform all the searches), this constructor will have to create a new instance of wordlist into the new instance of levtree and copy the content of the input tuple into the new instance of wordlist. Here is my first code snippet and my first question:
static int
wlevtree_python_init(wlevtree_wlevtree_obj *self, PyObject *args, PyObject *kwds)
{
int numLines; /* how many lines we passed for parsing */
wchar_t** carg; /* argument to pass to the C function*/
unsigned i;
PyObject * strObj; /* one string in the list */
PyObject* intuple;
/* the O! parses for a Python object (listObj) checked
to be of type PyList_Type */
if (!(PyArg_ParseTuple(args, "O!", &PyTuple_Type, &intuple)))
{
return -1;
}
/* get the number of lines passed to us */
numLines = PyTuple_Size(intuple);
carg = malloc(sizeof(char*)*numLines);
/* should raise an error here. */
if (numLines < 0)
{
return -1; /* Not a list */
}
self->wordlist = PyList_New(numLines);
Py_IncRef(self->wordlist);
for(i=0; i<numLines; i++)
{
strObj = PyTuple_GetItem(intuple, i);
//PyList_Append(self->wordlist, string);
PyList_SetItem(self->wordlist, i, strObj);
Py_IncRef(strObj);
}
/* iterate over items of the list, grabbing strings, and parsing
for numbers */
for (i=0; i<numLines; i++)
{
/* grab the string object from the next element of the list */
strObj = PyList_GetItem(self->wordlist, i); /* Can't fail */
/* make it a string */
if(PyUnicode_Check(strObj))
{
carg[i] = PyUnicode_AsUnicode( strObj );
if(PyErr_Occurred())
{
return -1;
}
}
else
{
strObj = PyUnicode_FromEncodedObject(strObj,NULL,NULL);
if(PyErr_Occurred())
{
return -1;
}
carg[i] = PyUnicode_AsUnicode( strObj );
}
}
self->tree = (wlevtree*) malloc(sizeof(wlevtree));
wlevtree_init(self->tree,carg,numLines);
free(carg);
return 0;
}
Do I have to call Py_IncRef(self->wordlist); after self->wordlist = PyList_New(numLines); or it is redundant because references are already incremented in PyList_new?
Then I have the same doubt on PyList_SetItem(self->wordlist, i, strObj); and Py_IncRef(strObj);..
-when I destroy an instance of levtree i want to call the C function that frees the space occupied by tree, destroy wordlist and decrement all reference count on all the strings contained into wordlist.. Here is my tp_dealloc:
static void
wlevtree_dealloc(wlevtree_wlevtree_obj* self)
{
//wlevtree_clear(self);
if(self->tree!=NULL)
{
wlevtree_free(self->tree);
}
free(self->tree);
PyObject *tmp, *strObj;
unsigned i;
int size = PyList_Size(self->wordlist);
for(i=0; i<size; i++)
{
strObj = PyList_GetItem(self->wordlist, i);
Py_CLEAR(strObj);
}
Py_CLEAR(self->wordlist);
Py_TYPE(self)->tp_free((PyObject *)self);
}
Is it correct to make all the deallocation work here?
At the moment I don't have a tp_clear and a tp_free, do I need them?
My code at the moment works on allocation but not on deallocation because even though I can call init on the same python variable more than once, at the end of every python script (which works correctly) I get a "Segmentation Fault" which makes me think that something in the deallocation process goes wrong..
tp_clear is only needed if you implement cyclic garbage collection. It appears that this is not needed because you only maintain references to Python unicode objects.
tp_dealloc is called when the reference count of the object goes down to zero. This is where you destroy the object and its members. It should then free the memory occupied by the object by calling tp_free.
tp_free is where the memory for the object is freed. Implement this only if you implement tp_alloc yourself.
The reason for the separation between tp_dealloc and tp_free is that if your type is subclassed, then only the subclass knows how the memory was allocated and how to properly free the memory.
If your type is a subclass of an exisiting type, your tp_dealloc may need to call the tp_dealloc of the derived class, but that depends on the details of the case.
To summarize, it seems that you are handling object destruction correctly (except that you leak carg when exiting the function with an error).
I'm new to swig and I have the following function which i cant fix:
int get_list(IN const char * string, OUT struct entry ** results);
where struct entry is defined:
struct flux_entry
{
char * addr_str;
char cc[2];
};
the entry struct is properly converted to a python class.
I googled but couldn't find any explanation i could use.
I want to make it return a tuple of: (original get_list int return value, python list of entry python objects, based on the results buffer), but don't know how to convert the C entry to a python object in the argout code snippet.
I've managed to get thus far:
%typemap(argout) struct entry **
{
PyObject *o = PyList_New(0);
int i;
for(i=0; $1[i] ; i++)
{
PyList_Append(o, SWIG_HOW_TO_CONVERT_TO_PYOBJECT($1[i]));
}
$result = o;
}
what should i replace SWIG_HOW_TO_CONVERT_TO_PYOBJECT with?
passed results is supposed to be a pointer to a (struct entry *) type, set to NULL before calling get_list and should be set to an allocated array of struct entry * pointers. maybe a small wrapper function could make that easier?
the struct entry array is allocated within the C function using malloc, after calculating (inside get_list) how many elements are needed, and ends with a NULL pointer to indicate the end of the array.
i'd also like to make sure it's freed somewhere :)
thanks!
This should at least give you a starting point that works. I still wasn't sure how the data was returned, since to return an array of pointers so that the final one was NULL I'd think you'd need a struct entry ***, so I just set addr_str = NULL on the last one as a sentinel, and just put some dummy data partially based on the input string into the fields. Modify as needed to suit your needs:
%module example
// Insert the structure definition and function to wrap into the wrapper code.
%{
struct entry {
char* addr_str;
char cc[2];
};
int get_list(const char* string, struct entry** results)
{
*results = malloc(3 * sizeof(struct entry));
(*results)[0].addr_str = malloc(10);
strcpy((*results)[0].addr_str,"hello");
(*results)[0].cc[0] = string[0];
(*results)[0].cc[1] = string[1];
(*results)[1].addr_str = malloc(10);
strcpy((*results)[1].addr_str,"there");
(*results)[1].cc[0] = string[2];
(*results)[1].cc[1] = string[3];
(*results)[2].addr_str = NULL;
return 0;
}
%}
#include <typemaps.i>
// Define the structure for SWIG
struct entry {
char* addr_str;
char cc[2];
};
// Define a set of typemaps to be used for an output parameter.
// This typemap suppresses requiring the parameter as an input.
// A temp variable is created and passed instead.
%typemap(in,numinputs=0) struct entry **OUTPUT (struct entry* temp) %{
$1 = &temp;
%}
// Build a list of tuples containing the two entries from the struct.
// Append the new Python list object to the existing "int" result.
%typemap(argout) struct entry **OUTPUT {
int i = 0;
PyObject* out = PyList_New(0);
while((*$1)[i].addr_str != NULL)
{
//PyObject* t = PyTuple_New(2);
//PyTuple_SET_ITEM(t,0,PyBytes_FromString((*$1)[i].addr_str));
//PyTuple_SET_ITEM(t,1,PyBytes_FromStringAndSize((*$1)[i].cc,2));
//PyList_Append(out,t);
//Py_DECREF(t);
PyObject* s = SWIG_NewPointerObj(*$1+i,$descriptor(struct entry*),0);
PyList_Append(out,s);
Py_DECREF(s);
++i;
}
$result = SWIG_AppendOutput($result,out);
}
// Since a Python object was created and the data copied for each entry struct,
// free the memory returned in the structure.
//%typemap(freearg) struct entry **OUTPUT {
// int i=0;
// while((*$1)[i].addr_str != NULL) {
// free((*$1)[i].addr_str);
// ++i;
// }
// free(*$1);
//}
// Apply the OUTPUT typemap set to the "results" parameter.
%apply struct entry **OUTPUT {struct entry** results};
// Finally, define the function for SWIG
int get_list(const char* string, struct entry** results);
Demo (Python 3.3):
>>> import example
>>> example.get_list('abcd')
[0, [(b'hello', b'ab'), (b'there', b'cd')]]
Hope that helps.
Edit:
I commented out the tuple creation and just save the entry* proxy instead. This doesn't leak Python objects, but the memory malloced for use by an entry* is not freed. I'm not sure where to put that, although I'm experimenting with %extend.
In part of the code of a C module I am integrating with Python, I have a char** (array of strings) which is repeatedly allocated, filled with allocated strings, then freed and allocated again. The general pattern is that when a certain function is called (from Python) supplying the new contents of the array (as a list), it iterates through the array of strings, freeing each of them, then frees the array itself. It then allocates the array again to hold the contents of the new Python list, then allocates memory for each of the strings to hold.
All that to say that I am getting an error when attempting to free one of the strings in the list. This error is deterministic; it is always the same word from the same list of words at the same point in the program, but there is nothing extraordinary about that word or list of words. (It is just ["CCellEnv", "18", "34"], which is a similar format to many others) I tried adding some debug code to the loop that allocates the strings; here is the function that produces the error:
static PyObject* py_set_static_line(PyObject* self, PyObject* args)
{
int i;
//Free the old values of the allocated variables, if there are any
if (numStaticWords > 0)
{
for (i = 0; i < numStaticWords; i++)
{
printf("Freeing word %d = '%s'\n", i, staticWords[i]);
free(staticWords[i]);
}
free(staticWords);
free(staticWordMatches);
}
//Parse arguments
PyObject* wordList;
unsigned short numWords;
PyObject* wordMatchesList;
if (!PyArg_ParseTuple(args, "O!HO!", &PyList_Type, &wordList, &numWords, &PyList_Type, &wordMatchesList))
return NULL;
numStaticWords = numWords;
if (numStaticWords > 0)
{
staticWords = malloc(sizeof(char*) * numStaticWords);
staticWordMatches = malloc(sizeof(int) * numStaticWords);
PyObject* wordObj;
PyObject* matchObj;
char* word;
for (i = 0; i < numStaticWords; i++)
{
//wordList is the list of strings passed from Python
wordObj = PyList_GetItem(wordList, i);
word = PyString_AsString(wordObj); //word is "18" in the failing case
//staticWords is the char** array of strings, which has already been malloc'd
staticWords[i] = malloc(sizeof(char) * strlen(word));
//Test freeing the word to see if it crashes
free(staticWords[i]); //Crashes for one specific word
staticWords[i] = malloc(sizeof(char) * strlen(word));
strcpy(staticWords[i], word);
matchObj = PyList_GetItem(wordMatchesList, i);
if (matchObj == Py_None)
{
staticWordMatches[i] = -1;
}
else
{
staticWordMatches[i] = PyInt_AsLong(matchObj);
}
}
}
Py_RETURN_NONE;
}
So, somehow, always and only for this specific string, allocating the memory to put it in, then immediately freeing that memory causes an error. The actual text of the string is not even copied into the memory. What could be causing this mysterious behavior?
Here
staticWords[i] = malloc(sizeof(char) * strlen(word));
strcpy(staticWords[i], word);
you are missing to allocate the 0-termination for the "strings". So any operation on those character arrays as strings, most likely will lead to undefined behaviour.
Do it this way:
{
int isNull = !word;
staticWords[i] = calloc(sizeof(*staticWords[i]), (isNull ?0 :strlen(word)) + 1);
strcpy(staticWords[i], isNull ?"" :word);
}
While attempting to read a Python list filled with float numbers and to populate real channels[7] with their values (I'm using F2C, so real is just a typedef for float), all I am able to retrieve from it are zero values. Can you point out the error in the code below?
static PyObject *orbital_spectra(PyObject *self, PyObject *args) {
PyListObject *input = (PyListObject*)PyList_New(0);
real channels[7], coefficients[7], values[240];
int i;
if (!PyArg_ParseTuple(args, "O!", &PyList_Type, &input)) {
return NULL;
}
for (i = 0; i < PyList_Size(input); i++) {
printf("%f\n", PyList_GetItem(input, (Py_ssize_t)i)); // <--- Prints zeros
}
//....
}
PyList_GetItem will return a PyObject*. You need to convert that to a number C understands. Try changing your code to this:
printf("%f\n", PyFloat_AsDouble(PyList_GetItem(input, (Py_ssize_t)i)));
Few things I see in this code.
You leak a reference, don't create that empty list at the beginning, it's not needed.
You don't need to cast to PyListObject.
PyList_GetItem returns a PyObject, not a float. Use PyFloat_AsDouble to extract the value.
If PyList_GetItem returns NULL, then an exception has been thrown, and you should check for it.